Model deployment method, electronic device, chip system, medium and program product

Abstract

The embodiments of the present application provide a model deployment method, electronic device, chip system, medium and program product, which are applied to the field of terminal technology, and are conducive to avoiding the situation of freezing or application crash in electronic devices, and improving user experience. The method includes: at a first moment, P sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the first memory, the memory occupied by the P sub-models in the first computing unit is the second memory, and the second memory is less than or equal to the first memory; at a second moment, Q sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the third memory, the memory occupied by the Q sub-models in the first computing unit is the fourth memory, and the third memory is less than or equal to the fourth memory; wherein the first memory is different from the third memory, and the second memory is different from the fourth memory.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a model deployment method, electronic device, chip system, medium and program product. Background Art

[0002] As people's demand for electronic device functions continues to grow, electronic device manufacturers have deployed models in electronic devices to improve the intelligence level of electronic devices. For example, image processing models can be deployed in electronic devices to improve the image processing capabilities of electronic devices. Artificial intelligence (AI) models can also be deployed in electronic devices to improve the interaction experience between users and devices.

[0003] However, the model usually requires a large amount of running memory. When the model is running on an electronic device, the electronic device may become stuck or the application may crash, affecting the user experience. Summary of the invention

[0004] The embodiments of the present application provide a model deployment method, an electronic device, a chip system, a medium, and a program product, which are applied to the field of terminal technology and are helpful in avoiding freezes in electronic devices or crashes of applications, thereby improving user experience.

[0005] In a first aspect, an embodiment of the present application proposes a model deployment method, which is applied to an electronic device, wherein the electronic device includes a first computing unit and a target model, wherein the target model includes multiple sub-models, and the method includes: at a first moment, P sub-models of the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the first memory, the memory occupied by the P sub-models in the first computing unit is the second memory, and the second memory is less than or equal to the first memory; at a second moment, Q sub-models of the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the third memory, the memory occupied by the Q sub-models in the first computing unit is the fourth memory, and the third memory is less than or equal to the fourth memory; wherein the first memory is different from the third memory, and the second memory is different from the fourth memory.

[0006] In an embodiment of the present application, the memory occupied by the sub-model loaded on the first computing unit is less than or equal to the remaining available memory of the first computing unit. This is conducive to achieving reasonable allocation of resources and avoiding the problem of resource preemption caused by the memory occupied by the sub-models loaded on the first computing unit exceeding the remaining available memory of the first computing unit. In other words, it is conducive to avoiding the problem of electronic device freezing or application crash, thereby improving user experience.

[0007] Furthermore, at different times, the remaining available memory of the first computing unit and the memory occupied by the sub-model loaded on the first computing unit may be different, which is conducive to improving the flexibility of model deployment.

[0008] It should be understood that the selection of the P sub-models loaded on the first computing unit is related to the first memory and the memory required for the P sub-models; the selection of the Q sub-models loaded on the first computing unit is related to the third memory and the memory required for the Q sub-models.

[0009] It should also be understood that the memory required for the P sub-models can be understood as the running memory theoretically required by each of the P sub-models obtained during the training and testing process of the P sub-models, and the first memory can be understood as the memory actually occupied by the P sub-models when running on the first computing unit.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the electronic device also includes a second computing unit, and the method also includes: loading R sub-models on the second computing unit, the execution order of the R sub-models in the target model is after the P sub-models, and when the R sub-models are loaded on the second computing unit, the status of the P sub-models on the first computing unit is any one of loaded and not running, running, releasing, or released.

[0011] In an embodiment of the present application, after loading P sub-models in the target model on the first computing unit, R sub-models in the target model whose execution order is after the P sub-models can be loaded onto the second computing unit. In this way, after the first computing unit obtains the inference result based on the P sub-models, it can continue to use the inference result and the R sub-models loaded on the second computing unit for inference, without waiting for the release time of the P sub-models and the loading time of the R sub-models, which is conducive to shortening the overall inference time of the target model, improving the inference efficiency of the target model, and providing users with a better user experience.

[0012] In combination with the first aspect, in certain implementations of the first aspect, before loading P sub-models in the target model on the first computing unit, the method also includes: obtaining the remaining available memory of the first computing unit, the sub-models to be loaded, the memory required by each of the sub-models to be loaded, and the execution order of each sub-model in the sub-models to be loaded, where the sub-models to be loaded are part or all of the multiple sub-models; starting from the sub-model with the first execution order in the sub-models to be loaded, traversing according to the execution order of the sub-models to be loaded, accumulating the memory required by each traversed sub-model to obtain a first accumulated memory and a second accumulated memory, the number of sub-models corresponding to the second accumulated memory is one more than the number of sub-models corresponding to the first accumulated memory; when the first accumulated memory is less than or equal to the remaining available memory of the first computing unit, and the second accumulated memory is greater than the remaining available memory of the first computing unit, the sub-model corresponding to the first accumulated memory is determined to be P sub-models.

[0013] This helps to ensure that the memory occupied by the P sub-models in the target model loaded on the first computing unit is less than or equal to the remaining available memory of the first computing unit, which helps to avoid resource preemption, that is, it helps to avoid the problem of electronic device freezing or application crashes, thereby improving user experience.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the electronic device includes multiple computing units, the multiple computing units include a first computing unit, and P sub-models are sub-models that the first computing unit is expected to bear. The method also includes: determining the sub-models that each of the multiple computing units other than the first computing unit is expected to bear; calculating the efficiency of each of the multiple computing units in running the sub-model that it is expected to bear; loading the P sub-models of the target model on the first computing unit, including: when the efficiency of the first computing unit running the P sub-models is the maximum value of the efficiency of each of the multiple computing units in running the sub-model that it is expected to bear, loading the P sub-models of the target model on the first computing unit.

[0015] In the embodiment of the present application, the efficiency of each computing unit in the multiple computing units in running the sub-models that are expected to be undertaken by each computing unit can be used to characterize the speed of the multiple computing units in running the sub-models that are expected to be undertaken by each computing unit. Among the efficiencies of each computing unit in the multiple computing units in running the sub-models that are expected to be undertaken by each computing unit, the computing unit with the highest efficiency is selected, and the sub-model that is expected to be undertaken is loaded on the computing unit with the highest efficiency. In this way, it is beneficial to load the sub-model to be loaded onto the computing unit with the highest processing capability on the basis of avoiding resource preemption, which is beneficial to improve the reasoning efficiency of the target model.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the efficiency of the first computing unit in running P sub-models is proportional to the sum of the total number of floating-point operations of each of the P sub-models, and / or is inversely proportional to the sum of the theoretical time for the first computing unit to run each of the P sub-models; wherein, the theoretical time for the first computing unit to run each of the P sub-models is obtained through experiments, and the theoretical time for the first computing unit to run each of the P sub-models is related to the frequency of the first computing unit.

[0017] Optionally, the calculation method of the efficiency of the first computing unit running P sub-models can be similar to the calculation method of the inference efficiency of the i-th computing unit in method 200, which is not repeated here. The "efficiency" here and the "inference efficiency" in the above text can be understood to have the same meaning.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the electronic device includes a main thread and at least one sub-thread, the at least one sub-thread including a first sub-thread; calculating the efficiency of each of the multiple computing units in running the respective sub-models that are expected to be undertaken, including: the first sub-thread calculates the efficiency of each of the multiple computing units in running the respective sub-models that are expected to be undertaken, and sends model deployment reference information containing the efficiency of each of the multiple computing units in running the respective sub-models that are expected to be undertaken to the main thread; when the main thread determines that there is no model deployment reference information from other sub-threads other than the first sub-thread in at least one sub-thread, the main thread loads P sub-models in the target model on the first computing unit, including: when the efficiency of the first computing unit running the P sub-models is the maximum value of the efficiency of each of the multiple computing units running the respective sub-models that are expected to be undertaken, the main thread loads the P sub-models in the target model on the first computing unit.

[0019] Optionally, the first sub-thread here may have the same meaning as the sub-thread 1 described in method 200, the main thread may have the same meaning as the main thread in method 200, and the model deployment reference information received by the main thread may have the same meaning as the model deployment reference information described in method 200, but this application does not make any specific limitations on this.

[0020] It is worth noting that the main thread determines that there is no model deployment reference information from at least one sub-thread other than the first sub-thread. It can be understood that the first sub-thread did not traverse the sub-model with the same-level branches in the process of calculating the efficiency of each computing unit in multiple computing units running the sub-model that each computing unit is expected to undertake.

[0021] In combination with the first aspect, in certain implementations of the first aspect, when the main thread determines that there is model deployment reference information from other sub-threads other than the first sub-thread in at least one sub-thread, the main thread obtains the most efficient computing unit corresponding to each model deployment reference information in other sub-threads and the sub-model that the most efficient computing unit is expected to undertake; and loads P sub-models in the target model on the first computing unit, including: when the efficiency of the first computing unit obtained by the first sub-thread to run the P sub-models is greater than the maximum value of the efficiency of each computing unit in the other computing units to run the sub-models that are expected to be undertaken, and the most efficient computing unit corresponding to each model deployment reference information in other sub-threads is not the first computing unit, the main thread loads the P sub-models in the target model on the first computing unit.

[0022] Since the process of each sub-thread obtaining its model deployment reference information can be independent of each other, there is a possibility that the most efficient computing unit corresponding to the model deployment reference information obtained by each sub-thread is the same. In this case, if the sub-model that is expected to be undertaken is loaded directly on the most efficient computing unit, the loading may fail due to insufficient memory. Therefore, in an embodiment of the present application, when the most efficient computing unit corresponding to each model deployment reference information in other sub-threads is not the first computing unit, loading P sub-models in the target model on the first computing unit is beneficial to the successful loading of the P sub-models and to maintaining the stability of the target model operation.

[0023] Furthermore, when the efficiency of the first computing unit obtained by the first sub-thread in running P sub-models is greater than the maximum value of the efficiency of each computing unit in other computing units in running the sub-models that they are expected to undertake, and the computing unit with the highest efficiency corresponding to each model deployment reference information in other sub-threads is the same as the first computing unit, the solution can refer to the expanded description at S214 in method 200, and will not be repeated here.

[0024] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the main thread loads the sub-model that the most efficient computing unit is expected to undertake on the most efficient computing unit corresponding to each model deployment reference information in other sub-threads.

[0025] In this way, sub-models on different branches in the same-level branches can be loaded onto different computing units, which is conducive to improving model operation efficiency and saving model inference time.

[0026] Optionally, the main thread can load P sub-models in the target model on the first computing unit, while loading the sub-model that the most efficient computing unit is expected to undertake on the most efficient computing unit corresponding to each model deployment reference information in other sub-threads. This is conducive to synchronous reasoning of sub-models in branches at the same level, which is conducive to improving model operation efficiency and saving model reasoning time.

[0027] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: at a third moment, T sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the fifth memory, the memory occupied by the T sub-models in the first computing unit is the sixth memory, and the sixth memory is less than or equal to the fifth memory; at a fourth moment, U sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the seventh memory, the memory occupied by the U sub-models in the first computing unit is the eighth memory, and the eighth memory is less than or equal to the seventh memory; wherein, the fifth memory is different from the seventh memory, and the sixth memory is the same as the eighth memory.

[0028] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: at a fifth moment, V sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the ninth memory, the memory occupied by the V sub-models in the first computing unit is the tenth memory, and the tenth memory is less than or equal to the ninth memory; at a sixth moment, Y sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the eleventh memory, the memory occupied by the Y sub-models in the first computing unit is the twelfth memory, and the twelfth memory is less than or equal to the eleventh memory; wherein, the ninth memory is the same as the eleventh memory, and the tenth memory is different from the twelfth memory.

[0029] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: at a seventh moment, Z sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the thirteenth memory, the memory occupied by the Z sub-models in the first computing unit is the fourteenth memory, and the fourteenth memory is less than or equal to the thirteenth memory; at an eighth moment, K sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the fifteenth memory, the memory occupied by the K sub-models in the first computing unit is the sixteenth memory, and the sixteenth memory is less than or equal to the fifteenth memory; wherein, the thirteenth memory is the same as the fifteenth memory, and the fourteenth memory is the same as the sixteenth memory.

[0030] Optionally, the meanings of the third moment and the fourth moment, the fifth moment and the sixth moment, and the seventh moment and the eighth moment may be similar to the meanings of the first moment and the second moment, but the present application does not make specific limitations on this.

[0031] In the embodiment of the present application, letters such as P, Q, R, S, T, U, V, Y, Z, and K are used to identify the number of sub-models, and their values ​​can all be integers greater than zero.

[0032] In a second aspect, an embodiment of the present application provides a model deployment device, which may be an electronic device, or a chip or chip system in an electronic device. When the model deployment device is an electronic device, the processing unit may be a processor. The model deployment device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements a model deployment method described in the first aspect or any possible implementation of the first aspect. When the model deployment device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements a model deployment method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit in the chip (for example, a register, a cache, etc.), or a storage unit in the electronic device located outside the chip (for example, a read-only memory, a random access memory, etc.).

[0033] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory, wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program or instructions. When the computer program or instructions are run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0036] In a sixth aspect, the present application provides a chip or a chip system, which includes one or more processors and a communication interface, wherein the communication interface and the one or more processors are interconnected by lines, and the one or more processors are used to run computer programs or instructions to execute the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0037] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0038] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic block diagram of the software structure of the electronic device provided in the embodiment of the present application;

[0040] Figure 2 A schematic flow chart of a model deployment method provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the hierarchical relationship of the target model provided in the embodiment of the present application;

[0042] Figure 4 A schematic flow chart of a model deployment method provided in an embodiment of the present application;

[0043] Figure 5 A schematic block diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0044] Figure 6 A schematic block diagram of a chip structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0046] 1. Computing unit: It can be the basic unit in an electronic device responsible for performing computing operations and is used to perform various arithmetic and logical operations.

[0047] The computing unit in the embodiments of the present application may include one or more of the following types of computing units: a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application processor (AP), an image signal processor (ISP), a digital signal processor (DSP), a neural-network processing unit (NPU), or an application specific integrated circuit (ASIC), etc., but the present application does not make any specific limitation on this.

[0048] The computing units in the embodiments of the present application may be heterogeneous computing units or homogeneous computing units, and the present application does not specifically limit this. Among them, heterogeneous computing units refer to computing units containing different types of instruction sets and architectures, which can be integrated into the same system or platform to achieve more efficient computing performance and lower power consumption through joint computing; homogeneous computing units refer to computing units containing the same type of instruction sets and architectures. In a homogeneous computing system, all computing units follow the same instruction set and architecture, which makes them have similar performance and characteristics when processing tasks.

[0049] 2. Model: An abstract representation or simplified description of the real world, system, process, concept or phenomenon. For example, in the field of artificial intelligence, a model usually refers to a machine learning model that has been trained and optimized to perform specific tasks, such as image recognition, speech recognition, intelligent question answering, text generation and other functions.

[0050] It should be understood that as more data is added and algorithms are improved, models can be continuously trained, updated, and iterated to improve their performance and accuracy.

[0051] The target model involved in the embodiments of the present application may be stored in the electronic device, or may be obtained by the electronic device in real time from the cloud. In some implementations, when the target model is executed, the electronic device needs to load the model into the running memory of the computing unit.

[0052] 3. Hierarchical relationship: The target model can include multiple sub-models. The hierarchical relationship between sub-models mainly refers to a hierarchical arrangement between sub-models organized according to a certain logic or structure. This hierarchical relationship helps to clearly express the complex relationships such as subordination, inclusion or association between different sub-models. It can also be understood that the hierarchical relationship can indicate the execution order of each sub-model in the target model.

[0053] In the embodiments of the present application, a sub-model can be understood as a part of the target model that is divided according to functional, logical or physical boundaries and can be responsible for one or a class of tasks. For example, in a convolutional neural network (CNN), a convolutional layer can be regarded as a sub-model. A sub-model can also be called a node, which can be understood as the smallest division unit or the lowest level of the model.

[0054] 4. Floating Point Operations (FLOPs): refers to the total number of floating point operations actually performed by the model during inference or training, including multiplication, division, addition, subtraction, etc.

[0055] It should be understood that FLOPs is a measure of the computational complexity of the model, which indicates the number of floating-point operations required for the model to complete one forward propagation (or one inference). A higher FLOPs value usually indicates that the model has a higher computational complexity and requires more computing resources to process.

[0056] 5. Other terms

[0057] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0058] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0059] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0060] 6. Electronic equipment

[0061] The electronic device of the embodiment of the present application may be a handheld device including a model, a vehicle-mounted device, etc. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, electronic devices in 5G networks or future evolved public land mobile communication networks (public landmobile networks). The electronic devices in the network (PLMN) and the like are not limited to this in the embodiments of the present application.

[0062] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0063] In addition, in the embodiments of the present application, the electronic device may also be an electronic device in an Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0064] The electronic device in the embodiments of the present application may also be referred to as: electronic device, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0065] In an embodiment of the present application, the electronic device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software.

[0066] The memory required for electronic devices to run models is usually large, and the model cannot be fully run in one computing unit. In some related technologies, the models are grouped, and for the multiple computing units included in the electronic device, the sub-models to be run are preset for each computing unit (or it can also be understood as specifying the corresponding computing units for each sub-model in the model), for example, computing unit A is preset to correspond to sub-model 1 and sub-model 2, computing unit B corresponds to sub-model 3, computing unit C corresponds to sub-model 4 and sub-model 5, etc. This means that the memory occupied by the sub-models on each computing unit is fixed when running. In this way, when the remaining running memory of the computing unit cannot meet the model running requirements, resource preemption may occur, causing other running applications to freeze or flash back, and the application that calls the model cannot run normally, affecting the user experience.

[0067] In view of this, the embodiments of the present application provide a model deployment method, an electronic device, a chip system, a medium and a program product, which can make the memory occupied by the sub-model loaded on the first computing unit less than or equal to the remaining available memory of the first computing unit. At different times, the remaining available memory of the first computing unit and the memory occupied by the sub-model loaded on the first computing unit may be different. This is conducive to the rational allocation of resources and helps to avoid problems such as freezes in electronic devices and application crashes caused by applications competing for resources, thereby improving user experience.

[0068] Furthermore, in an embodiment of the present application, the electronic device can allocate and load sub-models for the computing unit according to the remaining memory of the computing unit, and the sub-models running on each computing unit can change dynamically based on the remaining available memory of the computing unit, which is conducive to improving the flexibility of model deployment.

[0069] In order to facilitate understanding of the embodiments of the present application, the software structure of the electronic device provided by the present application is first described in detail below. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiments of the present application take the Android system of the layered architecture as an example to exemplify the software structure of the electronic device.

[0070] Figure 1 The software structure block diagram of the electronic device is shown as an example. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The layered architecture can adopt an Android system, a Linux operating system, a Unix operating system, an iOS operating system, a Windows operating system, or other operating systems, which is not limited in the embodiments of the present application. The following takes the Android system with a layered architecture as an example to illustrate the software architecture of the electronic device provided in the embodiments of the present application.

[0071] In some implementations, the layered architecture can divide the software system of the electronic device into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom: application layer (applications), application framework layer (applicationframework), system layer (native), kernel layer (kernel) and hardware layer.

[0072] like Figure 1 As shown, the application layer may include a series of application packages. The application layer runs the application by calling the application programming interface (API) provided by the application framework layer, which may be referred to as the target application in the embodiment of the present application. The target application may be a system application or a third-party application, and the present application does not limit this.

[0073] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0074] In the embodiment of the present application, the application framework layer may include a target application access interface and an activity manager service (AMS), etc. Among them, the target application access interface can be used to provide an application programming interface and a programming framework for the target application. The activity manager service is a core service in the Android system responsible for managing the application life cycle and the Activity stack, and can be used to create a process for the target application.

[0075] The system layer may include one or more target models. For example, the target model is a camera algorithm model related to image processing. The camera algorithm model may include the operating code and data for implementing the image processing method provided in the embodiment of the present application. In the embodiment of the present application, the target model includes multiple sub-models (for example Figure 1 Sub-model 1, sub-model 2, sub-model 3...sub-model N shown in the figure, N is a non-zero positive integer), each sub-model can also be called a node of the target model.

[0076] The kernel layer is a layer between hardware and software, and may include drivers for various hardware. In the present application, in an embodiment, the kernel layer may include a processor driver, which may be, for example, a graphics processor driver and / or a central processing unit driver.

[0077] The hardware layer may include one or more computing units, such as Figure 1The computing unit 1, computing unit 2, computing unit 3, ... computing unit M shown in the figure, M is a non-zero positive integer. Optionally, the types of computing units may include a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC).

[0078] It should be understood that in some embodiments, the layers that implement the same functions may be called other names, or the layers that can implement the functions of multiple layers may be regarded as one layer, or the layers that can implement the functions of multiple layers may be divided into multiple layers. The module names involved in the embodiments of the present application may be defined as other names as long as the functions of each module can be implemented. The embodiments of the present application do not impose specific restrictions on the names of the layers in the software structure and the names of the modules included in the layers.

[0079] In one possible scenario, in response to a user's launch operation on a target application, the desktop application (Launcher) in the electronic device sends a target application launch request (startActivity) to the system service process (System_server) through an inter-process communication (Binder IPC) mechanism. After receiving the startActivity request, the activity management service in the system service process checks whether the requested application already has a target application process running. If not, a target application process is created.

[0080] It should be understood that the target application process can call the target model to implement its functions during operation. For example, if the target application is a camera application and the target model is a camera algorithm model, the target application can call the camera algorithm model to perform image recognition, image fusion, image enhancement and beautification, image denoising and restoration, etc.

[0081] Optionally, the process of the target application can determine how to deploy the model during the process of creating the target process, or after the process of the target application is created and the user triggers an operation that requires the participation of the target model, the model deployment strategy can be determined in real time. This application does not limit this.

[0082] It should be understood that the target model may include the identifiers of 1 to N sub-models, and the hierarchical relationship between the sub-models. Figure 3 The following example shows the possible hierarchical relationship of some sub-models in the target model. Figure 3As shown, the output of sub-model 1 can be used as the input of sub-model 2, the output of sub-model 2 can be used as the input of sub-model 3, the output of sub-model 3 can be used as the input of sub-model 4 and sub-model 5, sub-model 1 can be regarded as the predecessor of sub-model 2, sub-model 3 can be regarded as the successor of sub-model 2, and sub-model 2 can be regarded as the predecessor of sub-model 3, and sub-model 4 and sub-model 5 can be regarded as the successors of sub-model 3. It can be seen that under such a hierarchical relationship, electronic devices can be preferentially deployed as sub-models that are higher in the hierarchical relationship, such as sub-model 1.

[0083] Since the target model usually requires a large amount of operating memory, a single computing unit in an electronic device often has difficulty in independently performing operations on the target model. Therefore, the sub-models in the target model can be grouped and deployed to different computing units in the form of one or more sub-models as a group. The deployment of a sub-model to a computing unit described in this application can be understood as loading the sub-model onto the computing unit and performing operations on the sub-model through the computing unit.

[0084] The model deployment method 200 provided in an embodiment of the present application is described below.

[0085] It should be understood that the process of the target application may include multiple threads, for example, a main thread and multiple sub-threads. Method 200 may be implemented through the interaction of the main thread and the sub-threads. Among them, sub-thread 1 among the multiple sub-threads may be used to determine the inference efficiency of each computing unit among the multiple computing units, and the main thread may be used to determine on which computing unit to load the sub-model based on the inference efficiency of each computing unit determined by the sub-thread, and complete the loading.

[0086] Optionally, the child thread 1 may be created during the process creation of the target application, or may be created in response to a user-triggered operation that requires the participation of the target model, which is not limited in the present application.

[0087] The method 200 may specifically include the following steps:

[0088] S201, sub-thread 1 obtains the memory and hierarchical relationship required by each sub-model in the target model during operation, and the remaining available memory of each computing unit in multiple computing units.

[0089] In some implementations, information of the target model has been stored in the electronic device, and the information of the target model stored in the electronic device may include one or more of the following: identification of each sub-model in the target model, hierarchical relationship of each sub-model in the target model, memory required for each sub-model in the target model to run, convolution kernel size of each sub-model, number of channels of the input feature map of each sub-model, number of channels of the output feature map of each sub-model, height of the output feature map of each sub-model, width of the output feature map of each sub-model, or FLOPs of each sub-model, etc.

[0090] Sub-thread 1 can obtain the memory required for the operation of each sub-model included in the target model and the hierarchical relationship of each sub-model in the target model from a folder or database storing information of the target model based on the identifier and storage address of the target model included in the calling instruction to the target model from the target application.

[0091] In some implementations, multiple computing units may maintain their own operating conditions, which may include one or more of the following: total available memory of the computing unit, real-time remaining available memory, and / or real-time used memory.

[0092] Optionally, the remaining available memory of each computing unit available for the target application may be related to the priority of the target application. The remaining available memory of each computing unit available for the target application may account for a certain proportion of the real-time remaining available memory of each computing unit, and this proportion may be w, and the value range of w may be (0,100%), but this application does not make specific limitations on this.

[0093] Exemplarily, when the priority of the target application is the highest priority among all applications running in the electronic device, the remaining memory available to the target application in each computing unit may be the entire remaining available memory of each computing unit in real time. At this time, w can take a value of 100%. In this way, the running requirements of the target application can be guaranteed to the greatest extent without affecting other running applications. When the priority of the target application is not the highest priority among all applications running in the electronic device, the value range of w can be in the interval of (0,100%). In this way, memory space can be reserved for other applications with a higher priority than the target application, which is beneficial to guarantee the running requirements of other applications with a higher priority than the target application and to avoid the electronic device from freezing when running other applications with a higher priority than the target application.

[0094] Subthread 1 can obtain the remaining available memory of each computing unit based on w, the total memory of each computing unit and the real-time used memory. For example, the remaining available memory of a computing unit can satisfy: the remaining available memory of the computing unit = (the total memory of the computing unit - the real-time used memory of the computing unit) * w; or it can be obtained based on w and the real-time remaining available memory of each computing unit. For example, the remaining available memory of a computing unit can satisfy: the remaining available memory of the computing unit = the real-time remaining memory of the computing unit * w. This application does not specifically limit the method for obtaining the remaining available memory of each computing unit.

[0095] Optionally, the specific value of w may be set manually by a user or automatically by an electronic device system, and this application does not impose any specific limitation on this.

[0096] It should be understood that the “*” involved in the entire application is used to represent the multiplication operation symbol, and no further explanation will be given later.

[0097] S202. Sub-thread 1 obtains the jth sub-model in the execution order among the sub-models to be loaded based on the execution order indicated by the hierarchical relationship of the sub-models in the target model.

[0098] For example, Figure 3 Taking the execution order of sub-models indicated by the hierarchical relationship shown as an example, the execution order of the models is sub-model 1, sub-model 2, sub-model 3, sub-model 4 and sub-model 5, where the order of sub-model 4 and sub-model 5 can be parallel.

[0099] The sub-model to be loaded can be understood as including the sub-model in the target model obtained by the sub-thread 1 that has not been loaded on the computing unit. In some implementations, after the main thread loads the sub-model on the computing unit, the loaded sub-model can be marked as a loaded state (for example, it can be marked as true, false, 1, or 0, etc., which is not limited in this application), and the loaded state flag can be mapped to the sub-model flag. The sub-thread 1 can traverse the sub-model flags and determine the sub-models without the loaded state flag as the sub-models to be loaded.

[0100] In a possible implementation, let j = 1. When the sub-models to be loaded include sub-model 1, sub-model 2, sub-model 3, sub-model 4, and sub-model 5, the sub-model in the jth order of execution, that is, the sub-model in the first order of execution, is sub-model 1; when the sub-models to be loaded include sub-model 2, sub-model 3, sub-model 4, and sub-model 5, the sub-model in the first order of execution is sub-model 2; when the sub-models to be loaded include sub-model 4 and sub-model 5, the sub-models in the first order of execution include sub-model 4 and sub-model 5, and sub-model 4 can be called a sibling branch of sub-model 5, or sub-model 5 can be called a sibling branch of sub-model 4.

[0101] S203, sub-thread 1 determines whether the j-th sub-model has a branch at the same level.

[0102] Sub-thread 1 executes S204-S209 for the case where there is no branch at the same level and for one of the branches when there is a branch at the same level; in addition, when there is a branch at the same level, sub-thread 1 executes S210-S211.

[0103] S204. Sub-thread 1 determines whether the sub-model to be loaded satisfies: the sum of the memory required for the j sub-models to be loaded in the order of execution from the 1st to the jth is less than or equal to the remaining available memory of the ith computing unit, and the sum of the memory required for the j+1 sub-models to be loaded in the order of execution from the 1st to the j+1th is greater than the remaining available memory of the ith computing unit; if so, execute S205; if not, set j=j+1 and return to execute S202~S204 until the sub-model to be loaded satisfies the condition in S204, and then execute S205.

[0104] S205. Sub-thread 1 takes j sub-models in the sub-models to be loaded whose execution order is from the 1st to the jth as the sub-model group that the i-th computing unit is expected to undertake.

[0105] Below, a specific example is used to describe the process of obtaining the sub-model group that a computing unit is expected to undertake. In this example, the electronic device may include 4 computing units, namely computing unit 1, computing unit 2, computing unit 3 and computing unit 4. The sub-models to be loaded described above may include sub-model 1, sub-model 2, sub-model 3, sub-model 4 and sub-model 5, and the execution order is sub-model 1, sub-model 2, sub-model 3, sub-model 4 and sub-model 5. It should be understood that in this example, sub-model 1 is the sub-model with the first execution order among the sub-models to be loaded.

[0106] In this example, the memory required for sub-model 1 during runtime can be 500 megabytes (MB), the memory required for sub-model 2 during runtime can be 300MB, the memory required for sub-model 3 during runtime can be 600MB, the memory required for sub-model 4 during runtime can be 300MB, and the memory required for sub-model 5 can be 300MB; the remaining available memory of computing unit 1 can be 1 gigabyte (GB, 1GB=1024MB), the remaining available memory of computing unit 2 can be 800MB, the remaining available memory of computing unit 3 can be 600MB, and the remaining available memory of computing unit 4 can be 300MB.

[0107] It can be seen that the remaining available memory of computing unit 1 is 1GB, and the memory required for sub-model 1 and sub-model 2 to run simultaneously is 500M+300M=800M, 1GB is less than 800M, so computing unit 1 can meet the simultaneous running of sub-model 1 and sub-model 2, but the memory required for sub-model 3 to run is 600MB, 800M+600MB=1400M, 1400M exceeds the remaining available memory of computing unit 1. The sum of the memory required for sub-model 1 and sub-model 2 in the execution order of the sub-models to be loaded is less than the remaining available memory of computing unit 1, and the sum of the memory required for sub-model 1, sub-model 2, and sub-model 3 in the execution order of the sub-models to be loaded is greater than the remaining available memory of computing unit 1, that is, the sub-model group that computing unit 1 is expected to bear can include sub-model 1 and sub-model 2, but not sub-model 3.

[0108] By the same token, the sub-model group that computing unit 2 is expected to undertake may include sub-model 1 and sub-model 2; the sub-model group that computing unit 3 is expected to undertake may include sub-model 1; computing unit 4 cannot undertake the operation of sub-model 1, that is, in this link, the model can be deployed in computing unit 4.

[0109] It is worth noting that the sub-model group that each computing unit is expected to bear is not fixed. With the change of the sub-models to be loaded and the change of the remaining available memory of each computing unit, the sub-model group that each computing unit is expected to bear can change. For example, after the main thread of the target application deploys sub-model 1 and sub-model 2 on computing unit 1, the sub-models to be loaded are sub-model 3, sub-model 4 and sub-model 5, and the remaining available memory of each computing unit may also change at this time, so the sub-model group that each computing unit is expected to bear may also change.

[0110] S206. Sub-thread 1 obtains the real-time frequency of the ith computing unit, and based on mapping relationship 1 and the real-time frequency of the ith computing unit, obtains the inference time of the sub-model group that the ith computing unit is expected to bear.

[0111] Optionally, sub-thread 1 may read the real-time frequency of the ith computing unit from the ith computing unit or the driver of the ith computing unit, which is not specifically limited in the present application.

[0112] In some implementations, in the target model, for a sub-model, there may be two sub-models with the same functions but different bit widths. For example, for sub-model 1, there may be an 8-bit-width sub-model 11 and a 16-bit-width sub-model 12. Sub-model 11 and sub-model 12 have the same functions and in one call to the target model, sub-model 11 and sub-model 12 will not be loaded at the same time.

[0113] It should be understood that the bit width of a model refers to the number of binary bits used to represent or process data in the model. The bit width can determine the accuracy of the model. For example, in a neural network, a model with the same function can provide higher accuracy to capture complex nonlinear relationships than an 8-bit model with a 16-bit bit width. It should also be understood that a higher bit width requires higher computing power and more memory.

[0114] In an embodiment of the present application, when it is determined that the sub-model group that the computing unit 1 is expected to undertake may include sub-model 1 and sub-model 2, whether the loading is sub-model 11 or sub-model 12 can be determined based on the real-time frequency of the computing unit 1. For example, when the real-time frequency of the computing unit 1 is less than or equal to 500Mhz, the sub-model 11 with an 8-bit width is loaded, and when the real-time frequency of the computing unit 1 is greater than or equal to 500Mhz, the sub-model 12 with a 16-bit width is loaded. The present application does not make any specific limitations on this.

[0115] After determining the bit width of the sub-model to be loaded by the real-time frequency of the ith computing unit based on the real-time frequency of the ith computing unit, sub-thread 1 can use mapping relationship 1 to obtain the inference time of the sub-model group that the ith computing unit is expected to bear.

[0116] In a possible implementation, mapping relationship 1 may include the corresponding relationship between the identifier of each computing unit, the identifier of the sub-model (or the bit width information of the sub-model), the frequency, and the inference time, which is used to represent the inference time of each computing unit using models with different bit widths to perform inference at any frequency. For example, it includes the inference time of computing unit 1 using sub-model 11 with 8 bit width to perform inference at any frequency, and the inference time of computing unit 1 using sub-model 12 with 16 bit width to perform inference at any frequency.

[0117] Optionally, the mapping relationship 1 may be obtained by R&D personnel testing and recording each sub-model using each computing unit at any frequency, but the present application does not limit this.

[0118] Subthread 1 can uniquely determine the inference time of computing unit 1 using submodel 11 with 8-bit width at this frequency based on the identifier of computing unit 1, the identifier of submodel 11 (or the identifier of submodel 1 and bit width information), and the real-time frequency of computing unit 1. The inference time of computing unit 1 using submodel 11 with 8-bit width at this frequency can be considered as the inference time of computing unit 1 using submodel 1 at this frequency. Similarly, the inference time 21 of computing unit 1 using submodel 2 at this frequency can be obtained.

[0119] The inference time of the sub-model group that the computing unit 1 is expected to bear may be the sum of the inference time 1 and the inference time 2 .

[0120] S207 , obtaining the total number of floating-point operations of the sub-model group that the i-th computing unit is expected to undertake.

[0121] Optionally, the total number of floating-point operation of the sub-model group that the i-th computing unit is expected to bear may be the sum of the total number of floating-point operation of each sub-model in the sub-model group that the i-th computing unit is expected to bear.

[0122] In some implementations, subthread 1 can directly obtain the total number of floating-point operations FLOPs of each submodel from the folder storing the information of the target model, or can also obtain the convolution kernel size K, the number of channels C1 of the input feature map, the number of channels C2 of the output feature map, the height H of the output feature map, and the width W of the output feature map from the folder storing the information of the target model, and use FLOPs=(2*C1*K 2 -1) *H*W*C2 to get the FLOPs of each sub-model.

[0123] Still taking the i-th computing unit as computing unit 1 as an example, the total number of floating-point computing operations of the sub-model group that computing unit 1 is expected to undertake may be the sum of the total number of floating-point computing operations of sub-model 1 and the total number of floating-point computing operations of sub-model 2.

[0124] S208. Obtain the inference efficiency of the ith computing unit based on the total number of floating-point operations and inference time of the sub-model group that the ith computing unit is expected to undertake.

[0125] Optionally, the inference efficiency of the ith computing unit may be the quotient of the total number of floating-point operations of the sub-model group that the ith computing unit is expected to undertake and the inference time, but this application is not limited to this.

[0126] At this point, taking computing unit 1 as an example, the inference efficiency of computing unit 1 when running the sub-model group that it is expected to undertake can be obtained.

[0127] Furthermore, in an embodiment of the present application, sub-thread 1 can let i=i+1 to traverse each computing unit based on the number or identification of the computing unit, calculate and obtain the inference efficiency of each computing unit when running the sub-model group it is expected to undertake based on the above S204~S208, and then execute S209.

[0128] S209. Sub-thread 1 sends the inference efficiency of each computing unit and its sub-model group to the main thread.

[0129] The information sent by sub-thread 1 to the main thread can be called model deployment reference information, for example. The model deployment reference information may include: the identification of each computing unit, the identification of each sub-model in the sub-model group that each computing unit corresponds to and is expected to undertake, and the inference efficiency of each computing unit when running the sub-model group that it is expected to undertake.

[0130] In some implementations, the information sent by sub-thread 1 to the main thread can be presented in a list or any other form that can express the relationship between the computing unit, the sub-model group and the reasoning efficiency, which is not limited in this application.

[0131] Optionally, sub-thread 1 may send model deployment reference information of a computing unit to the main thread each time the inference efficiency of the computing unit and its sub-model group are obtained, or it may send model deployment reference information to the main thread after accumulating the inference efficiencies of several or traversing all computing units and their sub-model groups. The present application does not make any specific limitation on the timing of sending this information.

[0132] In one possible implementation, when sub-thread 1 executes the judgment in S203 above and learns that the j-th sub-model does not have any branches at the same level, that is, the main thread has not received the branch quantity information sent by sub-thread 1, then after receiving the information described in S209 above sent by sub-thread 1, the main thread can load (or deploy) the sub-model group corresponding to the computing unit with the highest inference efficiency and run it on the computing unit.

[0133] In another possible implementation, when the sub-thread 1 executes the above S203 and learns that the j-th sub-model has a same-level branch, the sub-thread 1 can execute S210-S211.

[0134] S210. Sub-thread 1 obtains the number X of branches of the sub-model whose execution order is the jth in the sub-model to be loaded.

[0135] For example, Figure 3 Taking the hierarchical relationship shown as an example, when the j-th sub-model in the execution order among the sub-models to be loaded is sub-model 4 or sub-model 5, the j-th sub-model has a branch at the same level.

[0136] Optionally, sub-thread 1 obtains the number of branches by determining the number of sub-models at the jth position in the execution order in the sub-model to be loaded. For example, when the sub-model at the jth position in the execution order in the sub-model to be loaded is sub-model 4 or sub-model 5, the number of sub-models at the jth position in the execution order in the sub-model to be loaded is 2, that is, the number of branches can be obtained as 2.

[0137] In this case, child thread 1 can continue to execute steps S204~S209 based on one of the branches 1, and child thread 1 executes S211, sends the number of branches X to the main thread, to instruct the main thread to execute S212 to create X-1 child threads, so as to use the created X-1 child threads to execute steps similar to S204~S209 for the X-1 branches except branch 1. In this way, the deployment planning of each branch thread can be carried out synchronously, which is beneficial for the sub-models on each branch to be loaded and inferred synchronously at the same time, and is beneficial for improving the model inference efficiency.

[0138] S211~S212 include:

[0139] S211. Sub-thread 1 sends branch number X to the main thread, and correspondingly, the main thread receives branch number X.

[0140] Optionally, the information sent by sub-thread 1 to the main thread in S211 may also include identifiers and hierarchical relationships of sub-models in X-1 branches except branch 1.

[0141] S212: The main thread creates X-1 child threads, and uses the X-1 child threads to execute steps similar to S204 to S209 for the X-1 branches except branch 1.

[0142] S213. X-1 sub-threads respectively send the inference efficiency and sub-model groups of each computing unit for different branches to the main thread. Correspondingly, the main thread receives the inference efficiency and sub-model groups of each computing unit for different branches from each sub-thread.

[0143] Furthermore, based on the model deployment reference information from each sub-thread, the main thread obtains a ranked list of computing units corresponding to each sub-thread based on reasoning efficiency, obtains the computing unit with the highest reasoning efficiency in each ranked list and the sub-model group that the computing unit corresponds to and is expected to undertake.

[0144] In one implementation, when the computing units with the highest inference efficiency obtained by each sub-thread are different, S214 is executed.

[0145] S214, loading the sub-model group that the computing unit with the highest inference efficiency obtained by each sub-thread is expected to bear on each computing unit.

[0146] In another implementation, if the computing units with the highest inference efficiency obtained by each sub-thread are the same, then the main thread can select computing units one by one in order of inference efficiency from high to low based on the sorted list, until the computing units selected from each list are different, and then load the sub-model group that the computing unit is expected to undertake on the computing unit.

[0147] For example, based on the model deployment reference information from sub-thread 1, the main thread obtains the order of the computing units based on reasoning efficiency corresponding to sub-thread 1 from high to low: computing unit 1, computing unit 3, computing unit 2; based on the model deployment reference information from sub-thread 2, the order of the computing units based on reasoning efficiency corresponding to sub-thread 2 from high to low: computing unit 1, computing unit 2, computing unit 3. In order to avoid deploying the sub-model group that computing unit 1 is expected to undertake obtained by sub-thread 1 and the sub-model group that computing unit 1 is expected to undertake obtained by sub-thread 2 to computing unit 1 (where the sub-model group that computing unit 1 is expected to undertake obtained by sub-thread 1 and the sub-model group that computing unit 1 is expected to undertake obtained by sub-thread 2 belong to different branches and are different sub-model groups), which causes resource preemption and affects the model operation efficiency. Therefore, main thread 1 can choose any computing unit whose reasoning efficiency is lower than that of computing unit 1 from the computing unit ranking based on reasoning efficiency corresponding to child thread 1, such as computing unit 3, and deploy the sub-model group it is expected to undertake on computing unit 3. This is beneficial to the synchronous operation of sub-models in each branch of the same level, which is beneficial to improving the model operation efficiency and saving model reasoning time.

[0148] Optionally, when the computing units with the highest reasoning efficiency obtained by the various sub-threads are the same, the main thread can randomly select a sub-thread and make a secondary selection of the computing unit based on the result of the sub-thread. In addition, the main thread can also randomly select the computing unit with the second highest reasoning efficiency from the ranking of the computing units corresponding to the sub-thread. The present application does not make any specific limitations on this.

[0149] Furthermore, when the main thread selects a computing unit with the second highest reasoning efficiency from the ranking of computing units corresponding to the sub-threads, the computing unit with the second highest reasoning efficiency can be selected as the computing unit that needs to load the sub-model. This is helpful in ensuring the operating efficiency of the branch while avoiding resource preemption.

[0150] It is worth noting that after the main thread completes one loading, if the target model has not been fully loaded, the other X-1 sub-threads except sub-thread 1 can be destroyed to reduce the running burden of the target application, and sub-thread 1 can be used to continue to execute S202~S209 for the remaining sub-models to be loaded, or based on the branch steps after S203, to continue to deploy the remaining sub-models to be loaded, until all the sub-models included in the target model are deployed, and the main thread can destroy sub-thread 1.

[0151] Combine the following Figure 4A model deployment method 400 provided in an embodiment of the present application is described in detail. The method 400 can be applied to an electronic device, the electronic device includes a first computing unit and a target model, and the target model includes multiple sub-models. The software structure of the electronic device can be as follows Figure 2 As shown, the hardware structure of the electronic device can be as follows Figure 5 As shown, this application is not specifically limited to this.

[0152] The method 400 comprises the following steps:

[0153] S401. At a first moment, P sub-models in a target model are loaded on a first computing unit, the remaining available memory of the first computing unit is a first memory, the memory occupied by the P sub-models in the first computing unit is a second memory, and the second memory is less than or equal to the first memory.

[0154] S402. At the second moment, Q sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the third memory, the memory occupied by the Q sub-models in the first computing unit is the fourth memory, and the third memory is less than or equal to the fourth memory; wherein, the first memory is different from the third memory, and the second memory is different from the fourth memory.

[0155] In an embodiment of the present application, the memory occupied by the sub-model loaded on the first computing unit is less than or equal to the remaining available memory of the first computing unit. This is conducive to achieving reasonable allocation of resources and avoiding the problem of resource preemption caused by the memory occupied by the sub-models loaded on the first computing unit exceeding the remaining available memory of the first computing unit. In other words, it is conducive to avoiding the problem of electronic device freezing or application crash, thereby improving user experience.

[0156] It should be understood that the application scenario of the embodiments of the present application may be a scenario where it is impossible to load all the sub-models included in the target model on the first computing unit at one time. In this scenario, the electronic device can load a part of the sub-models in the target model on the first electronic device in response to the call to the target model, such as the P sub-models and Q sub-models mentioned above. This part of the sub-models loaded on the first computing unit can be called a sub-model group.

[0157] Optionally, the sub-model can be understood as the smallest unit that can be divided in the target model, for example, it can be a convolutional layer, but this application does not make specific limitations on this.

[0158] In a possible implementation, the P sub-models loaded on the first computing unit are selected in accordance with the first memory and the memory required by the P sub-models; the Q sub-models loaded on the first computing unit are selected in accordance with the third memory and the memory required by the Q sub-models. The memory required by the P sub-models can be understood as the running memory theoretically required by each of the P sub-models obtained during the training and testing process, and the first memory can be understood as the memory actually occupied by the P sub-models when running on the first computing unit.

[0159] In some implementations, the first moment and the second moment can be understood as different moments in the process of two independent calls of the target model by the electronic device, and each call can obtain an inference result. In these two calls, the first computing unit can be used. Since the tasks carried by the first computing unit may be different, the remaining available memory of the first computing unit may be different, and based on the difference in the remaining available memory of the first computing unit, the memory occupied by the sub-model loaded on the first computing unit on the first computing unit may also be variable. In this scenario, compared with the first moment and the second moment, the sub-model loaded on the first computing unit may be the same or different. In this way, the flexibility of model deployment is higher.

[0160] In some other implementations, the first moment and the second moment can be understood as two different moments in a call of the target model by the electronic device, that is, in the process of the electronic device using the target model for reasoning, different sub-models of the target model can be loaded on the first computing unit at different moments (for example, P sub-models of the target model can be loaded at the first moment, released after the operation is completed, and Q sub-models of the target model can be loaded at the second moment, and the P sub-models and the Q sub-models are different sub-models). The remaining available memory of the first computing unit corresponding to these two loadings can be different, and based on the difference in the remaining available memory of the first computing unit, the memory occupied by the sub-models loaded on the first computing unit can also be variable. In this way, the model deployment is more flexible.

[0161] Optionally, the meaning and acquisition method of the remaining available memory of the first computing unit here may be the same as the meaning and acquisition method of the remaining available memory of each computing unit described in the above method 200, and will not be repeated here.

[0162] As an optional embodiment, the electronic device also includes a second computing unit, and the method also includes: loading R sub-models on the second computing unit, the execution order of the R sub-models in the target model is after the P sub-models, and when the R sub-models are loaded on the second computing unit, the status of the P sub-models on the first computing unit is any one of loaded and not running, running, releasing, or released.

[0163] In an embodiment of the present application, after loading P sub-models in the target model on the first computing unit, R sub-models in the target model whose execution order is after the P sub-models can be loaded onto the second computing unit. In this way, after the first computing unit obtains the inference result based on the P sub-models, it can continue to use the inference result and the R sub-models loaded on the second computing unit for inference without waiting for the release time of the P sub-models and the loading time of the R sub-models, which is conducive to shortening the overall inference time of the target model, improving the inference efficiency of the target model, and providing users with a better user experience.

[0164] As an optional embodiment, before loading the P sub-models in the target model on the first computing unit, the method also includes: obtaining the remaining available memory of the first computing unit, the sub-models to be loaded, the memory required by each of the sub-models to be loaded, and the execution order of each sub-model in the sub-models to be loaded, wherein the sub-models to be loaded are part or all of multiple sub-models; starting from the sub-model with the first execution order in the sub-models to be loaded, traversing according to the execution order of the sub-models to be loaded, accumulating the memory required by each traversed sub-model to obtain a first accumulated memory and a second accumulated memory, the number of sub-models corresponding to the second accumulated memory is one more than the number of sub-models corresponding to the first accumulated memory; when the first accumulated memory is less than or equal to the remaining available memory of the first computing unit, and the second accumulated memory is greater than the remaining available memory of the first computing unit, the sub-models corresponding to the first accumulated memory are determined to be P sub-models.

[0165] This helps to ensure that the memory occupied by the P sub-models in the target model loaded on the first computing unit is less than or equal to the remaining available memory of the first computing unit, which helps to avoid resource preemption, that is, it helps to avoid the problem of electronic device freezing or application crashes, thereby improving user experience.

[0166] Optionally, the specific method of obtaining the sub-model to be loaded may be similar to the method of determining the sub-model to be loaded in the above method 200, and the specific process of determining the P sub-models loaded on the first computing unit may be similar to the process of determining the sub-model group that the i-th computing unit is expected to undertake described in the above method 200, which will not be repeated here.

[0167] As an optional embodiment, the electronic device includes multiple computing units, the multiple computing units include a first computing unit, and P sub-models are sub-models that the first computing unit is expected to bear. The method also includes: determining the sub-models that each of the multiple computing units other than the first computing unit is expected to bear; calculating the efficiency of each of the multiple computing units in running the sub-model that is expected to be undertaken; loading the P sub-models of the target model on the first computing unit, including: when the efficiency of the first computing unit running the P sub-models is the maximum value of the efficiency of each of the multiple computing units in running the sub-model that is expected to be undertaken, loading the P sub-models of the target model on the first computing unit.

[0168] In the embodiment of the present application, the efficiency of each computing unit in the multiple computing units in running the sub-models that are expected to be undertaken by each computing unit can be used to characterize the speed of the multiple computing units in running the sub-models that are expected to be undertaken by each computing unit. Among the efficiencies of each computing unit in the multiple computing units in running the sub-models that are expected to be undertaken by each computing unit, the computing unit with the highest efficiency is selected, and the sub-model that is expected to be undertaken is loaded on the computing unit with the highest efficiency. In this way, it is beneficial to load the sub-model to be loaded onto the computing unit with the highest processing capability on the basis of avoiding resource preemption, which is beneficial to improve the reasoning efficiency of the target model.

[0169] As an optional embodiment, the efficiency of the first computing unit running P sub-models is proportional to the sum of the total number of floating-point operations of each of the P sub-models, and / or inversely proportional to the sum of the theoretical time for the first computing unit to run each of the P sub-models; wherein, the theoretical time for the first computing unit to run each of the P sub-models is obtained through experiments, and the theoretical time for the first computing unit to run each of the P sub-models is related to the frequency of the first computing unit.

[0170] Optionally, the theoretical time for the first computing unit to run each of the P sub-models can be obtained based on the mapping relationship 1 described in method 200, and the calculation method of the efficiency of the first computing unit running the P sub-models can be similar to the calculation method of the inference efficiency of the i-th computing unit in the above method 200, which is not repeated here. The "efficiency" here can be understood to have the same meaning as the "inference efficiency" mentioned above.

[0171] As an optional embodiment, the electronic device includes a main thread and at least one sub-thread, the at least one sub-thread includes a first sub-thread; calculating the efficiency of each computing unit in a plurality of computing units in running a respective sub-model that is expected to be undertaken, including: the first sub-thread calculates the efficiency of each computing unit in a plurality of computing units in running a respective sub-model that is expected to be undertaken, and sends model deployment reference information including the efficiency of each computing unit in a plurality of computing units in running a respective sub-model that is expected to be undertaken to the main thread; the main thread loads P sub-models in the target model on the first computing unit when determining that there is no model deployment reference information from other sub-threads in at least one sub-thread except the first sub-thread, including: when the efficiency of the first computing unit in running the P sub-models is the maximum value of the efficiency of each computing unit in a plurality of computing units in running a respective sub-model that is expected to be undertaken, the main thread loads the P sub-models in the target model on the first computing unit.

[0172] Optionally, the first sub-thread here may have the same meaning as the sub-thread 1 described in the above method 200, the main thread may have the same meaning as the main thread in the above method 200, and the model deployment reference information received by the main thread may have the same meaning as the model deployment reference information described in method 200, but this application does not make any specific limitations on this.

[0173] It is worth noting that the main thread determines that there is no model deployment reference information from at least one sub-thread other than the first sub-thread. It can be understood that the first sub-thread did not traverse the sub-model with the same-level branches in the process of calculating the efficiency of each computing unit in multiple computing units running the sub-model that each computing unit is expected to undertake.

[0174] As an optional embodiment, when determining that there is model deployment reference information from other sub-threads other than the first sub-thread in at least one sub-thread, the main thread obtains the most efficient computing unit corresponding to each model deployment reference information in other sub-threads and the sub-model that the most efficient computing unit is expected to undertake; and loads P sub-models in the target model on the first computing unit, including: when the efficiency of the first computing unit obtained by the first sub-thread to run the P sub-models is greater than the maximum value of the efficiency of each computing unit in the other computing units to run the sub-models that are expected to be undertaken, and the most efficient computing unit corresponding to each model deployment reference information in other sub-threads is not the first computing unit, the main thread loads the P sub-models in the target model on the first computing unit.

[0175] It should be understood that the main thread determines that there is or has received model deployment reference information from at least one sub-thread other than the first sub-thread, that is, the first sub-thread traverses to the sub-model with the same-level branch in the process of calculating the efficiency of each computing unit in the multiple computing units running the sub-model that each is expected to bear, and the model deployment reference information of the sub-threads other than the first sub-thread is related to the branch of the sub-model. In this case, the sub-threads other than the first sub-thread can be sub-thread 2 to sub-thread X-1 in the above method 200, and the creation process of the sub-threads other than the first sub-thread, and the acquisition process of the model deployment reference information of the sub-threads other than the first sub-thread can refer to the steps S210 to S213 in the above method 200 and the related extended description, which will not be repeated here.

[0176] Since the process of each sub-thread obtaining its model deployment reference information can be independent of each other, there is a possibility that the most efficient computing unit corresponding to the model deployment reference information obtained by each sub-thread is the same. In this case, if the sub-model that is expected to be undertaken is loaded directly on the most efficient computing unit, the loading may fail due to insufficient memory. Therefore, in an embodiment of the present application, when the most efficient computing unit corresponding to each model deployment reference information in other sub-threads is not the first computing unit, loading P sub-models in the target model on the first computing unit is beneficial to the successful loading of the P sub-models and to maintaining the stability of the target model operation.

[0177] Furthermore, when the efficiency of the first computing unit obtained by the first sub-thread in running P sub-models is greater than the maximum value of the efficiency of each computing unit in other computing units in running the sub-models that they are expected to undertake, and the computing unit with the highest efficiency corresponding to each model deployment reference information in other sub-threads is the same as the first computing unit, the solution can refer to the expanded description at S214 in the above method 200, and will not be repeated here.

[0178] As an optional embodiment, the method further includes: the main thread loads the sub-model that is expected to be undertaken by the most efficient computing unit on the most efficient computing unit corresponding to each model deployment reference information in other sub-threads.

[0179] In this way, sub-models on different branches in the same-level branches can be loaded onto different computing units, which is conducive to improving model operation efficiency and saving model inference time.

[0180] Optionally, the main thread can load P sub-models in the target model on the first computing unit, while loading the sub-model that the most efficient computing unit is expected to undertake on the most efficient computing unit corresponding to each model deployment reference information in other sub-threads. This is conducive to synchronous reasoning of sub-models in branches at the same level, which is conducive to improving model operation efficiency and saving model reasoning time.

[0181] As an optional embodiment, the embodiment of the present application may also include the following scenarios: at a third moment, T sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the fifth memory, the memory occupied by the T sub-models in the first computing unit is the sixth memory, and the sixth memory is less than or equal to the fifth memory; at a fourth moment, U sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the seventh memory, the memory occupied by the U sub-models in the first computing unit is the eighth memory, and the eighth memory is less than or equal to the seventh memory; wherein, the fifth memory is different from the seventh memory, and the sixth memory is the same as the eighth memory.

[0182] As an optional embodiment, the embodiment of the present application may also include the following scenarios: at a fifth moment, V sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the ninth memory, the memory occupied by the V sub-models in the first computing unit is the tenth memory, and the tenth memory is less than or equal to the ninth memory; at a sixth moment, Y sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the eleventh memory, the memory occupied by the Y sub-models in the first computing unit is the twelfth memory, and the twelfth memory is less than or equal to the eleventh memory; wherein, the ninth memory is the same as the eleventh memory, and the tenth memory is different from the twelfth memory.

[0183] As an optional embodiment, the embodiment of the present application may also include the following scenarios: at the seventh moment, Z sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the thirteenth memory, the memory occupied by the Z sub-models in the first computing unit is the fourteenth memory, and the fourteenth memory is less than or equal to the thirteenth memory; at the eighth moment, K sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the fifteenth memory, the memory occupied by the K sub-models in the first computing unit is the sixteenth memory, and the sixteenth memory is less than or equal to the fifteenth memory; wherein, the thirteenth memory is the same as the fifteenth memory, and the fourteenth memory is the same as the sixteenth memory.

[0184] Optionally, the meanings of the third moment and the fourth moment, the fifth moment and the sixth moment, and the seventh moment and the eighth moment may be similar to the meanings of the first moment and the second moment, but the present application does not make specific limitations on this.

[0185] In the embodiment of the present application, letters such as P, Q, R, S, T, U, V, Y, Z, and K are used to identify the number of sub-models, and their values ​​can all be integers greater than zero.

[0186] The model deployment method of the embodiment of the present application has been described above, and the device for executing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above method.

[0187] Figure 5 FIG. 1 is a schematic diagram showing the hardware structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 500 may include a processor 510 , a memory 520 , a display screen 530 , a sensor module 540 , a universal serial bus (USB) interface 550 , a charging management module 560 , a power management module 561 , a battery 562 , and the like.

[0188] The processor 510 may include one or more processing units, for example, the processor 510 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. The processing unit here can be understood as the computing unit described above.

[0189] The memory 520 may be used to store computer executable program codes, where the executable program codes include instructions, such as the program codes of the embodiments of the present application.

[0190] The display screen 530 is used to display images, videos, etc. The display screen 530 includes a display panel. In some embodiments, the electronic device may include 1 or G display screens 530, where G is a positive integer greater than 1. The electronic device implements the display function through a GPU, a display screen 530, and an application processor.

[0191] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.

[0192] The model deployment method provided in the embodiment of the present application can be applied to an electronic device including a model, and the software structure of the electronic device can be as follows: Figure 1 As shown in the figure, the hardware structure can be as follows Figure 5 As shown in , the specific form of the electronic device can refer to the above description and will not be repeated here.

[0193] Figure 6 The schematic diagram of the structure of a chip provided by the embodiment is exemplarily shown. The chip 600 includes one or more (including two) processors 601 , a communication line 602 , a communication interface 603 and a memory 604 .

[0194] In some implementations, the memory 604 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0195] The method described in the above embodiment of the present application can be applied to the processor 601, or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 601 or an instruction in the form of software. The above processor 601 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 601 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0196] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 604, and the processor 601 reads the information in the memory 604 and completes the steps of the above method in combination with its hardware.

[0197] The processor 601 , the memory 604 , and the communication interface 603 may communicate with each other via the communication line 602 .

[0198] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0199] In the embodiment of the present application, the chip 600 may also be a chip system, such as a system on chip (SOC), which is not limited in the present application.

[0200] An embodiment of the present application provides an electronic device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory stores computer program code, and the computer program code includes computer instructions; the one or more processors call the computer instructions to enable the electronic device to execute the method in the above embodiment.

[0201] The embodiment of the present application provides a chip or a chip system. The chip or chip system is applied to an electronic device, and the chip or chip system includes one or more processors, and the one or more processors are used to call computer instructions to enable the electronic device to execute the method in the above embodiment. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, and will not be repeated here.

[0202] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are run on an electronic device, the electronic device executes the method in the above embodiment. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0203] In one possible implementation, a computer readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile discs (DVD), floppy disks and Blu-ray discs, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer readable media.

[0204] An embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on an electronic device, the electronic device executes the method in the above embodiment.

[0205] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0206] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A model deployment method, characterized in that: Applied to an electronic device, the electronic device includes a first computing unit and a target model, the target model includes a plurality of sub-models, and the method includes: At a first moment, P sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the first memory, the memory occupied by the P sub-models in the first computing unit is the second memory, and the second memory is less than or equal to the first memory; At the second moment, Q sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the third memory, the memory occupied by the Q sub-models in the first computing unit is the fourth memory, and the fourth memory is less than or equal to the third memory; Wherein, the first memory is different from the third memory, and the second memory is different from the fourth memory; The electronic device includes a plurality of computing units, the plurality of computing units include the first computing unit, the P sub-models are sub-models that the first computing unit is expected to bear, and the method further includes: Determine the sub-models that each of the other computing units except the first computing unit among the multiple computing units is expected to undertake; calculate the efficiency of each computing unit among the multiple computing units in running the sub-model that each computing unit is expected to undertake; Loading P sub-models in the target model on the first computing unit includes: The P sub-models in the target model are loaded on the first computing unit when the efficiency of the first computing unit in running the P sub-models is the maximum value of the efficiency of each computing unit in the multiple computing units in running the respective sub-models that are expected to be undertaken; the efficiency of the first computing unit in running the P sub-models is proportional to the sum of the total number of floating-point operations of the P sub-models, and / or inversely proportional to the sum of the theoretical time for the first computing unit to run each of the P sub-models; wherein the theoretical time for the first computing unit to run each of the P sub-models is obtained through experiments, and the theoretical time for the first computing unit to run each of the P sub-models is related to the frequency of the first computing unit.

2. The method according to claim 1, characterized in that: The electronic device further includes a second computing unit, and the method further includes: R sub-models are loaded on the second computing unit, and the execution order of the R sub-models in the target model is after the P sub-models. When the R sub-models are loaded on the second computing unit, the status of the P sub-models on the first computing unit is any one of loaded and not running, running, releasing, or released.

3. The method according to claim 1, characterized in that Before loading the P sub-models in the target model on the first computing unit, the method further includes: Obtaining the remaining available memory of the first computing unit, the sub-models to be loaded, the memory required by each of the sub-models to be loaded, and the execution order of each sub-model in the sub-models to be loaded, wherein the sub-models to be loaded are part or all of the multiple sub-models; Starting from the sub-model with the first execution order among the sub-models to be loaded, traversing in the execution order of the sub-models to be loaded, accumulating the memory required by each traversed sub-model to obtain a first accumulated memory and a second accumulated memory, wherein the number of sub-models corresponding to the second accumulated memory is one more than the number of sub-models corresponding to the first accumulated memory; When the first accumulated memory is less than or equal to the remaining available memory of the first computing unit, and the second accumulated memory is greater than the remaining available memory of the first computing unit, the sub-model corresponding to the first accumulated memory is determined as the P sub-models.

4. The method according to claim 1 or 3, characterized in that: The electronic device comprises a main thread and at least one sub-thread, wherein the at least one sub-thread comprises a first sub-thread; The calculating of the efficiency of each computing unit in the plurality of computing units in running the sub-model that is estimated to be undertaken by each computing unit includes: The first sub-thread calculates the efficiency of each of the multiple computing units running the sub-model that is expected to be undertaken by each computing unit, and sends the model deployment reference information including the efficiency of each of the multiple computing units running the sub-model that is expected to be undertaken by each computing unit to the main thread; When the main thread determines that there is no model deployment reference information from other sub-threads except the first sub-thread among the at least one sub-thread, the loading of the P sub-models in the target model on the first computing unit includes: When the efficiency of the first computing unit in running the P sub-models is the maximum value of the efficiency of each computing unit in the multiple computing units in running the sub-model that it is expected to undertake, the main thread loads the P sub-models in the target model on the first computing unit.

5. The method according to claim 4, characterized in that The main thread, when determining that there is model deployment reference information from other sub-threads except the first sub-thread among the at least one sub-thread, obtains the most efficient computing unit corresponding to each model deployment reference information in other sub-threads and the sub-model that the most efficient computing unit is expected to undertake; The step of loading the P sub-models in the target model on the first computing unit includes: When the efficiency of the first computing unit obtained by the first sub-thread in running the P sub-models is greater than the maximum value of the efficiencies of each of the other computing units in running the sub-models that they are expected to undertake, and the computing unit with the highest efficiency corresponding to each model deployment reference information in the other sub-threads is not the first computing unit, the main thread loads the P sub-models in the target model on the first computing unit.

6. The method according to claim 5, characterized in that The method further comprises: The main thread loads the sub-model that the most efficient computing unit is expected to undertake on the most efficient computing unit corresponding to each model deployment reference information in the other sub-threads.

7. The method according to claim 1, characterized in that The method comprises: At a third moment, T sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is a fifth memory, the memory occupied by the T sub-models in the first computing unit is a sixth memory, and the sixth memory is less than or equal to the fifth memory; At a fourth moment, U sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the seventh memory, the memory occupied by the U sub-models in the first computing unit is the eighth memory, and the eighth memory is less than or equal to the seventh memory; Among them, the fifth memory is different from the seventh memory, and the sixth memory is the same as the eighth memory.

8. The method according to claim 1, characterized in that: The method further comprises: At a fifth moment, V sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the ninth memory, the memory occupied by the V sub-models in the first computing unit is the tenth memory, and the tenth memory is less than or equal to the ninth memory; At the sixth moment, Y sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the eleventh memory, the memory occupied by the Y sub-models in the first computing unit is the twelfth memory, and the twelfth memory is less than or equal to the eleventh memory; Among them, the ninth memory is the same as the eleventh memory, and the tenth memory is different from the twelfth memory.

9. The method according to claim 1, characterized in that: The method further comprises: At the seventh moment, Z sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the thirteenth memory, the memory occupied by the Z sub-models in the first computing unit is the fourteenth memory, and the fourteenth memory is less than or equal to the thirteenth memory; At an eighth moment, K sub-models in the target model are loaded on the first computing unit, the remaining available memory of the first computing unit is the fifteenth memory, the memory occupied by the K sub-models in the first computing unit is the sixteenth memory, and the sixteenth memory is less than or equal to the fifteenth memory; Among them, the thirteenth memory is the same as the fifteenth memory, and the fourteenth memory is the same as the sixteenth memory.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 9.

11. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device is caused to perform the method as claimed in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 9.

Images