Method, device and medium for performing distributed reasoning based on model parallelism

Abstract

The present disclosure relates to a method, device and medium for performing distributed reasoning based on model parallelism. The method for performing distributed reasoning based on model parallelism at a mobile terminal includes: downloading and running a regression model constructed according to a current application model, the regression model is used to predict the number of inter-layer output data of the current application model and the network transmission rate required to upload the inter-layer output data to a cloud server, the inter-layer reasoning time and power consumption of the current application model; determining whether the current power of the mobile terminal is lower than a threshold power; calculating the exit point of the current application model based on the result of the determination; and transmitting the calculated exit point to the cloud server; wherein, at the exit point, the mobile terminal stops the reasoning of the current application model and pushes the obtained inter-layer output data to the cloud server for the cloud server to perform the reasoning of the remaining part of the current application model.

Description

Technical Field

[0001] The present disclosure relates generally to the field of artificial intelligence, and more particularly to methods, devices, and media for performing model-parallel based distributed reasoning. Background Art

[0002] Model parallelism is widely used in distributed technology. This article applies the model parallelism method to the inference calculation of deep neural networks on mobile terminals. At present, artificial intelligence (AI) applications on mobile terminals are mainly based on deep neural networks. The inference calculation of deep neural networks will bring computing burden and latency to mobile terminals. In this case, model parallelism can be used to deploy different layers of the model on different devices, such as deploying different layers of the deep network to different hardware computing units, which communicate with each other to realize a pipeline. In the processing based on model parallelism, the output data of the previous time is input to the next layer as input data, and all processing is completed at the last layer, and the result is returned.

[0003] In recent years, the computing resources, memory consumption, and power consumption of mobile applications based on AI / ML (Machine Learning) models have become increasingly high. However, mobile terminals, which are limited by hardware resources, can no longer meet the latency and power consumption required to support such a large amount of computing. However, if AI / ML model reasoning is offloaded from mobile terminals to the cloud for processing, it may expose users' privacy data or fail to meet the needs of applications with high real-time requirements.

[0004] Therefore, there is a need in the prior art for a technology that can perform model reasoning in a distributed manner between mobile terminals and edge / cloud servers. Summary of the invention

[0005] A brief overview of the disclosure is given below in order to provide a basic understanding of some aspects of the disclosure. However, it should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify the key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to give some concepts of the disclosure in a simplified form as a prelude to a more detailed description given later.

[0006] This patent proposes a distributed collaborative reasoning solution for terminal-edge / cloud servers based on model parallelism to solve the problems caused by pure cloud or terminal AI / ML model reasoning calculations, such as:

[0007] ① The computing power on the terminal side is limited, the power consumption is too high, and the inference time is too long;

[0008] ② Offloading to the cloud may lead to security issues such as user data exposure and fail to meet application requirements with high real-time requirements.

[0009] The terminal-side decision engine proposed by the inventor calculates the optimal exit point of each frame of the AI / ML model in real time, and splits the AI / ML model into two parts according to real-time environmental factors (such as model calculation amount, mobile terminal computing power, mobile terminal energy consumption, network environment, application delay requirements, etc.). The mobile terminal will perform model reasoning for the first part, and the edge / cloud service will perform model reasoning for the remaining part, thereby optimizing end-to-end delay and reducing the burden of model reasoning calculation on the mobile terminal.

[0010] According to one aspect of the present disclosure, a method for performing model-parallel-based distributed reasoning at a mobile terminal is provided, including: downloading and running a regression model constructed according to a current application model, the regression model being used to predict the amount of inter-layer output data of the current application model and the network transmission rate required for uploading the inter-layer output data to a cloud server, the inter-layer reasoning time and power consumption of the current application model; determining whether the current power of the mobile terminal is lower than a threshold power; calculating an exit point of the current application model based on the result of the determination, the calculation comprising: if the current power of the mobile terminal is lower than the threshold power, calculating an exit point for the next frame of the current application model based on the minimum computing power consumption of the mobile terminal, and if the current power of the mobile terminal is greater than the threshold power, calculating an exit point for the next frame of the current application model based on the latency requirement of the current application model, the required network transmission rate predicted by the regression model, the amount of inter-layer output data, and the inter-layer reasoning time and energy consumption; and transmitting the calculated exit point to the cloud server; wherein, at the exit point, the mobile terminal stops the reasoning of the current application model and pushes the obtained inter-layer output data to the cloud server for the cloud server to perform the reasoning of the remaining part of the current application model.

[0011] According to another aspect of the present disclosure, a mobile terminal is provided, comprising: a memory on which instructions are stored; and a processor configured to execute the instructions stored in the memory to perform the method according to the above aspect of the present disclosure.

[0012] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which includes computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors execute the method according to the above aspects of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which constitute a part of the specification, describe embodiments of the present disclosure and, together with the specification, are used to explain the principles of the present disclosure. With reference to the accompanying drawings, the present disclosure can be more clearly understood according to the following detailed description, in which:

[0014] Figure 1 is a schematic diagram showing an exemplary system for performing model-parallel based distributed reasoning according to an embodiment of the present disclosure.

[0015] Figure 2 Detailed description is a flowchart illustrating an exemplary method for performing model-parallel based distributed reasoning according to an embodiment of the present disclosure.

[0016] Figure 3 A histogram showing the computation time and output data volume of each layer using the AlexNet model.

[0017] Figure 4 A schematic diagram showing the energy consumption loss of a mobile terminal using the AlexNet model.

[0018] Figure 5 A schematic diagram showing the end-to-end latency of model collaborative inference using the AlexNet model.

[0019] Figure 6 An exemplary configuration of a mobile terminal that can implement distributed reasoning based on model parallelism according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following detailed description is made with reference to the accompanying drawings, and the following detailed description is provided to assist in a comprehensive understanding of the various exemplary embodiments of the present disclosure. The following description includes various details to assist in understanding, but these details are considered to be examples only and not to limit the present disclosure, which is defined by the appended claims and their equivalents. The words and phrases used in the following description are only used to enable a clear and consistent understanding of the present disclosure. In addition, for the sake of clarity and brevity, descriptions of well-known structures, functions, and configurations may be omitted. Those of ordinary skill in the art will recognize that various changes and modifications may be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0021] The present invention is suitable for applications that require heavy-duty deep learning model reasoning or high end-to-end latency in mobile terminals, such as object recognition, photo enhancement, intelligent video surveillance, AR, remote control cars, industrial control, and robotics. Through the method of the present invention, the best exit point of each frame of the model is calculated in real time, and an integrated optimal terminal-edge / cloud server distributed collaborative reasoning solution is realized, thereby reducing the computing burden of the mobile terminal and accelerating the reasoning calculation process while protecting user privacy.

[0022] Figure 1 FIG. 1 is a schematic diagram showing an exemplary system 100 for performing distributed reasoning based on model parallelism according to an embodiment of the present disclosure. Figure 1 As shown, the exemplary system 100 includes a cloud (eg, edge / cloud server) and a terminal (eg, mobile terminal). It is worth noting that the following description includes a large number of unnecessary details, and those skilled in the art may choose to discard these details as needed.

[0023] In the exemplary system 100, the following operations are performed:

[0024] 1. According to different application models, a regression model of the output data volume and the required transmission rate between model layers is constructed.

[0025] For example, the AlexNet model can be used as an example to illustrate the regression model of the output data volume between model layers according to different application models. AI / ML model reasoning requires a lot of computing and storage resources. Taking the image classification model AlexNet as an example, the AlexNet model has a total of 61 million weight values ​​and 724 million MACs. Figure 3 The data and computing characteristics of each layer in the AlexNet model are shown. The light-colored part is the layer delay, that is, the computing time required for each layer, and the dark-colored part is the size of the output data of each layer, which is also the input of the next layer. According to the output data volume and computing time between the layers of the model, the best exit point can be better selected. However, it can be seen from the figure that the earlier the model exit point, the smaller the computing burden of the mobile terminal, but the more data needs to be unloaded. The later the model exit point, the less data needs to be unloaded, but the greater the computing burden of the mobile terminal.

[0026] 2. Download the decision engine to the mobile terminal.

[0027] 3. If the decision engine uses a new model for the first time, run the model on the mobile terminal with test data, record the average inference time and average power consumption between model layers, predict the time and power consumption required for the model layers to run on the mobile terminal, and calculate the exit point with the lowest power consumption of the mobile terminal; otherwise, it can be skipped.

[0028] For example, you can use 10 test data to perform inference on a mobile terminal, and record the inference time and computing power consumption between model layers. The model inference performance of mobile terminals is limited by hardware performance, such as computing unit power, memory, power consumption, etc. Use 10 test data to run the model on a mobile terminal, and record the average inference time and average power consumption between model layers, and record the time and power consumption required for the model layers to run on the mobile terminal; and calculate the exit point with the lowest power consumption of the mobile terminal. This is to speed up the calculation process of the exit point of each frame of the model and reduce the computing power consumption of the mobile terminal; at the same time, when the battery of the mobile terminal is at a low level, the decision engine will directly select the exit point with the lowest power consumption.

[0029] 4. Before starting reasoning, the decision model sends a request to the edge / cloud server to find an edge / cloud server that can perform distributed reasoning based on the principle of proximity, and uses the network transmission environment reasoning as the first frame, as well as the test data to predict the model's reasoning time on the server.

[0030] 5. If the battery of the mobile terminal is too low, the exit point with the lowest computing power consumption of the mobile terminal is directly selected; otherwise, the decision engine calculates the best model exit point for the next frame based on the application delay requirements, network transmission rate, and the inter-layer computing amount, time, and energy consumption loss of the mobile terminal's running model reasoning. The decision engine sends the model exit point to the edge / cloud server and uses it as the network transmission environment reasoning for the next frame.

[0031] For example, based on the end-to-end application time and the hardware resources of the mobile terminal, the decision engine calculates the optimal exit point of the model for each frame, thereby dynamically adjusting the overall optimal end-edge / cloud server collaborative reasoning solution:

[0032] a. Energy loss of mobile terminals

[0033] The energy consumption loss of mobile terminals is mainly related to the energy consumption required for reasoning of some models on mobile terminals and the energy consumption of output data transmission between layers, such as Figure 4 This patent only considers the power consumption required for reasoning in the mobile terminal.

[0034] min(kL0X0f0 2 +t1P1+kL3X3f3 2 +t4P4) (1)

[0035] b. Application latency requirements

[0036] Application latency requirements are mainly related to the inference time of some models on mobile terminals, the output data transmission time between layers, and the inference time of some models on end / cloud servers, such as Figure 5 shown.

[0037] The end-to-end latency requirement depends on the application type, but needs to meet the following requirements:

[0038] 0 <t3≤t-t1-t4 (2)

[0039] 0 <t4<t1 (3)

[0040] t0+t1+t2+t4≤t (4)

[0041] The calculation time of the mobile terminal is the prediction time or Time required to offload model layer data to edge / cloud servers

[0042] exist Figure 3-5 In the above description of this article, E indicates the energy consumption required for each frame of the mobile terminal; A0...A3 indicate the computing tasks performed in the mobile terminal; L0...L2 indicate the amount of input data (bits); X0...X3 indicate the computing intensity (cycle / bit) of the computing unit; t1, P1 indicate the transmission time and power consumption of data unloaded between model layers to the edge / cloud; t4, P4 indicate the transmission time and power consumption of the decision engine broadcasting the next exit point to the edge / cloud; t indicates the processing time of each frame; t0 indicates the time required for the model to perform reasoning in the mobile terminal; t1 indicates the transmission time for data unloaded between model layers to the edge / cloud; t2 indicates the time required for the model to perform reasoning in the edge / cloud server; t3 indicates the time when the decision engine calculates the model exit point of the next frame in the mobile terminal; t4 indicates the time when the decision engine broadcasts the next exit point.

[0043] 6. The mobile terminal performs partial model reasoning;

[0044] 7. After completing part of the model inference, the mobile terminal transmits the output data between the model layers to the edge / cloud server;

[0045] 8. The mobile terminal repeats step 5, while the edge / cloud server performs reasoning on the remaining model parts;

[0046] 9. The edge / cloud server returns the inference results to the mobile terminal;

[0047] 10. Repeat steps 6-9.

[0048] The inventors have proposed a distributed collaborative reasoning solution for end-edge / cloud servers based on model parallelism to solve the problems brought about by pure cloud or pure terminal model reasoning and calculation. The decision engine calculates the optimal exit point of the AI / ML model based on real-time environmental factors (such as model calculation amount, mobile terminal computing power, mobile terminal energy consumption, network environment, application delay requirements, etc.), and splits the model into two parts. The mobile terminal will perform model reasoning on the first part, and the edge / cloud service will perform model reasoning on the remaining part. This solution can optimize end-to-end delay and reduce the burden of model reasoning and calculation on mobile terminals.

[0049] Figure 2 is a flowchart illustrating an exemplary method 200 for performing model-parallel based distributed reasoning according to one embodiment of the present disclosure.

[0050] The process starts at step 201. At this step, a regression model constructed according to the current application model is downloaded and run, and the regression model is used to predict the amount of inter-layer output data of the current application model and the network transmission rate required to upload the inter-layer output data to the cloud server, the inter-layer reasoning time and power consumption of the current application model.

[0051] At step 202, it is determined whether the current power level of the mobile terminal is lower than a threshold power level.

[0052] At step 203, the exit point of the current application model is calculated based on the result of the determination. The exit point can be calculated by the following operations in step 203:

[0053] If the current power level of the mobile terminal is lower than the threshold power level, an exit point is calculated for the next frame of the current application model based on the minimum computing power consumption of the mobile terminal, and

[0054] If the current power level of the mobile terminal is greater than the threshold power level, the exit point is calculated for the next frame of the current application model based on the latency requirement of the current application model, the required network transmission rate predicted by the regression model, the amount of inter-layer output data, the inter-layer inference time and energy consumption, or one or more of the above.

[0055] At step 204, the calculated exit point is transmitted to the cloud server.

[0056] In the method, at the exit point the mobile terminal stops the inference of the current application model and pushes the obtained inter-layer output data to the cloud server for the cloud server to perform the inference of the remaining part of the current application model.

[0057] Optionally, the method further includes: when initially running the regression model, running the current application model using test data to obtain an average inter-layer inference time and average power consumption of the current application model for inference of a first frame of the current application model.

[0058] Optionally, the method further includes: sending a request to a cloud server to find a cloud server for distributed reasoning based on a proximity principle; and using the network environment parameters of the found cloud server for reasoning of the first frame of the current application model.

[0059] Optionally, the method further includes: receiving inference results of the remaining part of the current application model from the cloud server to complete the inference of the current frame.

[0060] Optionally, the method further includes: repeatedly performing the above operations until the last frame of the current application model.

[0061] This patent has a wide range of application scenarios and can be used in mobile terminals for heavy-loaded deep learning model reasoning or applications with high end-to-end latency requirements, such as object recognition, intelligent video surveillance, AR, industrial control, and robotics.

[0062] Object recognition: The mobile terminal takes a picture of unknown content and uses the AI / ML image model to identify the content of the picture, such as the breed of a pet. The image recognition model is divided into two parts. The mobile terminal runs the first part, and the edge / cloud server runs the remaining part to obtain the pet's large label. The edge / cloud server then matches it again, and then returns the recognition result to the mobile terminal. For user experience, the end-to-end application latency can be completed within 1 second.

[0063] Robot target detection: The robot shoots video in real time, identifies objects in the video, and completes user instructions. The robot performs image reasoning on the model part, and the edge / cloud server performs model reasoning on the remaining part, and then returns the next action instruction to the robot based on the reasoning results and user instructions.

[0064] Compared with the prior art, the present invention has one or more of the following advantages:

[0065] ① According to different application models, build a regression model of the model's inter-layer data output size and required transmission rate online, run the model on the mobile terminal, and use test data to estimate the model's inter-layer running time and required power consumption, reducing the amount of calculation of the decision engine's calculation exit point. According to actual environmental factors, calculate the model's exit point for the next frame more accurately.

[0066] ② When the mobile terminal is at a low power level, the decision engine directly adopts the previously calculated minimum power consumption model exit point.

[0067] ③ Placing the decision engine in the terminal will grasp the changes in the mobile terminal environment in real time and reduce communication transmission consumption; calculate the exit point of the next frame of the model during the idle time of the mobile terminal and plan the time slot reasonably.

[0068] Figure 6 An exemplary configuration of a mobile terminal 1200 capable of performing model parallelism-based distributed reasoning according to an embodiment of the present disclosure is shown.

[0069] The computing device 1200 is an example of a hardware device to which the above aspects of the present disclosure can be applied. The computing device 1200 can be any machine configured to perform processing and / or computing. The computing device 1200 can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal data assistant (PDA), a smart phone, a vehicle-mounted computer, or a combination thereof.

[0070] like Figure 6As shown, the computing device 1200 may include one or more elements that can be connected or communicated with the bus 1202 via one or more interfaces. The bus 2102 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, etc. The computing device 1200 may include, for example, one or more processors 1204, one or more input devices 1206, and one or more output devices 1208. The one or more processors 1204 may be any kind of processor, and may include, but is not limited to, one or more general-purpose processors or special-purpose processors (such as special-purpose processing chips). The processor 1202 may be configured, for example, to implement a mobile terminal that performs distributed reasoning based on model parallelism. The input device 1206 may be any type of input device that can input information to the computing device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. The output device 1208 may be any type of device that can present information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer.

[0071] The computing device 1200 may also include or be connected to a non-transitory storage device 1214, which may be any non-transitory storage device that can implement data storage, and may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a tape, or any other magnetic medium, a compressed disk or any other optical medium, a cache memory, and / or any other storage chip or module, and / or any other medium from which a computer can read data, instructions, and / or code. The computing device 1200 may also include a random access memory (RAM) 1210 and a read-only memory (ROM) 1212. The ROM 1212 may store programs, utilities, or processes to be executed in a non-volatile manner. The RAM 1210 may provide volatile data storage and store instructions related to the operation of the computing device 1200. The computing device 1200 may also include a network / bus interface 1216 coupled to a data link 1218. The network / bus interface 1216 may be any type of device or system capable of enabling communication with an external device and / or network, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device, and / or a chipset (such as a Bluetooth TM equipment, 802.11 equipment, WiFi equipment, WiMax equipment, cellular communication facilities, etc.).

[0072] The present disclosure may be implemented as any combination of an apparatus, a system, an integrated circuit, and a computer program on a non-transitory computer-readable medium. One or more processors may be implemented as an integrated circuit (IC), an application specific integrated circuit (ASIC), or a large scale integrated circuit (LSI), a system LSI, a super LSI, or a super LSI component that performs some or all of the functions described in the present disclosure.

[0073] The present disclosure includes the use of software, applications, computer programs, or algorithms. The software, applications, computer programs, or algorithms may be stored on a non-transitory computer-readable medium to enable a computer such as one or more processors to perform the above steps and the steps described in the accompanying drawings. For example, one or more memories store the software or algorithm in executable instructions, and one or more processors may associate a set of instructions to execute the software or algorithm to provide various functions according to the embodiments described in the present disclosure.

[0074] Software and computer programs (which may also be referred to as programs, software applications, applications, components or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logic programming language, or an assembly language or a machine language. The term "computer-readable medium" refers to any computer program product, apparatus or device for providing machine instructions or data to a programmable data processor, such as magnetic disks, optical disks, solid-state storage devices, memories and programmable logic devices (PLDs), including computer-readable media that receive machine instructions as computer-readable signals.

[0075] For example, computer readable media may include dynamic random access memory (DRAM), random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the desired computer readable program code in the form of instructions or data structures and can be accessed by a general or special purpose computer or a general or special purpose processor. As used herein, disk or disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically by laser. Combinations of the above are also included in the scope of computer readable media.

[0076] In addition, the above description provides examples without limiting the scope, applicability or configuration set forth in the claims. Without departing from the spirit and scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Various embodiments may appropriately omit, replace or add various processes or components. For example, features described with respect to certain embodiments may be combined in other embodiments.

Claims

1. A method for performing distributed reasoning based on model parallelism at a mobile terminal, include: Download and run a regression model built according to the current application model, wherein the regression model is used to predict the amount of inter-layer output data of the current application model, the network transmission rate required to upload the inter-layer output data to the cloud server, the inter-layer inference time and power consumption of the current application model; Determining whether the current power level of the mobile terminal is lower than a threshold power level; Calculating an exit point of the current application model based on the determination result, the calculation comprising: If the current power level of the mobile terminal is lower than the threshold power level, an exit point is calculated for the next frame of the current application model based on the minimum computing power consumption of the mobile terminal, and If the current power level of the mobile terminal is greater than the threshold power level, calculating an exit point for the next frame of the current application model according to one or more of the latency requirement of the current application model, the required network transmission rate predicted by the regression model, the amount of inter-layer output data, the inter-layer inference time, and the energy consumption; and transmitting the calculated exit point to the cloud server; Wherein, at the exit point, the mobile terminal stops the reasoning of the current application model and pushes the obtained inter-layer output data to the cloud server for the cloud server to perform the reasoning of the remaining part of the current application model.

2. The method according to claim 1, further comprising: include: When the regression model is initially run, the current application model is run using test data to obtain an average inter-layer inference time and average power consumption of the current application model for inference of a first frame of the current application model.

3. The method according to claim 1, further comprising: include: Send a request to the cloud server to find a cloud server for distributed reasoning based on the proximity principle; as well as The found network environment parameters of the cloud server are used for inference of the first frame of the current application model.

4. The method according to claim 1, further comprising: include: The inference results of the remaining part of the current application model are received from the cloud server to complete the inference of the current frame.

5. The method according to claim 4, further comprising: include: The above operations are repeatedly performed until the last frame of the current application model.

6. A mobile terminal, include: a memory having instructions stored therein; as well as A processor configured to execute instructions stored on the memory to perform the method according to any one of claims 1 to 5.

7. A computer-readable storage medium comprising computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 5.

Images