Distributed management method and system, and management system

Abstract

A distributed management system and method are disclosed. The distributed management system is applicable to a first computing system, wherein the first computing system and at least one second computing system form a task distribution network, and the distributed management system includes a manager, an orchestrator, and a scheduler, wherein the manager is used to organize and manage a predefined set of operations, wherein the set of operations is used to complete a specified function; the orchestrator is used to determine an operation flow composed of the set of operations according to requirements; and the scheduler is used to distribute the operation flow to one of at least one of the second computing systems for operation according to current load information of at least one second computing system. The system can construct an operation flow according to requirements and deploy the operation flow according to load information.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a distributed management method and system, and a management system. Background Art

[0002] In recent years, with the continuous rise of artificial intelligence, various neural network models and acceleration units used to support neural network models have emerged. Many customers hope to use neural network models in the application services they provide.

[0003] However, although many manufacturers are able to provide various hardware processors (acceleration units, central processing units) and software versions of neural network models, few have the resources to provide a complete platform architecture to assist customers or R&D personnel in the rapid development and deployment of AI (artificial intelligence) systems. Summary of the invention

[0004] Based on this, the purpose of this disclosure is to provide a platform architecture to help customers or R&D personnel quickly develop and deploy AI systems.

[0005] In a first aspect, an embodiment of the present disclosure provides a distributed management system, which is applicable to a first computing system, wherein the first computing system and at least one second computing system form a task distribution network, and the distributed management system includes a manager, an orchestrator, and a scheduler, wherein:

[0006] The manager is used to organize and manage a predefined set of operations, where the set of operations is used to complete a specified function;

[0007] The orchestrator is used to determine the operation flow composed of the operation set according to the demand;

[0008] The scheduler is used to distribute the operation flow to one of the at least one second computing systems for execution according to current load information of the at least one second computing system.

[0009] In some embodiments, the scheduler is further used to decompose the operation flow into multiple sub-operation flows according to current load information of the multiple second computing systems, and distribute the multiple sub-operation flows to run on some of the multiple second computing systems.

[0010] In some embodiments, the scheduler decomposes the operation flow into the plurality of sub-operation flows when it is determined that none of the plurality of second computing systems is suitable for independently running the operation flow based on current load information of the plurality of second computing systems.

[0011] In some embodiments, the scheduler determines whether the amount of input and output data between adjacent operation sets exceeds a set threshold to determine whether to divide the adjacent operation sets into the same sub-operation flow.

[0012] In some embodiments, the manager includes: an operation set registration unit, the operation set registration unit is used to register the operation set edited by the user, and use the registered operation set as the predefined operation set.

[0013] In some embodiments, the orchestrator includes a graphical interface, on which icons corresponding to the predefined operation set are displayed, so that a user can construct a directed acyclic graph on the graphical interface by dragging icons, and the orchestrator generates the operation flow according to the directed acyclic graph.

[0014] In some embodiments, the graphical interface includes: a viewing area for displaying the predefined operation set, a drawing area for drawing the directed acyclic graph, and a drawing area for displaying an operation flow corresponding to the directed acyclic graph.

[0015] In some embodiments, the current load information of the second computing system includes the following information:

[0016] The usage of the hardware execution unit inside the acceleration unit;

[0017] processor and memory usage;

[0018] Usage of input and output devices.

[0019] In some embodiments, both the first computing system and the second computing system are embedded systems.

[0020] In a second aspect, an embodiment of the present disclosure provides a distributed management system, applicable to a first computing system, wherein the first computing system and at least one second computing system form a task distribution network, and the distributed management system includes a manager, an orchestrator, and a scheduler.

[0021] The manager is used to organize and manage predefined operation sets, where the operation sets are divided into neural network operation sets and non-neural network operation sets;

[0022] The orchestrator is used to determine the network structure of the neural network model to be used according to the requirements, and construct an operation flow using the neural network operation set and the non-neural network operation set;

[0023] The scheduler is used to distribute the operation flow to one of the at least one second computing systems for execution according to current load information of the at least one second computing system.

[0024] In some embodiments, the requirements include: video processing, face recognition and target detection, and the non-neural network model operation set is classified into: image processing operation set, data encryption operation set, video encoding and decoding operation set, data storage operation set, image acquisition operation set.

[0025] In a third aspect, an embodiment of the present disclosure provides a management system, including a first computing system and at least one second computing system, wherein the first computing system includes a manager, an orchestrator, and a scheduler, wherein:

[0026] The manager is used to organize and manage a predefined set of operations, where the set of operations is used to complete a specified function;

[0027] The orchestrator is used to determine the operation flow composed of the operation set according to the demand;

[0028] The scheduler is used to allocate the operation flow to one of the at least one second computing system for execution according to current load information of the at least one second computing system,

[0029] Upon receiving the operation flow, the second computing system executes the operation flow.

[0030] In some embodiments, the first computing system performs cloud computing in a cloud environment, and the second computing system performs edge computing on an edge side of the cloud environment.

[0031] In some embodiments, the operation flow received and executed by the second computing system is an operation flow of a lightweight neural network model.

[0032] In a fourth aspect, an embodiment of the present disclosure provides a distributed management method, applicable to a first computing system, wherein the first computing system and at least one second computing system form a task distribution network, the management method comprising:

[0033] Determine an operation flow consisting of an operation set according to requirements, wherein the operation set is a predefined operation set used to complete a specified function;

[0034] The operation flow is distributed to one of the at least one second computing systems for execution according to current load information of the at least one second computing system.

[0035] In some embodiments, the process further includes: decomposing the operation flow into a plurality of sub-operation flows according to current load information of the plurality of second computing systems, and distributing the plurality of sub-operation flows to run on a plurality of the plurality of second computing systems.

[0036] In some embodiments, decomposing the operation flow into a plurality of sub-operation flows according to the current load information of the plurality of second computing systems comprises:

[0037] Determine whether there is a second computing system suitable for independently running the operation flow based on current load information of the plurality of second computing systems; and if it is determined that none of the plurality of second computing systems is suitable for independently running the operation flow, decompose the operation flow into the plurality of sub-operation flows.

[0038] In some embodiments, decomposing the operation flow into multiple sub-operation flows based on current load information of multiple second computing systems includes: determining whether the amount of input and output data between adjacent operation sets exceeds a set threshold to determine whether to divide the adjacent operation sets into the same sub-operation flow.

[0039] In some embodiments, the operation set is divided into a neural network operation set and a non-neural network operation set, and the operation flow composed of the operation set is determined according to requirements, including: determining the network structure of the neural network model to be used according to requirements, and using the neural network operation set to construct the operation flow of the neural network model, and using the non-neural network model operation set to construct the operation flow of the input-output module that provides input data for the neural network model and processes output data of the neural network model according to requirements.

[0040] In some embodiments, it also includes: registering the operation set edited by the user, and using the registered operation set as the operation set.

[0041] In some embodiments, the current load information of the second computing system includes the following information:

[0042] The usage of the hardware execution unit inside the acceleration unit;

[0043] processor and memory usage;

[0044] Usage of input and output devices.

[0045] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable medium, wherein the computer-readable medium stores computer instructions that can be executed by an electronic device, and when the computer instructions are executed, any of the above-mentioned management methods is implemented.

[0046] The distributed management system provided by the embodiments of the present disclosure is not only suitable for building AI systems equipped with neural network models, but also suitable for other application systems. The distributed management system can build operation flows as required and deploy operation flows according to load information, thereby realizing rapid construction and deployment of application systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects, features and advantages of the present disclosure will become more apparent through the description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0048] Figure 1 is a schematic diagram of a network structure applicable to an embodiment of the present disclosure;

[0049] Figure 2 is an example of a “hub” system architecture;

[0050] Figure 3 is an example of an embedded system;

[0051] Figure 4 A distributed management system according to an embodiment of the present disclosure is shown;

[0052] Figure 5 yes Figure 4 An exemplary graphical interface for the arranger in .

[0053] Figure 6 Used to illustrate the data flow relationship between managers, schedulers, orchestrators, and computing systems;

[0054] Figure 7 A flow chart of a management method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0055] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0056] The following terms are used in this article.

[0057] Acceleration unit: also known as neural network acceleration unit, it is a processing unit designed to improve the data processing speed in some special-purpose fields (for example, image processing, various operations of neural network processing, video processing, etc.) where the general-purpose processor is inefficient. It is often used in conjunction with a general-purpose processor CPU, and is controlled by the general-purpose processor to perform processing for some specific purposes or specific fields, thereby improving computer processing efficiency in specific purposes or specific fields.

[0058] Neural network: generally refers to artificial neural network (ANN), which is an algorithmic network that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. A classic neural network, which is also the simplest neural network structure, contains three levels: input layer, output layer and middle layer (also called hidden layer). The input layer, output layer and middle layer each include multiple neurons (also called nodes). Neurons are the smallest processing unit in a neural network. A very complex neural network structure can be formed through the extensive interconnection of a large number of neurons with simple functions.

[0059] Neural network model: In a neural network, neurons are mathematized to generate a neuron mathematical model. A large number of neuron mathematical models in a neural network form a neural network model.

[0060] Figure 1 10 is a schematic diagram of a network structure. Referring to the figure, a first computing system 101 and a plurality of second computing systems 103 communicate via a plurality of interconnection units 102. The plurality of interconnection units 102 are software and hardware communication platforms that support different interconnection protocols. The first computing system 101 and the second computing system 103 are a collection of software and hardware that have communication capabilities and can process certain tasks. The first computing system 101 can be implemented as various devices, including but not limited to the following devices: cellular phones, Internet protocol devices, digital cameras, personal digital assistants (PDAs), personal computers, notebooks, workstations, network computers (NetPC) set-top boxes, network hubs, wide area network (WAN) switches, and Internet of Things devices.

[0061] In some embodiments, the first computing system 101 and the second computing system 103 may be Figure 2 In the illustrated computer system 200, the interconnection unit 102 represents the Internet, or may be a local area network (LAN) or a wide area network (WAN), such as a company's private network.

[0062] like Figure 2 As shown, system 200 is an example of a "center" system architecture. The system 200 can be built based on various processors currently on the market and is powered by WINDOWS TM Operating system version, UNIX operating system, Linux operating system and other operating system drivers. In addition, the system 200 is generally implemented in a PC, a desktop, a notebook, or a server. Therefore, the system 200 in the disclosed embodiment is not limited to any specific combination of hardware circuits and software.

[0063] refer to Figure 2As shown, system 200 includes processor 201. Processor 201 has data processing capabilities known in the art. It can be a processor of complex instruction set (CISC) architecture, reduced instruction set (RISC) architecture, very long instruction set (VLIW) architecture, or a processor that implements a combination of the above instruction sets, or any processor device built for a special purpose.

[0064] The processor 201 is coupled to the system bus 202, which may be an interconnection circuit for connecting the processor 201 and various other components, and the interconnection circuit supports various interconnection protocols and interface circuits required by different requirements, thereby achieving communication between the processor 201 and various components.

[0065] The system 200 also includes a memory 204 and a graphics card 207. The memory 204 may be a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory or other memory. The memory 204 includes an instruction memory 205 and a data memory 206. By storing the instruction information and the data information represented by the data signal separately, when the processor 201 executes the instruction pipeline, it can fetch instructions from the instruction memory 205 and fetch data from the data memory 206 at the same time. The graphics card 207 includes a display driver for controlling the correct display of the display signal on the display screen.

[0066] The system 200 also includes a memory controller hub (MCH) 203, which can be coupled to the system bus 202, the memory 204, the graphics card 207, and the I / O controller hub (ICH) 208. The processor 201 can communicate with the memory controller hub (MCH) 203 via the system bus 202. The memory controller hub (MCH) 203 provides a high-bandwidth memory access path to the memory 204 for storing and reading instruction information and data information. At the same time, the memory controller hub (MCH) 203 and the graphics card 207 transmit display signals based on the graphics card signal input and output interface. The graphics card signal input and output interface is, for example, an interface type such as DVI, HDMI, etc.

[0067] The memory controller hub (MCH) 203 not only transmits digital signals between the processor 201 , the memory 204 and the graphics card 207 , but also bridges digital signals between the system bus 202 and the memory 204 and the I / O controller hub (ICH) 208 .

[0068] The system 200 also includes an I / O controller hub (ICH) 208, which is coupled to the memory controller hub (MCH) 203 via a dedicated hub interface bus, and some I / O devices are connected to the input / output control hub (ICH) 208 via a local I / 0 bus. The local I / 0 bus is used to couple peripheral devices to the input / output control hub (ICH) 208, and then to the memory controller hub (MCH) 203 and the system bus 202. The peripheral devices include, but are not limited to, the following devices: a hard disk 209, an optical drive 120, a sound card 211, a serial expansion port 212, an audio controller 213, a keyboard 214, a mouse 215, a GPIO interface 216, a flash memory 217, and a network card 218.

[0069] Of course, different computer systems have different structure diagrams according to different motherboards, operating systems and instruction set architectures. For example, many computer systems currently integrate the memory controller hub (MCH) 203 into the processor 201, so that the input / output control hub (ICH) 208 becomes the control hub coupled to the processor 201.

[0070] In some embodiments, the first computing system 101 and the second computing system 103 are as follows Figure 3 In the embedded system 300 shown, the interconnection unit 102 represents the Internet, a local area network (LAN), a wide area network (WAN), or the Internet of Things. Although the specific functions, appearance interfaces, operations, etc. of embedded systems are different, even very different, the basic hardware structure is similar, and has a high degree of similarity with the hardware system of a general-purpose computer. However, the characteristics of embedded system applications make the embedded system have great differences in hardware composition and implementation form from general-purpose computer systems.

[0071] First, in order to meet the requirements of the embedded system 300 in terms of speed, volume and power consumption, the operating system, application software, special data and other data that need to be stored for a long time usually do not use storage media such as disks with large capacity and slow speed, but mostly use random access memory 302 or flash memory (Flash Memory) 303.

[0072] In addition, in the embedded system 300, an A / D (analog / digital conversion) interface 305 and a serial interface 306 are required for the needs of measurement and control, which are rarely used in general-purpose computers. The A / D interface 305 mainly completes the conversion of analog signals to digital signals and the conversion of digital signals to analog signals required in the test. When the embedded system 300 is applied to industrial production, it often needs to be tested. Since the single-chip microcomputer generates digital signals, it needs to be converted into analog signals for testing during testing. Therefore, unlike general-purpose computers, an A / D (analog / digital conversion) interface 305 is required to complete the relevant conversion. In addition, multiple embedded systems are often required to be connected in series in industry to complete relevant functions, so a serial interface 306 is required for connecting multiple embedded systems in series, which is mostly not required in general-purpose computers.

[0073] In addition, as a basic processing unit, the embedded system 300 often needs to connect multiple embedded systems 300 into a network in industrial design, so it is necessary to connect the embedded system 300 to the network interface 307. This is also not needed in general-purpose computers. In addition, according to the actual application and scale, some embedded systems 300 need to use external buses 304. With the rapid expansion of the application field of embedded systems 300, embedded systems 300 are becoming more and more personalized, and the types of buses used according to their own characteristics are also increasing. In addition, in order to test the internal circuits of the embedded processor 301, the processor chip generally adopts boundary scan test technology. In order to adapt to this test, a debug interface 308 is adopted.

[0074] With the rapid development of very large scale integration (VLSI) and semiconductor technology, embedded systems can be implemented on a silicon chip, which is an embedded system on chip (SoC). Accordingly, the first computing system 301 and the second computing system 303 mentioned above can be a first system on chip and a second system on chip, and the interconnection unit 302 is a connection unit between each system on chip, for example, it can be an AHB bus, an APB bus, an AXI (Advanced eXtensibleInterface) bus, etc.

[0075] In addition, the first computing system 301 and / or the second computing system 303 may also be an embedded system or computer system with a neural network acceleration unit built in order to run the neural network acceleration unit. For example, the first computing system 301 or the second computing system 303 is an AI product prototype built according to demand, such as pairing a selected neural network computing card with a selected open source motherboard and equipping it with a high-performance processor, thereby forming a hardware platform with rich hardware interfaces.

[0076] It should be noted that the above description is only used to illustrate the applicable software and hardware environment of the embodiment of the present disclosure, and is not intended to limit the embodiment of the present disclosure. In fact, the first computing system 101 and the second computing system 103 can be any type of computing system described above, thereby forming a cross-system, cross-platform network structure.

[0077] Figure 4 A distributed management system according to an embodiment of the present disclosure is shown. As shown in the reference figure, the distributed management system 50 is located between individual or enterprise users and service providers, connecting the upper and lower parts. Individual or enterprise users can build application systems based on the distributed management system 50 and deploy them on the hardware platform 70. The hardware platform 70 is composed of computing systems 1-3. Individual or enterprise users can also provide requirements 60 to the service provider. The service provider decomposes and processes the requirements 60 based on the distributed management system 50 to form an application system, and deploys the application system to each computing system on the hardware platform 70. With the distributed management system 50, R&D personnel only need to focus on development work related to the requirements 60, without having to manage the underlying equipment. They only need to connect the system to the hardware platform to give play to the advantages of hardware acceleration.

[0078] Can be based on Figure 1 The network structure shown in FIG. 5 is a distributed management system 50. The distributed management system 50 is deployed in Figure 1 On the first computing system 101 in the example, each computing system under the hard disk platform 70 is equivalent to Figure 1 The second computing system 103 in the network structure. The distributed management system 50 is responsible for task allocation and management in the network structure, and each computing system under the hard disk platform 70 is responsible for executing the assigned tasks in the network structure.

[0079] The distributed management system 50 includes a manager 501, a scheduler 502, and an orchestrator 503. The manager 501 is used to uniformly organize and manage predefined operation sets. An operation set is a set of a series of operations used to implement a certain function or task. The orchestrator 503 is used to determine an operation flow composed of an operation set according to requirements. The scheduler 502 is used to obtain the current load information of each computing system on the hardware platform and distribute the operation flow to at least one computing system for operation.

[0080] To facilitate the management of the operation set, the manager 501 defines three basic components for the operation set, namely: startup interface, function definition and call, and transmission data type. Startup interface, just as all programs have an entry function, the operation set defines a unified startup interface to facilitate external instantiation and startup. Function definition and call, as a unit that provides computing services to the outside, the service capability range of the operation set is limited and clear. The function definition describes the services provided to the outside, and the function call describes the specific calling method of the service. Transmission data type, as an execution node, the operation set describes its own input and output data type. Only when the input and output data types match can data be transmitted. Based on the operation set, cross-platform, cross-machine, cross-network, and cross-protocol operations can be unified in the same operation set.

[0081] The manager 501 can also define multiple operation set templates, which specify the operation set specifications, including the operation set startup interface, function definition and call, and basic requirements for transmission data types. R&D personnel can use the operation set templates to develop custom operation sets according to actual needs. In this way, R&D personnel can quickly develop custom operation sets and build more personalized application systems accordingly.

[0082] In some embodiments, the manager 501 includes an operation set registration unit, which is used to register the operation set edited by the user and use the registered operation set as a predefined operation set. The manager 501 can provide an editing interface to the R&D personnel, so that the R&D personnel can use the editing interface to make an operation set, and then use the operation set registration unit to register it in the system. The manager 501 can also only provide an upload interface to the R&D personnel, so that the R&D personnel can upload the already made operation set to the system based on the upload interface, and then use the operation set registration unit to register it in the system.

[0083] After the manager 501 constructs the operation set, it can provide the detailed information of the operation set to the scheduler 503 and the scheduler 502 through a storage unit such as a database or a table. The scheduler 503 analyzes the demand in combination with the operation set stored in the system, determines the operation set to be used, and combines the operation set to be used into an operation flow, and the combination methods include but are not limited to time sequence combination, conditional combination, loop combination, jump combination, etc.

[0084] In some embodiments, the orchestrator 503 constructs the operation flow based on DAG (Directed Acyclic Graph). Figure 5It is an exemplary graphical interface of the orchestrator 503. The graphical interface includes multiple functional areas. Among them, the operation set view area 601, the drawing area 602, the operation flow display area 603 and the code display area 604 are the main areas involved in the construction of the operation flow. The operation set view area 601 is used to display the predefined operation set. Icon 6011 is an icon corresponding to each predefined operation set. Clicking icon 6011 can display detailed information of the operation set including the startup interface, function definition and call, and transmission data type on the right side of the icon. The drawing area 602 is used to draw and display the directed acyclic graph of the operation set, and the operation flow display area 603 is used to display the operation flow corresponding to the current directed acyclic graph (the figure is the operation flow of the neural network model used as an example). The code display area 604 is used to display the source code corresponding to the current directed acyclic graph. When the developer is working on the task 0, he will first determine the required operation set according to the requirements, then drag the icon corresponding to the required operation set to the drawing area 602, and then establish a directed connection line between the operation sets on the drawing area 602 to form a directed acyclic graph. The operation flow corresponding to the directed acyclic graph and the source code formed according to the operation flow will be displayed in the operation flow display area 603 and the code display area 604. The developer can also refer to the displayed operation flow and source code, and then modify the directed acyclic graph in turn. This graphical interface helps the developer to build the operation flow in a visual way.

[0085] In other embodiments, the arranger 503 does not adopt the mode of building a directed acyclic graph to build the operation flow. For example, the arranger 503 provides a textual operation flow editing interface to the research and development personnel. On the basis of understanding the demand, the research and development personnel map the realization of the demand to the combination of the operation set, display the combination of the operation set on the operation flow editing interface, and then make various adjustments to the combination of the operation set in the operation flow editing interface, and finally obtain the operation flow. In this process, the research and development personnel also need to pay attention to whether the data types of adjacent operation sets match. For example, the output data type of the previous operation set is a floating point type, and the input data type of the latter operation set is an integer type. At this time, it is necessary to insert a type conversion operation set between the adjacent operation sets to convert the floating point type data into an integer type data.

[0086] The scheduler 502 is responsible for allocating the operation stream to each computing system. There are several specific allocation methods: the first is to allocate the operation stream as a whole to a computing system for operation. The advantage of this is that it is relatively simple to handle, but the disadvantage is that the hardware performance of the computing system cannot be fully utilized; the second is to divide the operation stream into multiple sub-operation streams, and then allocate the multiple sub-operation streams to each computing system for operation, that is, to disperse the operation stream to different computing systems for execution. The advantage of this is that the hardware performance of the computing system can be fully utilized, but the disadvantage is that the data dependency between each sub-operation stream needs to be considered, which is relatively complicated to handle. In the embodiment of dividing the operation stream into multiple sub-operation streams, the scheduler 503 determines whether the input and output data volume between adjacent operation sets exceeds a set threshold value. If the input and output data volume does not exceed the set threshold value, the adjacent operation sets are divided into two sub-operation streams. If the input and output data volume exceeds the set threshold value, the adjacent operation sets are divided into the same sub-operation stream.

[0087] The above-mentioned distributed management system 50 can be used to build an AI system, which is an application system equipped with a neural network model. The application system is used to process requirements such as video processing, target detection, face recognition, etc. 70. Specifically, first, the manager divides the operation set into two categories: the operation set used by the neural network model is called the neural network model operation set; the operation set used by the remaining modules is called the non-neural network model operation set. As shown in the figure, the non-neural network operation set includes an image processing operation set, a data encryption operation set, a video encoding and decoding operation set, a data storage operation set, and an image acquisition operation set, and the neural network model operation set is the model operation set. Then, the R&D personnel can collect the information of the hardware execution unit of the acceleration unit, and also collect the network structure information of the formed neural network model and summarize and refine this information, and then define one or more neural network model operation sets; the R&D personnel can also collect the application requirements of enterprises and individuals for the neural network model, extract one or more functional units from them, and form these functional units into a non-neural network model operation set. When it is a specific requirement, the scheduler 503 first determines the network structure of the neural network model to be used according to the requirement. The network structure is preferably a network structure that has been trained and verified. Then, according to the correspondence between each node in the network structure and the operation set, the operation flow corresponding to the network structure is constructed. Then, according to the requirement, the non-neural network model operation set is used to construct the operation flow of the input and output modules that provide the input data of the neural network model and process the output data of the neural network model. Finally, the operation flow is combined and output. Of course, in the execution stage, the neural network operation flow and the non-neural network operation flow can also be executed separately on different computing systems. The neural network operation flow is mainly used for neural network model calculations, and the non-neural network operation flow is mainly used for operations such as data collection, preprocessing and transmission.

[0088] The operation and maintenance development host 40 in the figure represents the host configuration used by R&D personnel, including but not limited to a graphical development environment, remote management (web) and remote management (SSH).

[0089] Figure 6Used to illustrate the data flow relationship between manager 501, scheduler 502, orchestrator 503 and computing system. Orchestrator 503 obtains various predefined operation sets from manager 501, and constructs operation flow in combination with demand. Scheduler 502 receives the operation flow and the current load information of computing systems 1-3, and allocates the operation flow to one or more of computing systems 1-3. The current load information reflects the usage of various resources on the computing system. Various resources include hardware devices such as central processing units, acceleration units, memories, and input and output devices. For example, computing system 1 reports to orchestrator 503 that there are 8 acceleration units in the system, and there are currently 4 acceleration units running the neural network model. Computing system 1 can also report to orchestrator 503 the usage of the hardware execution units in each acceleration unit, for example, there are 8 convolution operators, and 4 are occupied. For the central processor, the total processor utilization of all cores can be reported. If the scheduler 503 finds that the indicator exceeds the set threshold, the central processor can be considered to be in a busy state. For the memory, the total amount of physical memory occupied by all currently running processes can be reported. The scheduler 503 measures its busyness based on this. If the indicator exceeds the set threshold, the physical memory can be considered to be in a busy state. The scheduler 503 evaluates the various indicators reported by the computing system to decide whether to split the operation flow into multiple sub-operation flows and then deploy them or to deploy the operation flow as a whole.

[0090] For an AI system equipped with a neural network model, since the neural network model mainly runs on the acceleration unit, the orchestrator 503 can allocate the operation flow according to the usage of the acceleration unit. For example, the operation flow as a whole can be allocated to the computing system containing the most unused acceleration units, or the operation flow for data preprocessing and the operation of the neural network model can be deployed on at least two computing systems.

[0091] When the orchestrator 503 determines that no computing system is suitable for independently running the operation flow based on the current load information of each computing system, the operation flow is decomposed into multiple sub-operation flows, and the sub-operation flows are deployed based on the current load information. When the operation flow is deployed in a dispersed manner to each computing system, a higher data input and output data communication cost will be generated. For example, when the output data of the operation set 1 deployed on the computing system 1 is the input data of the operation set 2 deployed on the computing system 2, it is necessary to transfer the output data of the operation set 1 from the computing system 1 to the computing system 2, thereby generating the data input and output data communication cost. This resource overhead will affect the overall execution efficiency of the system. In this case, the amount of input and output data divided between the various operation sets running on different computing systems can be calculated. If this amount of data exceeds the set threshold, the adjacent operation sets whose input and output data amounts exceed the set threshold are re-divided or divided and merged to run on the same computing system.

[0092] It should be understood that the above network structure can be a network structure in a private environment built by the enterprise itself, or a network structure in a cloud environment, such as a data center. When the operation set is assigned to each computing system in the data center for operation, the scheduler 502 also needs to consider the physical distribution information of each computing system. A data center generally includes multiple computing system clusters located in different regions. When the scheduler 502 assigns the operation set to each computing system in the data center for operation, it is necessary to select a region first, and then the operation set can be assigned to a computing system in the computing system cluster in the selected region for operation.

[0093] The distributed management system provided by the embodiment of the present disclosure is not only applicable to the construction of an AI system equipped with a neural network model, but also applicable to other application systems. The distributed management system can construct operation flows according to requirements and deploy operation flows according to load information, thereby realizing the rapid construction and deployment of application systems. For example, for an AI system, after defining various available operation sets, the required AI system can be built in a manner similar to building blocks, thereby realizing the AI ​​system at a relatively low cost and with relatively high performance.

[0094] The above-mentioned distributed management system also provides an integrated development and deployment environment including development verification, product integration and product deployment. Specifically, R&D personnel can build a hardware platform (such as an embedded platform or an AI platform) according to requirements, and then use the above-mentioned distributed management system to build the operation flow of the application system according to requirements, repeatedly test and verify the operation flow of the application system on the distributed management system, and finally use the distributed management system to deploy the tested and verified operation flow to the computing system in the actual environment.

[0095] Accordingly, an embodiment of the present disclosure also provides a management system. The management system includes the above-mentioned first computing system and the second computing system. The distributed management system runs on the first computing system. Optionally, the first computing system and the second computing system form a cloud environment, the first computing system runs in the cloud, and the second computing system performs edge computing on the edge side. The edge side refers to the side close to the source of objects or data, and the computing system that performs edge computing is between the physical entity and the industrial connection, or at the top of the physical entity to meet the basic requirements of real-time, intelligence, security and privacy protection. Furthermore, the distributed management system deploys the operation flow of the constructed AI system on the edge side for execution, and the AI ​​system can be a lightweight or simplified version of the neural network model.

[0096] Figure 7 The flowchart of the management method according to one embodiment of the present disclosure is shown, which specifically includes the following steps.

[0097] Step S110 determines the operation flow composed of the operation set according to the demand. The system includes multiple predefined operation sets. The detailed information of the operation set will be saved in the storage unit. In order to ensure the validity of the operation set, the detailed information of the edited operation set is stored in the storage unit through the registration operation. When arranging the operation flow, the detailed information of the operation set is obtained from the storage unit to determine which operation sets are selected to arrange the operation flow. In other words, only the registered operation set can be used to arrange the operation flow.

[0098] In the case where it is necessary to build and configure a neural network model, the network structure of the neural network model to be used is determined according to the requirements, and the operation flow is constructed using a neural network operation set and a non-neural network operation set. Optionally, the operation flow of the neural network model is constructed first. After the operation flow of the neural network model is determined, the input type and output type of the neural network model can be determined. At this time, it is necessary to provide the neural network model with input data that conforms to its input type, and to process the output data of the neural network model into data that meets the requirements, that is, to use a non-neural network model operation set to construct an operation flow that provides input data for the neural network model, and to use a non-neural network model operation set to construct an operation flow that processes the output data of the neural network model. At this point, the operation flow of the system equipped with a neural network model is successfully constructed.

[0099] Step S120 obtains current load information of multiple computing systems. The load information includes load information of hardware such as acceleration units, processors, memories, input and output, and corresponding software. Each computing system can report the current load information.

[0100] Step S130 determines whether there is a computing system suitable for independently running the operation flow. Specifically, based on the load information of each computing system, it is determined whether there is a computing system suitable for independently running the operation flow, and according to the determination result, it is determined whether to run the operation flow on one computing system or to divide the operation flow into multiple sub-operation flows. The computing system of the independently running operation flow, for example, has an idle acceleration unit, and the usage of the processor and memory meets the requirements. If there is a computing system suitable for independently running the operation flow, S160 is executed, otherwise S140 is executed.

[0101] Step S140 decomposes the operation flow into multiple sub-operation flows. For example, for each operation set in the operation flow, the amount of input and output data of the adjacent operation sets is calculated. If the amount of data exceeds a set threshold, it is divided into the same sub-operation flow, otherwise it is divided into different operation flows. For another example, according to the current load information of each computing system, the number of sub-operation flows and the computing system adapted to each sub-operation flow are determined, and then the sub-operation flows are divided.

[0102] Step S150 distributes the multiple sub-operation flows to multiple computing systems for execution, for example, deploying the multiple sub-operation flows to multiple computing systems according to the adapted computing systems determined during the division.

[0103] Step S160 distributes the operation flow to a computing system for execution.

[0104] The disclosed embodiments provide a distributed management system that constructs operation flows according to requirements and determines whether to decompose and deploy the operation flows according to load information, so as to fully utilize the software and hardware performance of the computing system.

[0105] In some embodiments, when constructing an operation flow, a directed acyclic graph is constructed on a graphical interface by dragging icons, and icons corresponding to a predefined operation set are displayed on the graphical interface, and then the operation flow is created according to the directed acyclic graph.

[0106] In some embodiments, allocating operation flows to various servers for execution includes: calculating whether the amount of input and output data between adjacent operation sets exceeds a set threshold; when the amount of input and output data between adjacent operation sets exceeds the set threshold, dividing the adjacent operation sets to run on the same computing system, otherwise, dividing the adjacent operation sets to run on different computing systems.

[0107] In some embodiments, the current load information of the computing system includes the following information:

[0108] The occupancy status of the hardware execution units in the acceleration unit;

[0109] CPU and memory usage;

[0110] Usage of input and output devices.

[0111] The commercial value of the disclosed embodiments

[0112] AI systems equipped with neural network models have good application effects in target detection, face recognition, natural language processing, etc. Taking the field of face recognition as an example, by collecting video surveillance through cameras, recognizing facial images through neural network models, and comparing facial images with faces stored in the cloud, criminals in surveillance videos can be identified. In the field of speech recognition, speech recognition is performed through neural network models to achieve simultaneous interpretation. These application scenarios can bring huge commercial benefits. The distributed management system provided in the embodiments of the present disclosure can help R&D personnel quickly build an AI system equipped with a neural network model, so the system also has broad application prospects and economic value.

[0113] Those skilled in the art will appreciate that the present disclosure can be implemented as a system, method, and computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely, complete hardware, complete software (including firmware, resident software, microcode), and can also be implemented in the form of a combination of software and hardware. For example, the present disclosure is implemented as a terminal device, including a memory and a processor, and the memory also stores computer instructions that can be executed by the processor. When the computer instructions are executed, the distributed management method provided by the present formula embodiment is implemented.

[0114] Furthermore, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, wherein the computer-readable media contains computer-readable program code.

[0115] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by a processing unit, a device or a device, or used in combination with the program.

[0116] Computer readable signal media may include data signals propagated in baseband or as part of a chopped wave, which carry computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any other suitable combination. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction system, apparatus, or device.

[0117] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., and any suitable combination of the foregoing.

[0118] The computer program code for executing the disclosed embodiments may be written in one or more programming languages ​​or combinations. The programming languages ​​include object-oriented programming languages, such as JAVA, C++, and may also include conventional procedural programming languages, such as C. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0119] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A distributed management system, applicable to a first computing system, wherein the first computing system and at least one second computing system form a task distribution network, wherein the distributed management system comprises a manager, an orchestrator and a scheduler, The manager is used to organize and manage predefined operation sets, where the operation sets are divided into neural network operation sets and non-neural network operation sets; The orchestrator is used to determine the network structure of the neural network model to be used according to the requirements, use the neural network operation set to construct the operation flow of the neural network model, and use the non-neural network model operation set to construct the operation flow of the input-output module that provides input data of the neural network model and processes output data of the neural network model; The scheduler is used to distribute the operation flow of the neural network model and the operation flow of the input-output module to at least one of the second computing systems for execution based on the current load information of at least one of the second computing systems.

2. The distributed management system according to claim 1, wherein: The scheduler is further configured to decompose the operation flow into a plurality of sub-operation flows according to current load information of the plurality of second computing systems, and distribute the plurality of sub-operation flows to run on a plurality of the plurality of second computing systems.

3. The distributed management system according to claim 2, wherein: The scheduler decomposes the operation flow into the plurality of sub-operation flows when it is determined that none of the plurality of second computing systems is suitable for independently running the operation flow according to current load information of the plurality of second computing systems.

4. The distributed management system according to claim 2 or 3, wherein: The scheduler determines whether the amount of input and output data between adjacent operation sets exceeds a set threshold to determine whether to divide the adjacent operation sets into the same sub-operation flow.

5. The distributed management system according to claim 1, wherein: The manager comprises: an operation set registration unit, which is used to register the operation set edited by the user and use the registered operation set as the predefined operation set.

6. The distributed management system according to claim 5, wherein: The orchestrator includes a graphical interface, on which icons corresponding to the predefined operation set are displayed, so that a user can construct a directed acyclic graph on the graphical interface by dragging icons, and the orchestrator generates the operation flow according to the directed acyclic graph.

7. The distributed management system according to claim 6, wherein: The graphical interface includes: a viewing area for displaying the predefined operation set, a drawing area for drawing the directed acyclic graph, and a drawing area for displaying the operation flow corresponding to the directed acyclic graph.

8. The distributed management system according to claim 1, wherein: The current load information of the second computing system includes the following information: The usage of the hardware execution unit inside the acceleration unit; processor and memory usage; Usage of input and output devices.

9. The distributed management system according to any one of claims 1 to 8, wherein: The first computing system and the second computing system are both embedded systems.

10. The distributed management system according to claim 1, wherein: The requirements include: video processing, face recognition and target detection, and the non-neural network model operation set is classified into: image processing operation set, data encryption operation set, video encoding and decoding operation set, data storage operation set, and image acquisition operation set.

11. A management system, comprising the first computing system according to any one of claims 1 to 10 and at least one of the second computing systems.

12. The management system according to claim 11, wherein: The first computing system executes in the cloud, and the second computing system executes edge computing on the edge side.

13. The management system according to claim 12, wherein the operation flow received and executed by the second computing system is an operation flow of a lightweight neural network model.

14. A distributed management method, applicable to a first computing system, wherein the first computing system and at least one second computing system form a task distribution network, the management method comprising: Determine an operation flow consisting of an operation set according to requirements, wherein the operation set is a predefined operation set used to complete a specified function; Allocating the operation flow to one of the at least one second computing system for execution according to current load information of the at least one second computing system, Among them, the operation set is divided into a neural network operation set and a non-neural network operation set, and the operation flow composed of the operation set determined according to requirements includes: determining the network structure of the neural network model to be used according to requirements, and using the neural network operation set to construct the operation flow of the neural network model, and according to requirements, using the non-neural network model operation set to construct the operation flow of the input-output module that provides input data for the neural network model and processes output data of the neural network model.

15. The distributed management method according to claim 14, wherein: Also includes: According to current load information of the plurality of second computing systems, the operation flow is decomposed into a plurality of sub-operation flows, and the plurality of sub-operation flows are distributed to run on a plurality of the plurality of second computing systems.

16. The distributed management method according to claim 15, wherein: Decomposing the operation flow into a plurality of sub-operation flows according to the current load information of the plurality of second computing systems comprises: Determine whether there is a second computing system suitable for independently running the operation flow based on current load information of the plurality of second computing systems; and if it is determined that none of the plurality of second computing systems is suitable for independently running the operation flow, decompose the operation flow into the plurality of sub-operation flows.

17. The distributed management method according to claim 15 or 16, wherein: Decomposing the operation flow into multiple sub-operation flows according to the current load information of the multiple second computing systems includes: judging whether the amount of input and output data between adjacent operation sets exceeds a set threshold to determine whether to divide the adjacent operation sets into the same sub-operation flow.

18. The distributed management method according to claim 14, wherein: Also includes: The operation set edited by the user is registered, and the registered operation set is used as the operation set.

19. The distributed management method according to claim 14, wherein: The current load information of the second computing system includes the following information: The usage of the hardware execution unit inside the acceleration unit; processor and memory usage; Usage of input and output devices.

20. A computer-readable medium storing computer instructions executable by an electronic device, wherein when the computer instructions are executed, the distributed management method according to any one of claims 14 to 19 is implemented.

Images