Neural network model scheduling and deployment method and its device, equipment, medium, and product

Abstract

The present application discloses a method for scheduling and deploying a neural network model and its device, equipment, medium, and product. The method includes: responding to a configuration request pushed by a configuration end, querying idle working nodes in a working node list; pushing the configuration information contained in the configuration request to an idle working node, driving the working node to obtain the corresponding neural network model for configuration and operation, and correspondingly updating the working state and model feature information of the working node in the working node list; monitoring the running state of the working node running the neural network model, and when the running state meets the preset running conditions, querying other working nodes currently running the neural network model from the working node list; pushing service update instructions to other working nodes, and updating the service state of the neural network model running on other working nodes. The present application can perform rapid deployment of neural network models and iterative updates of multiple versions to meet diverse business deployment requirements.

Description

Technical Field

[0001] The present application relates to the field of computer deep learning, and in particular to a method for scheduling and deploying a neural network model. In addition, it also relates to corresponding devices, equipment, non-volatile storage media and computer program products of the method. Background Art

[0002] As neural network models develop, the application demand for neural network models also grows. Developers often deploy neural network models to graphics processing units (GPUs) to meet the huge floating-point computing resources required for the operation of neural network models. Existing neural network model deployment services often require complicated manual operations for deployment. In addition, since neural network models occupy a high amount of floating-point computing resources, it is particularly important to effectively manage graphics processing units (GPUs) for deployment and utilization. However, existing deployment services often do not have the ability to manage graphics processing units (GPUs), resulting in developers managing the neural network models deployed by each graphics processing unit (GPU) on their own to solve the problem of tight computing resources.

[0003] In view of the problems existing in the existing neural network model deployment services, the applicant has made corresponding explorations in order to solve the problem. Summary of the invention

[0004] The purpose of this application is to provide a neural network model scheduling and deployment method to meet user needs. In addition, it also involves corresponding devices, equipment, non-volatile storage media and computer program products of the method.

[0005] In order to achieve the purpose of this application, the following technical solutions are adopted:

[0006] A neural network model scheduling and deployment method proposed for the purpose of this application includes the following steps:

[0007] In response to the configuration request pushed by the configuration terminal, query the working node in the working node list whose working state is idle, wherein the working node list stores address port information, working state and model feature information of the neural network model running by multiple working nodes, and the working node has a graphics processor for running the neural network model;

[0008] Push the configuration information contained in the configuration request to an idle working node, drive the working node to pull the neural network model corresponding to the configuration information from the model storage space for configuration operation, and then update the working status and model feature information of the working node in the working node list accordingly;

[0009] Monitor the running status of the neural network model corresponding to the configuration information when the working node runs it, and when the running status meets the running conditions preset by the configuration end, query other working nodes whose model feature information matches the neural network model from the working node list;

[0010] Push service update instructions to the other working nodes to update the service status of the other working nodes running the neural network model.

[0011] In a further embodiment, the step of pushing the service update instruction to the other working nodes to update the service status of the other working nodes running the neural network model includes the following steps performed by the working nodes:

[0012] In response to the service update instruction pushed by the scheduling node, the currently running neural network model is stopped, and its own service status is changed to an offline status;

[0013] Pushing a work update instruction to the scheduling node drives the scheduling node to update the work status and model feature information of the current working node in the working node list.

[0014] In a further embodiment, the step of pushing the service update instruction to the other working nodes to update the service status of the other working nodes running the neural network model includes the following steps performed by the working nodes:

[0015] In response to the service update instruction pushed by the scheduling node, obtain the weight information contained in the instruction and acting on the currently running neural network model, wherein the weight information is obtained by the scheduling node from the working node that trains the latest version of the neural network model to convergence;

[0016] The currently running neural network model is updated according to the weight information, and an update completion notification is pushed to the scheduling node, driving the scheduling node to update the model version number represented by the model feature information of the current working node in the working node list.

[0017] In a further embodiment, the step of pushing the configuration information contained in the configuration request to an idle working node, driving the working node to pull the neural network model corresponding to the configuration information from the model storage space for configuration operation, and then correspondingly updating the working status and model feature information of the working node in the working node list includes the following steps performed by the working node:

[0018] Receiving configuration information pushed by the scheduling node, querying and acquiring the neural network model corresponding to the configuration information in the model storage space for configuration;

[0019] After completing the configuration of the neural network model, a configuration completion notification is pushed to the scheduling node, driving the scheduling node to update the working status and model feature information of the current working node in the working node list.

[0020] In a further embodiment, the method comprises the following preparatory steps:

[0021] In response to a registration request pushed by a working node, the address and port information of the working node included in the registration request is obtained;

[0022] The address port information is stored in the working node list, and the working state and model feature information of the working node in the working node list are initialized.

[0023] In a further embodiment, the method comprises the following post-processing steps:

[0024] In response to the service request forwarded by the service proxy gateway, query the working node of the model feature information corresponding to the neural network model corresponding to the request in the working node list, and query and obtain the preset service rule corresponding to the working node from the service rule list;

[0025] When the preset service rule is a transparent rule, the address port information of the working node is pushed to the service proxy gateway so that the service proxy gateway calls the neural network model of the working node to execute the service through the address port information;

[0026] When the preset service rule is a security rule, the service data contained in the service request is pushed to the working node, and the inference result returned by the working node inputting the service data into its own neural network model is pushed to the service proxy gateway.

[0027] A neural network model scheduling and deployment device proposed for the purpose of this application includes:

[0028] A request response submodule, used to respond to the configuration request pushed by the configuration terminal, query the working node in the working node list that is in an idle working state, wherein the working node list stores the address port information, working state and model feature information of the neural network model running by multiple working nodes, and the working node has a graphics processor for running the neural network model;

[0029] A model configuration module, used to push the configuration information contained in the configuration request to an idle working node, drive the working node to pull the neural network model corresponding to the configuration information from the model storage space for configuration operation, and then update the working state and model feature information of the working node in the working node list accordingly;

[0030] An operation monitoring module is used to monitor the operation status of the neural network model corresponding to the configuration information when the working node runs it, and when the operation status meets the operation conditions preset by the configuration end, query other working nodes whose model feature information matches the neural network model from the working node list;

[0031] The service update module is used to push service update instructions to the other working nodes to update the service status of the other working nodes running the neural network model.

[0032] In a further embodiment, the model configuration module includes:

[0033] A model acquisition submodule, used to receive the configuration information pushed by the scheduling node, query and obtain the neural network model corresponding to the configuration information in the model storage space for configuration;

[0034] The notification push submodule is used to push the configuration completion notification to the scheduling node after completing the configuration of the neural network model, and drive the scheduling node to update the working status and model feature information of the current working node in the working node list.

[0035] In a further embodiment, the service update module includes:

[0036] A service offline submodule, used to respond to the service update instruction pushed by the scheduling node, stop the currently running neural network model, and change its own service state to an offline state;

[0037] The instruction push submodule is used to push the work update instruction to the scheduling node, driving the scheduling node to update the working status and model feature information of the current working node in the working node list.

[0038] In a preferred embodiment, the service update module further includes:

[0039] A weight acquisition submodule, for responding to the service update instruction pushed by the scheduling node, and acquiring the weight information contained in the instruction and acting on the currently running neural network model, wherein the weight information is acquired by the scheduling node from the working node that has trained the latest version of the neural network model to convergence;

[0040] The weight update submodule is used to update the currently running neural network model according to the weight information, and push the update completion notification to the scheduling node, driving the scheduling node to update the model version number represented by the model feature information of the current working node in the working node list.

[0041] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned neural network model scheduling and deployment method.

[0042] In order to solve the above technical problems, an embodiment of the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned neural network model scheduling and deployment method.

[0043] In order to solve the above technical problems, an embodiment of the present application also provides a computer program product, including a computer program and computer instructions. When the computer program and computer instructions are executed by a processor, the processor executes the steps of the above-mentioned neural network model scheduling and deployment method.

[0044] Compared with the prior art, the advantages of this application are as follows:

[0045] First of all, this application can provide developers with the function of one-click deployment of neural network models. The scheduling node manages the working status, address port information and model feature information of each working node with a graphics processing unit (GPU) in the working node cluster through a working node list, so that the scheduling node responds to the configuration request pushed by the configuration end where the developer is located, and quickly schedules the working nodes in the idle state to configure and run the neural network model.

[0046] Secondly, the scheduling node of the present application monitors the running status of the neural network model running on each working node in the working node cluster. When the working node configured with the new version of the neural network model can provide the corresponding online service, the working node running the old version of the neural network model will be taken offline to release its computing power resources, thereby realizing the hot update of the version of the neural network model deployed in the working node cluster, and efficiently utilizing the floating-point computing power resources of each graphics processing unit (GPU) in the cluster.

[0047] In addition, the scheduling node of the present application can realize the version update of the neural network model deployed in the cluster and the joint operation of multiple versions of the neural network model, so that various types and versions of neural network models can share the floating-point computing power resource pool in the same working node cluster, and at the same time maintain various types of neural network online services for the platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1A schematic diagram of a typical network deployment architecture related to implementing the technical solution of this application;

[0050] Figure 2 A flowchart of a typical embodiment of the neural network model scheduling and deployment method of the present application;

[0051] Figure 3 A flowchart of a specific implementation method of configuring a neural network model for a working node in this application;

[0052] Figure 4 A flowchart of a specific implementation method of the present application for performing service updates on working nodes running an old version of a neural network model;

[0053] Figure 5 A flowchart formed by another specific implementation method of the present application for performing service updates on working nodes running an old version of a neural network model;

[0054] Figure 6 A flowchart of a specific implementation method of the present application for a scheduling node to process a working node registered with the scheduling node;

[0055] Figure 7 A flowchart of a specific implementation method of the present application regarding a scheduling node responding to an external service request forwarded by a gateway;

[0056] Figure 8 A principle block diagram of a typical embodiment of a neural network model scheduling and deployment device of the present application;

[0057] Fig. 9 This is a basic structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0059] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0060] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0061] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; traditional laptop and / or palmtop computers or other devices, which have and / or include radio frequency receivers. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0062] The hardware referred to by the names such as "server", "client", and "work node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0063] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0064] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud. As a business server, it can be responsible for further connecting relevant data servers and other servers that provide relevant support, etc., to form a logically related service cluster to provide services for related terminal devices such as the smartphone 81 and personal computer 82 shown in the figure or a third-party server (not shown). The smartphone and personal computer can access the Internet through a well-known network access method and establish a data communication link with the server 80 in the cloud to run terminal applications related to the services provided by the server.

[0065] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.

[0066] The application described herein refers to an application running on a server or terminal device. This application implements the relevant technical solutions of the present application in a programming manner. Its program code can be stored in a non-volatile storage medium recognizable by a computer in the form of computer executable instructions, and can be loaded into the memory by the central processing unit for execution. The relevant device of the present application is constructed by the operation of the application on the computer.

[0067] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.

[0068] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0069] See also Figure 2 A neural network model scheduling and deployment method of the present application, in its typical embodiment, comprises the following steps:

[0070] Step S11, responding to the configuration request pushed by the configuration terminal, querying the working node in the working node list whose working state is idle, wherein the working node list stores the address port information, working state and model feature information of the neural network model running by multiple working nodes, and the working node has a graphics processor for running the neural network model:

[0071] The scheduling node responds to the configuration request pushed by the configuration end, and queries the working node in the working node list whose working status is characterized as an idle state. The working node whose working status is an idle state is a working node that is not configured and runs any neural network model.

[0072] The configuration end generally refers to the terminal where the developer is located. The developer pushes the configuration request containing the configuration information defined by the developer to the scheduling node through the interface provided by the scheduling node to drive the scheduling node to respond to the configuration request and schedule the idle working node to configure the neural network model pointed to by the configuration information for operation.

[0073] The scheduling node is responsible for scheduling the neural network model pushed by the configuration end to the idle working nodes in the working node cluster for configuration and operation. The scheduling node manages the operating status, address port information and model feature information of each working node in the working node cluster through the working node list. Correspondingly, the working nodes recorded in the working node list are the working nodes pre-registered with the scheduling node.

[0074] The working node list stores the address port information, working status and model feature information of the neural network model run by each working node in the working node cluster, wherein the working status is used to characterize whether the working node is currently running any neural network model, the address port information is used to call the neural network model run by the working node to provide corresponding online services, and the model feature information records the model name and model version number of the neural network model run by the working node.

[0075] The working node is a node used to run and train the neural network model pushed by the configuration end. It has a graphics processor for running the neural network model, that is, it is composed of GPU hardware so that it has the floating-point computing power required to run the neural network model, so that it can provide the platform or platform users with corresponding online services associated with the neural network model by configuring the neural network model.

[0076] It can be understood that in addition to being responsible for scheduling idle working nodes for the configuration end to configure and run the neural network model, the scheduling node also manages the address port information, working status and model feature information of the working nodes pre-registered with it, so as to provide the platform associated with it with working status monitoring of the working node cluster and version updates of the deployed neural network model. In addition, the scheduling node is also responsible for providing the platform's service gateway with service scheduling of online services associated with the configured neural network model in the working node cluster to meet the platform's service scheduling requirements.

[0077] Step S12, pushing the configuration information contained in the configuration request to an idle working node, driving the working node to pull the neural network model corresponding to the configuration information from the model storage space for configuration operation, and then correspondingly updating the working status and model feature information of the working node in the working node list:

[0078] After the scheduling node queries the working node whose working status is idle in the working node list, it pushes the configuration information in the configuration request pushed by the configuration end to the working node, so as to drive the working node to pull the neural network model pointed to by the configuration information from the model storage space for configuration and operation.

[0079] After receiving the configuration information pushed by the scheduling node, the working node determines the model identifier contained in the configuration information, obtains the neural network model matching the model identifier from the model storage space according to the model identifier, and configures the neural network model to run in its own graphics processor. After completing the configuration of the neural network model, it pushes a configuration completion notification to the scheduling node to drive the scheduling node to update the working status and model feature information of the working node in the working node list.

[0080] After the working node obtains the neural network model from the model storage space, it will run the neural network model according to the configuration information. For example, when the configuration information indicates that it is necessary to obtain the neural network model for training, in addition to obtaining the neural network model for configuration, the working node will also obtain the training set stored corresponding to the neural network model from the model storage space for training.

[0081] The model storage space is used to store neural network models, and each neural network model has a model identifier corresponding to it, so that the working node pulls the corresponding neural network model from the model storage space according to the model identifier contained in the configuration information, and when the neural network model has not been trained to a convergence state, the training set of the neural network model will be stored in the model storage space accordingly, so that the working node that pulls the neural network model for configuration obtains the training set to train the neural network model.

[0082] The model storage space is generally deployed in the scheduling node so that the configuration end can manage the neural network model stored in the model storage space. Of course, the model storage space can also be stored in the corresponding server to save the storage space of the scheduling node. Those skilled in the art can flexibly design the deployment location of the model storage space according to the actual business scenario, which will not be elaborated here.

[0083] After the scheduling node monitors that the working node has completed the configuration of the neural network model, the working state of the working node in the working node list is modified to the working state, and the model feature information of the working node is modified to represent the neural network model currently configured to run by the working node. Specifically, the model feature information includes a model name and a model version number. The scheduling node modifies the model name to the model name of the neural network model, and modifies the model version number to the model name of the neural network model.

[0084] Step S13, monitoring the running status of the neural network model corresponding to the configuration information when the working node runs the running status, when the running status meets the running conditions preset by the configuration end, querying other working nodes whose model feature information matches the neural network model from the working node list:

[0085] After the scheduling node completes the working status and model feature information of the working node in the working node list, it will monitor the running status of the neural network model corresponding to the configuration information run by the working node. When the running status of the neural network model run by the working node meets the running conditions of the configuration-end operation, the scheduling node will query other working nodes with model feature information matching the neural network model from the working node list.

[0086] The running status generally refers to the training result of the working node training the neural network model. When the training result of the working node training the neural network model meets the training result defined by the configuration end in the configuration information, that is, the working status of the working node running the neural network model meets the running conditions preset by the configuration end, the neural network model trained by the working node will be judged by the scheduling node as having been trained to a convergence state, and other working nodes in the working node list that are also running the neural network model will be queried.

[0087] Correspondingly, the running status may also refer to the configuration status of the neural network model corresponding to the configuration information configured by the working node. Since the neural network model pulled by the working node from the model storage space may also be a neural network model that has been trained to a convergence state, the running condition preset by the configuration end is that the model configuration is completed. When the scheduling node monitors that the working node has completed the configuration of the neural network model, the scheduling node will determine that the running status of the working node meets the running condition, and the scheduling node will query the working node list for other working nodes that are also running the neural network model.

[0088] When the scheduling node monitors that the running status of the neural network model run by the working node meets the running conditions, the scheduling node queries other working nodes whose model feature information matches the neural network model from the working node list, and the model version number represented in the model feature information corresponding to the other working node is generally lower than the model version number of the neural network model run by the working node, so that the scheduling node updates the service status of the other working node. The specific updating method is described in the subsequent steps and will not be repeated in this step.

[0089] Step S14, pushing a service update instruction to the other working nodes to update the service status of the other working nodes running the neural network model:

[0090] When the scheduling node queries the working node list for other working nodes that are running the neural network model corresponding to the old version of the configuration information, the service update instruction will be pushed to the other working nodes to update the service status of the other working nodes running the neural network model.

[0091] Updating the service status of a working node running an old version of the neural network model generally refers to changing the service status of the working node to an offline state. Specifically, after the neural network model running the old version receives the service update instruction pushed by the scheduling node, it will respond to the service update instruction, stop running the neural network model, and change its service status to an offline state. Thereafter, it pushes a work update instruction to the scheduling node, driving the scheduling node to change the working status of the working node in the working node list to an idle state and initialize the model feature information.

[0092] Correspondingly, when the scheduling node pushes the service update instruction to the working node running the old version of the neural network model, in addition to modifying the service status of the working node to the offline status as mentioned above, it can also update the weight information of the neural network model running in the working node. Specifically, the scheduling node obtains the weight information of the neural network model from the working node that has completed the training or configuration of the latest version of the neural network model, and pushes the service update instruction containing the weight information to other working nodes running the old version of the neural network model, so that the other working nodes respond to the weight information obtained from the service update instruction to update the weight information in the old version of the neural network model they are currently running, and push the update completion notification to the scheduling node after completing the weight update, driving the scheduling node to update the model version number represented by the model feature information of the other working nodes in the working node list.

[0093] It can be seen from this typical embodiment that the scheduling node manages the working status, address port information and model feature information of each working node in the working node cluster through the working node list, in response to the configuration request pushed by the configuration end, and schedules the working node in the idle state to configure and run the neural network model pointed to by the request, so as to provide developers with a one-click deployment function of the neural network model. Secondly, the scheduling node monitors the running status of the neural network model run by each working node in the working node cluster in real time. When the working node running the new version of the neural network model can provide the corresponding online service, the working node running the old version of the neural network model will be offline to release the computing power resources of the working node, realize the hot update of the version of the neural network model deployed in the working node cluster, and release the algorithm resources in the cluster at the same time. In addition, the scheduling node can realize the version update of the neural network model deployed in the cluster and the common operation of multiple versions of the neural network model, so that different neural network models can share the floating-point computing power resources in the same working node cluster, and maintain various types of neural network online services for the platform.

[0094] The above typical embodiments and their variations fully disclose the implementation scheme of the neural network model scheduling and deployment method of the present application. However, various variations of the method can still be derived by transforming and amplifying some technical means. Other embodiments are briefly described as follows:

[0095] In one embodiment, please refer to Figure 3 The step of pushing the configuration information contained in the configuration request to an idle working node, driving the working node to pull the neural network model corresponding to the configuration information from the model storage space for configuration operation, and then correspondingly updating the working status and model feature information of the working node in the working node list includes the following steps performed by the working node:

[0096] Step S121, receiving the configuration information pushed by the scheduling node, querying and acquiring the neural network model corresponding to the configuration information in the model storage space for configuration:

[0097] After receiving the configuration information pushed by the scheduling node, the working node in the idle state will query the neural network model matching the configuration information in the model storage space, and obtain the neural network model for deployment.

[0098] Step S122, after completing the configuration of the neural network model, push a configuration completion notification to the scheduling node, driving the scheduling node to update the working status and model feature information of the current working node in the working node list:

[0099] After the working node completes the configuration of the neural network model that matches the configuration information, it will push the configuration completion notification to the scheduling node, so that after receiving the configuration completion notification, the scheduling node will change the working status of the working node in the working node list to a busy status, and will change the model name and model version number in the model feature information of the working node to the model name and model version number of the neural network model.

[0100] In this embodiment, the working node has the ability to automatically deploy the neural network model. After receiving the configuration information pushed by the scheduling node, it will pull the corresponding neural network model from the model storage space for deployment, and after the deployment is completed, it will push a notification to the scheduling node so that the scheduling node can update the relevant information of the working node in the working node list.

[0101] In one embodiment, please refer to Figure 4 The step of pushing the service update instruction to the other working nodes to update the service status of the other working nodes running the neural network model includes the following steps performed by the working nodes:

[0102] Step S141, responding to the service update instruction pushed by the scheduling node, stopping the currently running neural network model, and changing its own service status to an offline state:

[0103] The working node running the old version of the neural network model responds to the service update instruction pushed by the scheduling node, stops the currently running neural network model, and changes the service status to an offline status to stop the online service provided by the working node through the neural network model.

[0104] Step S142, pushing a work update instruction to the scheduling node, driving the scheduling node to update the work status and model feature information of the current working node in the working node list:

[0105] After the working node changes its service status to the offline status, it will push the work update instruction to the scheduling node, so that after the scheduling node receives the work update instruction, it will change the working status of the working node in the working node list to the idle status and initialize the model feature information of the working node.

[0106] In this embodiment, after the working nodes running the new version of the neural network model can provide corresponding online services, the scheduling node will take the working nodes running the old version of the neural network model offline and stop running the old version of the neural network model, thereby achieving hot update and releasing the computing power resources of the working node cluster.

[0107] In one embodiment, please refer to Figure 5 The step of pushing the service update instruction to the other working nodes to update the service status of the other working nodes running the neural network model includes the following steps performed by the working nodes:

[0108] Step S141', responding to the service update instruction pushed by the scheduling node, obtaining the weight information contained in the instruction and acting on the currently running neural network model, wherein the weight information is obtained by the scheduling node from the working node that has trained the latest version of the neural network model to convergence:

[0109] The scheduling node obtains the weight information of the neural network model from the working node that has completed the training or configuration of the latest version of the neural network model, and responds to the working node running the old version of the neural network model to the service update instruction pushed by the scheduling node containing the weight offline, so that the working node responds to the service update instruction, obtains the weight offline contained in the instruction, and updates the weight information of the neural network model running itself accordingly.

[0110] Step S142', update the currently running neural network model according to the weight information, and push the update completion notification to the scheduling node, driving the scheduling node to update the model version number represented by the model feature information of the current working node in the working node list:

[0111] When the working node completes the update of the weight information of the neural network model, the update completion notification will be pushed to the scheduling node so that the scheduling node can modify the model version number represented by the model notification information of the working node in the working node list to the latest model version number of the neural network model.

[0112] In this embodiment, the scheduling node updates the weight information of the old version of the neural network model run by the working node in the old version of the neural network model, so as to synchronously update the same neural network model running in the working node cluster, and then update the online service associated with the neural network model to improve the service quality of the online service.

[0113] In one embodiment, please refer to Figure 6 , the method comprises the following pre-steps:

[0114] Step S09, responding to the registration request pushed by the working node, obtaining the address and port information of the working node contained in the registration request:

[0115] The scheduling node responds to the registration request pushed by the working node that is not registered in the working node list, and obtains the address port information included in the request.

[0116] Step S10, storing the address port information into the working node list, and initializing the working state and model feature information of the working node in the working node list:

[0117] The scheduling node stores the address port information of the working node in the working node list, and initializes the working state and model feature information of the working node in the working node list. The initialized working state is characterized as an idle state, and the initialized model feature information does not contain the model name and model version number of any neural network model.

[0118] In this embodiment, a working node with computing power resources can register with the scheduling node so that the working node registers the working node into the working node list, thereby increasing the computing power resources of the working node cluster.

[0119] In one embodiment, please refer to Figure 7 , the method comprises the following post-processing steps:

[0120] Step S15, responding to the service request forwarded by the service proxy gateway, querying the working node of the model feature information corresponding to the neural network model corresponding to the request in the working node list, and querying and obtaining the preset service rule corresponding to the working node from the service rule list:

[0121] The scheduling node responds to the service request forwarded by the service proxy gateway, determines the neural network model pointed to by the service request, and searches the working node list for the working node whose model name represented by the model feature information is the working node of the neural network model.

[0122] The service proxy gateway is responsible for forwarding the service request pushed by the external request end to the scheduling node, so as to drive the scheduling node to push the address port information of the working node corresponding to the request to the service proxy gateway according to the service request, or the inference result obtained by the working node through its own neural network model. As for whether the address port information of the working node or the inference result pushed by the scheduling node to the service proxy gateway is to be used, the scheduling node will determine it according to the service rules pre-stored by the working node in the service rule list.

[0123] The service rule list stores mapping relationship data consisting of working node identifiers and service rules, so that after the scheduling node responds to the service request and queries for a matching working node, it queries and obtains the service rules pre-stored in the service rule list of the working node based on the working node identifier of the working node, and then determines the information to be pushed to the service proxy gateway based on the service rules.

[0124] Step S16, when the preset service rule is a transparent rule, the address port information of the working node is pushed to the service proxy gateway, so that the service proxy gateway calls the neural network model of the working node to perform the service through the address port information:

[0125] When the preset service rule corresponding to the working node is a transparent rule, the scheduling node will push the address port information of the working node to the service proxy gateway. After receiving the address port information, the service proxy gateway will call the neural network model run by the working node through the address port information, so as to provide corresponding online services for the external request end that pushes the service request through the neural network model.

[0126] Step S17, when the preset service rule is a security rule, the service data contained in the service request is pushed to the working node, and the inference result returned by the working node inputting the service data into its own neural network model is pushed to the service proxy gateway:

[0127] When the preset service rule corresponding to the working node is a security rule, the scheduling node pushes the service data contained in the service request to the working node, drives the working node to input the service data into the neural network model running by itself, obtains the inference result output by the neural network model according to the service data and pushes it to the scheduling node, so that the scheduling node forwards the inference result to the service proxy gateway.

[0128] In this embodiment, the scheduling node has a flexible service push mode when providing services to the outside. It controls the service push mode of the neural network model run by the working node according to the service rules preset by the working node to prevent external attacks.

[0129] Furthermore, by functionalizing each step in the method disclosed in the above embodiments, a neural network model scheduling and deployment device of the present application can be constructed. According to this idea, please refer to Figure 8 In a typical embodiment, the device includes: a request response submodule 11, which is used to respond to the configuration request pushed by the configuration end, and query the working node whose working state is idle in the working node list, wherein the working node list stores the address port information, working state and model feature information of the neural network model running by multiple working nodes, and the working node has a graphics processor for running the neural network model; a model configuration module 12, which is used to push the configuration information contained in the configuration request to the idle working node, drive the working node to pull the neural network model corresponding to the configuration information from the model storage space for configuration operation, and then update the working state and model feature information of the working node in the working node list accordingly; an operation monitoring module 13, which is used to monitor the running state of the neural network model corresponding to the configuration information running by the working node, and when the running state meets the running conditions preset by the configuration end, query other working nodes with model feature information matching the neural network model from the working node list; a service update module 14, which is used to push service update instructions to the other working nodes to update the service state of the neural network model running by the other working nodes.

[0130] In one embodiment, the model configuration module 12 includes: a model acquisition submodule, which is used to receive configuration information pushed by the scheduling node, query and obtain the neural network model corresponding to the configuration information in the model storage space for configuration; a notification push submodule, which is used to push a configuration completion notification to the scheduling node after completing the configuration of the neural network model, and drive the scheduling node to update the working status and model feature information of the current working node in the working node list.

[0131] In one embodiment, the service update module 14 includes: a service offline sub-module, which is used to respond to the service update instruction pushed by the scheduling node, stop the currently running neural network model, and change its own service status to an offline state; an instruction push sub-module, which is used to push a work update instruction to the scheduling node, driving the scheduling node to update the working status and model feature information of the current working node in the working node list.

[0132] In another embodiment, the service update module 14 also includes: a weight acquisition submodule, which is used to respond to the service update instruction pushed by the scheduling node, and obtain the weight information contained in the instruction that acts on the currently running neural network model, and the weight information is obtained by the scheduling node from the working node that trains the latest version of the neural network model to convergence; a weight update submodule, which is used to update the currently running neural network model according to the weight information, and push an update completion notification to the scheduling node, driving the scheduling node to update the model version number represented by the model feature information of the current working node in the working node list.

[0133] In order to solve the above technical problems, the embodiment of the present application also provides a computer device for running a computer program implemented according to the neural network model scheduling and deployment method. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0134] like Fig. 9 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a neural network model scheduling and deployment method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a neural network model scheduling and deployment method. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In this embodiment, the processor is used to execute the specific functions of each module / sub-module in the neural network model scheduling and deployment device of this application, and the memory stores the program code and various data required to execute the above modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / sub-modules in the neural network model scheduling and deployment device, and the server can call the server's program code and data to execute the functions of all sub-modules.

[0136] The present application also provides a non-volatile storage medium, in which the neural network model scheduling and deployment method is written into a computer program and stored in the storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means that the program is run in the computer, thereby enabling one or more processors to execute the steps of the neural network model scheduling and deployment method of any of the above-mentioned embodiments.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0138] In summary, this application can perform rapid deployment of neural network models and iterative updates of multiple versions to meet diverse business deployment needs.

[0139] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0140] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0141] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A neural network model scheduling and deployment method, characterized in that: The steps include: In response to the configuration request pushed by the configuration terminal, query the working node in the working node list whose working state is idle, wherein the working node list stores address port information, working state and model feature information of the neural network model running by multiple working nodes, and the working node has a graphics processor for running the neural network model; Push the configuration information contained in the configuration request to an idle working node, drive the working node to pull the neural network model corresponding to the configuration information from the model storage space for configuration operation, and then update the working status and model feature information of the working node in the working node list accordingly; Monitor the running status of the neural network model corresponding to the configuration information when the working node runs it, and when the running status meets the running conditions preset by the configuration end, query other working nodes whose model feature information matches the neural network model from the working node list; Pushing a service update instruction to the other working nodes to update the service status of the other working nodes running the neural network model includes the following steps performed by the working nodes: In response to the service update instruction pushed by the scheduling node, the currently running neural network model is stopped, and its own service status is changed to an offline status; Pushing a work update instruction to the scheduling node drives the scheduling node to update the work status and model feature information of the current working node in the working node list.

2. The method according to claim 1, characterized in that The step of pushing the configuration information contained in the configuration request to an idle working node, driving the working node to pull the neural network model corresponding to the configuration information from the model storage space for configuration operation, and then correspondingly updating the working status and model feature information of the working node in the working node list includes the following steps performed by the working node: Receiving configuration information pushed by the scheduling node, querying and acquiring the neural network model corresponding to the configuration information in the model storage space for configuration; After completing the configuration of the neural network model, a configuration completion notification is pushed to the scheduling node, driving the scheduling node to update the working status and model feature information of the current working node in the working node list.

3. The method according to claim 1, characterized in that The step of pushing the service update instruction to the other working nodes to update the service status of the other working nodes running the neural network model includes the following steps performed by the working nodes: In response to the service update instruction pushed by the scheduling node, obtain the weight information contained in the instruction and acting on the currently running neural network model, wherein the weight information is obtained by the scheduling node from the working node that trains the latest version of the neural network model to convergence; The currently running neural network model is updated according to the weight information, and an update completion notification is pushed to the scheduling node, driving the scheduling node to update the model version number represented by the model feature information of the current working node in the working node list.

4. The method according to any one of claims 1 to 3, characterized in that: This method includes the following preparatory steps: In response to a registration request pushed by a working node, the address and port information of the working node included in the registration request is obtained; The address port information is stored in the working node list, and the working state and model feature information of the working node in the working node list are initialized.

5. The method according to any one of claims 1 to 3, characterized in that: This method includes the following post-processing steps: In response to the service request forwarded by the service proxy gateway, query the working node of the model feature information corresponding to the neural network model corresponding to the request in the working node list, and query and obtain the preset service rule corresponding to the working node from the service rule list; When the preset service rule is a transparent rule, the address port information of the working node is pushed to the service proxy gateway so that the service proxy gateway calls the neural network model of the working node to execute the service through the address port information; When the preset service rule is a security rule, the service data contained in the service request is pushed to the working node, and the inference result returned by the working node inputting the service data into its own neural network model is pushed to the service proxy gateway.

6. A neural network model scheduling and deployment device, characterized in that: include: A request response submodule, used to respond to the configuration request pushed by the configuration terminal, query the working node in the working node list that is in an idle working state, wherein the working node list stores the address port information, working state and model feature information of the neural network model running by multiple working nodes, and the working node has a graphics processor for running the neural network model; A model configuration module, used to push the configuration information contained in the configuration request to an idle working node, drive the working node to pull the neural network model corresponding to the configuration information from the model storage space for configuration operation, and then update the working state and model feature information of the working node in the working node list accordingly; An operation monitoring module is used to monitor the operation status of the neural network model corresponding to the configuration information when the working node runs it, and when the operation status meets the operation conditions preset by the configuration end, query other working nodes whose model feature information matches the neural network model from the working node list; The service update module is used to push service update instructions to the other working nodes to update the service status of the other working nodes running the neural network model, including the following steps performed by the working nodes: In response to the service update instruction pushed by the scheduling node, the currently running neural network model is stopped, and its own service status is changed to an offline status; Pushing a work update instruction to the scheduling node drives the scheduling node to update the work status and model feature information of the current working node in the working node list.

7. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.

8. A non-volatile storage medium, characterized in that: The computer program implemented according to the method described in any one of claims 1 to 5 is stored in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images