A method and system for model control in visual Internet of Things

Abstract

This specification discloses a method and system for model control in a visual Internet of Things. In response to a task request, the real-time operating status information of each computing node is determined. The description information of the model matching the task request is determined, and the exit point set is determined based on the description information. According to the task indicator, the indication information of the exit point that meets the task indicator is determined from the exit point set, and according to the operating status information, the target computing node for running the model is determined from each computing node. Based on the indication information and description information of the exit point that meets the task indicator, a model execution strategy for the target computing node is generated. The model execution strategy is sent to the target computing node so that the target computing node executes the model task corresponding to the task request according to the exit point that meets the task indicator. The computing power of the target computing node is fully utilized to achieve efficient execution of the model task.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a method and system for model control in a visual Internet of Things. Background Art

[0002] At present, with the widespread use of artificial intelligence, machine learning has developed rapidly in various application fields. More and more devices have integrated machine learning models combined with artificial intelligence to improve the data processing capabilities of the devices. For example, in the field of visual Internet of Things (IoT), there are a large number of IoT terminals with terminal computing capabilities, and machine learning models are deployed and run on the terminals to process the captured data. In order to process the data captured by the terminal more efficiently and accurately, the data captured by the terminal or the processed data can be sent to edge servers or cloud servers with stronger computing capabilities for further processing, so that the terminals, edge servers and cloud servers can be processed collaboratively to enhance data processing efficiency and improve data utilization.

[0003] However, how to reasonably arrange the tasks of these devices to perform model operations based on the received data based on factors such as the computing power of terminals, edge servers and cloud servers, and the conditions of the communication channels between them, so as to ensure the efficient operation of the system including terminals, edge servers and cloud servers, has become a problem of widespread concern in the industry.

[0004] To this end, this specification provides a method and system for model control in a visual Internet of Things. Summary of the invention

[0005] This specification provides a method and system for model control in a visual Internet of Things to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a method for model control in a visual Internet of Things, including:

[0008] Obtaining a task request, wherein the task request includes a task indicator required to represent the model task;

[0009] In response to the task request, determining real-time operating status information of each computing node;

[0010] Determine description information of a model that matches the task request, and determine, based on the description information, a set of exit points corresponding to the model that matches the description information;

[0011] According to the task indicator, determining indication information of an exit point that meets the task indicator from the exit point set, and according to the running status information, determining a target computing node for running the model from the computing nodes, wherein the indication information of the exit point is used to determine the exit point from the exit point set;

[0012] Generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information;

[0013] Sending the model execution strategy and the description information to the target computing node, so that the target computing node executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicator and returns the model execution information of the target computing node when executing the model task;

[0014] receiving the model execution information, and adjusting the model execution strategy according to the model execution information;

[0015] The adjusted model execution strategy is sent to the target computing node, so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

[0016] Optionally, the description information further includes a segmentation point set corresponding to a model matching the description information, and different segmentation points in the segmentation point set are used to segment the model into multiple sub-models.

[0017] Optionally, before generating a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information, the method further includes:

[0018] When it is determined that there are multiple target computing nodes, determining a model segmentation ratio and a model segmentation mode according to operation status information of each target computing node;

[0019] According to the model segmentation ratio, determining, from the segmentation point set, indication information of each segmentation point that meets the model segmentation ratio, and according to the model segmentation mode, determining the order in which each target computing node runs the model, wherein the indication information of the segmentation point is used to determine the segmentation point from the segmentation point set;

[0020] For each target computing node, the sub-model corresponding to the target computing node is determined according to the determined segmentation points in the order.

[0021] Optionally, generating a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information, specifically includes:

[0022] For each target computing node, a model execution strategy for the target computing node is generated according to the indication information of the exit point that meets the task indicator in the sub-model corresponding to the target computing node and the description information of the sub-model corresponding to the target computing node.

[0023] Optionally, generating a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information, specifically includes:

[0024] Generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator, the description information, and the model execution strategy update cycle;

[0025] The model execution strategy is sent to the target computing node, so that the target computing node executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicator, specifically including:

[0026] The model execution strategy is sent to the target computing node so that the target computing node determines the model that matches the description information, executes the model task corresponding to the task request according to the instruction information of the exit point that meets the task indicators, and returns the model execution information of the target computing node when executing the model task after determining that the model execution strategy update cycle has been reached.

[0027] Optionally, receiving the model execution information and adjusting the model execution strategy according to the model execution information specifically includes:

[0028] According to the model segmentation mode, determining, from each target computing node, a target computing node that last runs the model as an uploading node;

[0029] Obtaining model execution information returned by the upload node;

[0030] The model execution strategy is adjusted according to the model execution information returned by the uploading node.

[0031] Optionally, the method further includes:

[0032] In the case where multiple target computing nodes are determined, for each target computing node, receiving model execution information returned by the target computing node when the target computing node executes the model task corresponding to the task request as real-time status information corresponding to the target computing node;

[0033] If it is determined that the target computing node meets the preset execution condition according to the real-time status information corresponding to the target computing node, a node to be replaced is determined from other target computing nodes according to the real-time status information corresponding to other target computing nodes, and an exit point corresponding to the node to be replaced is determined, and the model branch of the exit point corresponding to the node to be replaced includes the model branch of the exit point corresponding to the target computing node;

[0034] According to the exit point corresponding to the node to be replaced, the model deployed in the target computing node is adjusted to obtain an adjusted model branch including the model branch of the exit point corresponding to the node to be replaced;

[0035] Executing the model task required by the node to be replaced through the adjusted model branch, obtaining an intermediate result during the execution, and determining resource data consumed to obtain the intermediate result;

[0036] If it is determined based on the resource data that the resources consumed by the target computing node to obtain the intermediate result are less than the resources consumed by the node to be replaced to obtain the intermediate result, the model task executed by the node to be replaced is stopped, and the target computing node is used to replace the node to be replaced to continue executing the model task that needs to be executed by the node to be replaced.

[0037] This specification provides a system for model control in a visual Internet of Things, the system comprising a function orchestrator and a computing node, wherein:

[0038] The function scheduler is used to receive the task request and operation status information sent by the computing node; determine the description information of the model matching the task request, and determine the exit point set corresponding to the model matching the description information based on the description information; determine the indication information of the exit point that meets the task indicator from the exit point set based on the task indicator in the task request, and determine the target computing node that runs the model from the computing nodes based on the operation status information; generate a model execution strategy for the target computing node based on the indication information of the exit point that meets the task indicator and the description information; send the model execution strategy and the description information to the target computing node; receive model execution information, and adjust the model execution strategy based on the model execution information; send the adjusted model execution strategy to the target computing node;

[0039] The computing node is used to send the task request and the operating status information to the function orchestrator, wherein the task request includes the task indicators required to represent the model task; when being determined as the target computing node, receiving the model execution strategy and the description information for the target computing node, and executing the model task corresponding to the task request according to the indication information of the exit point that meets the task indicators based on the model execution strategy and the description information, and returning the model execution information of the target computing node when executing the model task; receiving the adjusted model execution strategy, and continuing to execute the model task according to the adjusted model execution strategy.

[0040] Optionally, the function orchestrator includes a model manager and a model scheduler, wherein:

[0041] The model scheduler is used to receive the task request and the operation status information sent by the computing node; and send the task request to the model manager;

[0042] The model manager is used to receive the task request and, based on the task request, obtain a model that matches the task request to determine the description information of the model; and return the description information to the model scheduler.

[0043] Optionally, the function orchestrator further includes a model repository, wherein:

[0044] The model manager is used to send the task request to the model management library; receive the model returned by the model repository; receive the model execution information, and adjust the model according to the model execution information, and return the description information of the adjusted model to the model scheduler;

[0045] The model repository is configured to receive the task request, determine a model matching the task request, and return the model to the model manager;

[0046] The model scheduler is used to receive the model execution information and send the model execution information to the model manager; receive the description information of the adjusted model to adjust the model execution strategy of the target computing node according to the description information of the adjusted model, so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

[0047] Optionally, the function arranger further includes a data manager, wherein:

[0048] The model manager is used to send a data acquisition request to the data manager according to the received model execution information; receive a training data set, and adjust the model according to the training data set and the model execution information, and return the description information of the adjusted model to the model scheduler, so that the model scheduler adjusts the model execution strategy of the target computing node according to the description information of the adjusted model;

[0049] The data manager is used to return the training data set to the model manager according to the data acquisition request.

[0050] Optionally, the model scheduler includes a model execution strategy generation component, a model execution strategy distribution component, a model description information distribution component, and a model performance monitoring component, wherein:

[0051] The model execution strategy generation component is used to receive the task request and the operating status information sent by the computing node; send the task request to the model manager; receive the description information returned by the model manager; determine the exit point set corresponding to the model matching the description information according to the description information; determine the indication information of the exit point that meets the task indicator from the exit point set according to the task indicator in the task request, and determine the target computing node that runs the model from the computing nodes according to the operating status information; generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information; send the model execution strategy to the model execution strategy distribution component, and send the description information to the model description information distribution component; receive the model execution information, and adjust the model execution strategy according to the model execution information; send the adjusted model execution strategy to the model execution strategy distribution component;

[0052] The model execution strategy distribution component is used to receive the model execution strategy and send the model execution strategy to the target computing node; receive the adjusted model execution strategy and send the adjusted model execution strategy to the target computing node;

[0053] The model description information distribution component is used to receive the description information and send the description information to the target computing node;

[0054] The model performance monitoring component is used to receive the model execution information returned by the target computing node; and send the model execution information to the model execution strategy generation component.

[0055] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for performing model control in a visual Internet of Things is implemented.

[0056] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for performing model control in a visual Internet of Things is implemented.

[0057] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0058] The method for model control in the visual Internet of Things provided in this specification obtains a task request, and the task request includes a task indicator required for representing the model task. In response to the task request, the real-time operating status information of each computing node is determined. The description information of the model matching the task request is determined, and the exit point set corresponding to the model matching the description information is determined according to the description information. According to the task indicator, the indication information of the exit point that meets the task indicator is determined from the exit point set, and according to the operating status information, the target computing node for running the model is determined from each computing node, wherein the indication information of the exit point is used to determine the exit point from the exit point set. According to the indication information and description information of the exit point that meets the task indicator, a model execution strategy for the target computing node is generated. The model execution strategy is sent to the target computing node so that the target computing node executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicator and returns the model execution information of the target computing node when executing the model task. The model execution information is received, and the model execution strategy is adjusted according to the model execution information. The adjusted model execution strategy is sent to the target computing node so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

[0059] Each computing node in the visual Internet of Things system can run models according to its own computing power for the acquired image data or video stream data, process these data, and improve the efficiency of data processing. In this solution, first, through the task request, determine the model that matches the task request, and determine the exit point of the model branch that meets the task indicators in the task request. After that, select the target computing node for running the model according to the running status information of each computing node. For the selected target computing node, in this solution, according to its running status information, it is assigned a suitable model or sub-model to ensure that the target computing node fully utilizes the computing power of the target computing node during the task execution process to achieve efficient execution of the model task. In addition, the target computing node can output the model output result from the exit point corresponding to the model branch through the model branch that meets the task indicator. Execute the model task while taking into account the task indicator. According to the model branch that meets the task indicator, the target computing node runs the model assigned to it, which can not only achieve the accuracy required by the task request, but also reduce the computing power consumption and energy consumption of the computing node, and improve the model operation efficiency. At the same time, the model execution strategy can be adjusted according to the model execution information returned by the target computing node to further improve the resource utilization and model operation efficiency of the target computing node. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0061] Figure 1 A flowchart of a method for performing model control in a visual Internet of Things provided in an embodiment of this specification;

[0062] Figure 2 A schematic diagram of an exit point that meets the task indicators provided in this manual;

[0063] Figure 3 A schematic diagram of a sub-model provided for this specification;

[0064] Figure 4 A schematic diagram of a division ratio provided in this manual;

[0065] Figure 5 A first schematic diagram of an exit point provided in this specification;

[0066] Figure 6 A second schematic diagram of an exit point provided in this specification;

[0067] Figure 7 A schematic diagram of a device for model control in a visual Internet of Things provided in this specification;

[0068] Figure 8 This is an interactive diagram of a visual Internet of Things system provided in this manual;

[0069] Fig. 9 A schematic diagram of a visual Internet of Things system provided in this specification;

[0070] Fig.10 A schematic diagram of a device for model control in a visual Internet of Things provided in this specification;

[0071] Fig.11 A method corresponding to the Figure 1 Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0074] Figure 1 A flow chart of a method for model control in a visual Internet of Things provided in an embodiment of this specification includes the following steps:

[0075] S100: Obtain a task request, where the task request includes task indicators required to represent a model task.

[0076] In this specification, the process of model control in the visual Internet of Things can be performed by the server to generate a model strategy and send it to the computing node to perform the model task. Of course, this specification does not limit which device implements the process. For ease of description, the following description is based on the function scheduler as the execution subject, where the function scheduler can be deployed on a cloud server and / or an edge server, which is not limited by this specification.

[0077] At present, the visual Internet of Things is mainly composed of three parts: visual data acquisition, visual data transmission, and visual data intelligent analysis. Visual IoT terminals are mainly based on visual sensors, including two-dimensional cameras, three-dimensional cameras, and point cloud cameras, supplemented by other sensors, such as radio frequency identification (RFID) electronic tags, infrared sensors, and pressure sensors. Since the data collected by visual IoT terminals, such as visual sensor data, is all-weather and huge, the visual IoT terminal processes the collected data through a model to help reduce the bandwidth required for data transmission through the network and the computing load of the edge server or cloud server. The visual IoT terminal transmits the collected image data, original video streams, or visual features obtained by its own processing to the edge server or cloud server through the network, and the edge server or cloud server can use artificial intelligence or deep learning (AI / ML) algorithms or models to process the received visual sensor data and other sensor data.

[0078] Each computing node in the visual Internet of Things may include terminal devices, edge servers, and cloud servers. In addition, each computing node may also include devices with computing capabilities such as user plane function (UPF) components and network interface cards (NIC).

[0079] In one or more embodiments of the present specification, the function arranger obtains a task request to be executed. The task request may be issued by a user, and the user issues a task request to a terminal device, an edge server, a cloud server, or other device, and the device receiving the task request sends the task request to the function arranger. Of course, the user can also send a task request directly to the function arranger, so that the function arranger can send a model that meets the task request to the computing device, and the computing device inputs the model for calculation after receiving the data.

[0080] It should be noted that the task request includes the task indicators required to represent the model task. The task indicators included in the task request are the indicators indicated by the service level agreement (SLA) requirements, and the task indicators can be indicators in terms of energy consumption, accuracy or delay. For example, the task in the task request is "video stream target recognition", and the SLA requirement is "determine whether there is a person in the video stream with an identification accuracy of 80%". The 80% identification accuracy is the task indicator. Of course, the task request can also carry the task type and task ID, such as the Internet of Things for video (IoVT) type, the Industrial Internet of Things (I-IoT) type, and the Internet of Robotic Things (IoRT) type. For example, the task type of "video stream target recognition" is IoVT. The task ID is used to distinguish different tasks according to different task IDs when the function orchestrator obtains multiple task requests.

[0081] In one or more embodiments of the present specification, the function orchestrator includes a model manager, a model scheduler, a model repository, and a data manager.

[0082] The model manager is responsible for operations related to (Deep Neural Networks, DNN) model management, including DNN model selection, optimization, generation, segmentation, etc. For example, obtain the DNN model from the model repository, optimize and train the obtained DNN model, and then store the retrained DNN model in the model repository. Obtain the training data set from the data manager and provide the DNN model description information to the model scheduler.

[0083] The model manager includes a model selection component, a model optimization component, a model generation component, and a model segmentation component. The model selection component is used to select a suitable DNN model from the model repository. The model optimization component is used to optimize the selected DNN model. Model optimization includes model compression, weight quantization, weight sparsification, channel pruning, knowledge distillation, etc. to reduce the overhead of DNN model execution. The model generation component is used to generate a new model based on the selected DNN model. Generating a new model means training and generating a new model based on the selected DNN model and the offline training data set obtained from the data manager, such as generating a multi-branch network (also known as a progressive inference network) model with multiple exit points to accelerate the execution speed of the DNN model. There are many side branches (i.e., model branches) on the main branch in the multi-branch network model, and each side branch has an exit point. The model segmentation component is used to identify all the segmentation points in the DNN model that can split the calculation between the visual IoT terminal, the edge server, and the cloud server.

[0084] The model scheduler is responsible for the generation and distribution of model execution strategies. The model scheduler includes a model execution strategy generation component and a model execution strategy distribution component. The model execution strategy generation component is used to generate model execution strategies. It can use reinforcement learning, multi-objective optimization and other methods to generate model execution strategies to adapt to the ever-changing computing resources and communication conditions. The model execution strategy distribution component is used to distribute the model execution strategy to the corresponding computing nodes. The model execution strategy includes DNN model description information. The model execution strategy distribution component interacts with each computing node (i.e., visual IoT terminal, and / or edge server, and / or cloud server) through an interface to realize the strategy distribution function. In addition, the DNN model description information is distributed to the corresponding computing nodes.

[0085] The model repository stores a collection of pre-trained DNN models.

[0086] The data manager is used to provide training datasets for DNN model training.

[0087] S102: In response to the task request, determine the real-time operating status information of each computing node.

[0088] In one or more embodiments of the present specification, the function scheduler determines the real-time operating status information of each computing node in response to the acquired task request. The real-time operating status information of each computing node includes information such as bandwidth, transmission channel quality, available storage, available computing power, and IoT terminal power.

[0089] S104: Determine description information of a model that matches the task request, and determine, based on the description information, a set of exit points corresponding to the model that matches the description information.

[0090] In one or more embodiments of the present specification, after the model scheduler in the function scheduler obtains the task request, the task request can be sent to the model manager, and the model manager determines the DNN model that matches the task request from the model repository. After that, the model manager obtains the model from the model repository, determines the description information containing the exit point set, and then returns the description information to the model scheduler. The description information includes the model number (ID), model structure, model parameters, exit point set, and other necessary data for constructing the model corresponding to the description information.

[0091] It is worth noting that the exit point set is a set of exit points of multiple model branches in a multi-branch network model with multiple exit points generated or obtained by the model manager in the function choreographer, and different model branches correspond to model output results under different task indicators. Therefore, the function choreographer can directly determine the exit point set corresponding to the model matching the description information based on the description information. Different exit points in the exit point set correspond to different model branches in the multi-branch network model, and different model branches are used to obtain model output results under different task indicators through their corresponding exit points.

[0092] S106: According to the task indicator, determine the indication information of the exit point that meets the task indicator from the exit point set, and according to the operating status information, determine the target computing node for running the model from the computing nodes, wherein the indication information of the exit point is used to determine the exit point from the exit point set.

[0093] In one or more embodiments of the present specification, the function scheduler determines the indication information of the exit point that meets the task indicator from the exit point set according to the task indicator. And, according to the operation status information of each computing node, the target computing node suitable for running the model is determined from each computing node, wherein the indication information of the exit point is used to determine the exit point from the exit point set.

[0094] It is worth noting that, from the exit point set, the indication information of the exit point that meets the task indicator is determined, and the indication information of the exit point indicates a certain exit point or certain exit points in the exit point set. The indicated exit point refers to the model output result of the model branch corresponding to the exit point that meets the task indicator and satisfies the task indicator required by the task request, that is, the exit point is the exit point that meets the task indicator. Alternatively, if the model parameters of the model branch corresponding to the exit point that meets the task indicator are the parameters required by the task indicator, the exit point can be considered to be the exit point that meets the task indicator, and then the indication information of the exit point is determined. Of course, in this specification, there is no limitation on the method of determining the indication information of the exit point that meets the task indicator, and it can be set according to the actual situation.

[0095] S108: Generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information.

[0096] S110: Send the model execution strategy and the description information to the target computing node, so that the target computing node executes the model task corresponding to the task request according to the instruction information of the exit point that meets the task indicators and returns the model execution information of the target computing node when executing the model task.

[0097] In one or more embodiments of the present specification, the function arranger generates a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information of the model. Then, the model execution strategy and the description information of the model are sent to the target computing node so that the target computing node determines the model that matches the description information. Then, according to the indication information of the exit point that meets the task indicator, the model task corresponding to the task request is executed and the model execution information of the target computing node when executing the model task is returned.

[0098] Among them, the function arranger can determine the indication information of the exit point that meets the task indicator from the exit point set according to the task indicator. When the target computing node runs the multi-branch network model, the target computing node can directly run to the model branch where the exit point that meets the task indicator is located, and then output the model output result of the model branch where the exit point is located. Even if there are other model branches before the model branch where the exit point that meets the task indicator is located, it is not necessary to execute the model branches in the order of these model branches on the main branch, but directly run the model branch where the exit point that meets the task indicator is located, saving computing consumption, delay and energy consumption at each model branch.

[0099] Figure 2 This is a schematic diagram of an exit point that meets the task indicators provided in this manual. Figure 2 As shown in Figure 1, there are multiple edge branches (i.e., model branches) on the main branch of the multi-branch network model. Figure 2 The black bold line in the middle indicates that the target computing node runs the model directly to the side branch where the exit point that meets the task indicators is located, without having to execute the side branches in the order of other side branches on the main branch, and without having to return to the main branch to continue execution.

[0100] S112: receiving the model execution information, and adjusting the model execution strategy according to the model execution information;

[0101] S114: Send the adjusted model execution strategy to the target computing node, so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

[0102] In one or more embodiments of the present specification, the function scheduler receives the model execution information returned by the target computing node, and adjusts the model execution strategy of the target computing node according to the model execution information. The model execution information includes at least the actual reasoning time, the actual computing power consumed, the actual energy consumption, the actual transmission time of the intermediate results, etc.

[0103] Afterwards, the adjusted model execution strategy is sent to the target computing node so that the target computing node continues to execute the model task according to the adjusted model execution strategy. By adjusting the model execution strategy of the target computing node through the model execution information, the model operation efficiency and computing resource utilization of the target computing node can be further improved.

[0104] based on Figure 1 The present specification provides a method for model control in a visual Internet of Things, obtaining a task request, the task request including a task indicator required for representing a model task. In response to the task request, determining the real-time operating status information of each computing node. Determine the description information of the model matching the task request, and determine the exit point set corresponding to the model matching the description information according to the description information. According to the task indicator, determine the indication information of the exit point that meets the task indicator from the exit point set, and determine the target computing node for running the model from each computing node according to the operating status information, wherein the indication information of the exit point is used to determine the exit point from the exit point set. Generate a model execution strategy for the target computing node according to the indication information and description information of the exit point that meets the task indicator. Send the model execution strategy to the target computing node so that the target computing node executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicator and returns the model execution information of the target computing node when executing the model task. Receive the model execution information, and adjust the model execution strategy according to the model execution information. Send the adjusted model execution strategy to the target computing node so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

[0105] Each computing node in the visual Internet of Things system can run models according to its own computing power for the acquired image data or video stream data, process these data, and improve the efficiency of data processing. In this solution, first, through the task request, determine the model that matches the task request, and determine the exit point of the model branch that meets the task indicators in the task request. After that, select the target computing node for running the model according to the running status information of each computing node. For the selected target computing node, in this solution, according to its running status information, it is assigned a suitable model or sub-model to ensure that the target computing node fully utilizes the computing power of the target computing node during the task execution process to achieve efficient execution of the model task. In addition, the target computing node can output the model output result from the exit point corresponding to the model branch through the model branch that meets the task indicator. Execute the model task while taking into account the task indicator. According to the model branch that meets the task indicator, the target computing node runs the model assigned to it, which can not only achieve the accuracy required by the task request, but also reduce the computing power consumption and energy consumption of the computing node, and improve the model operation efficiency. At the same time, the model execution strategy can be adjusted according to the model execution information returned by the target computing node to further improve the resource utilization and model operation efficiency of the target computing node.

[0106] In one or more embodiments of the present specification, the description information further includes a segmentation point set corresponding to the model matching the description information, and different segmentation points in the segmentation point set are used to segment the model into multiple sub-models.

[0107] Figure 3 This is a schematic diagram of a sub-model provided in this manual. Figure 3 In the multi-branch network model shown, there are two black dots on the main branch, which are the split points. Along these two split points, the model is divided into three sub-models, and each sub-model has a model branch that can be exited.

[0108] In one or more embodiments of the present specification, the function scheduler may determine, based on the operating status information of each computing node, the computing nodes that are partially or completely used to run the DNN model that matches the task request. For the computing nodes determined to run the matching DNN model, a model execution strategy is generated so that the determined computing nodes execute the model task corresponding to the task request according to the model execution strategy. Of course, the generated model execution strategy can be sent to all computing nodes, and the computing nodes that receive the model execution strategy can determine whether to run the model according to the model execution strategy. Among them, the computing nodes that are partially used to run the DNN model that matches the task request in each computing node may be only one computing node, and the model may be run only on the terminal, or only on the edge server, or only on the cloud server. In this specification, there is no restriction on what kind of device the computing node is.

[0109] In one or more examples of this specification, when it is determined that there are multiple target computing nodes, the function arranger determines the model segmentation ratio and the model segmentation mode according to the operation status information of each target computing node. According to the model segmentation ratio, the indication information of each segmentation point that meets the model segmentation ratio is determined from the segmentation point set, and according to the model segmentation mode, the order in which each target computing node runs the model is determined, wherein the indication information of the segmentation point is used to determine the segmentation point from the segmentation point set. For each target computing node, the sub-model corresponding to the target computing node is determined in order according to each determined segmentation point.

[0110] Figure 4 This is a schematic diagram of the division ratio provided in this manual. Figure 4 The DNN model shown has 10 DNN layers. When there are three target computing nodes, the model segmentation ratio is determined according to the operating status information of each target computing node, as shown in Figure 4 The model split ratio shown is 1:3:6. Therefore, there are two black dots on the main branch of the model, which are split points. Along these two split points, the model is divided into three sub-models with a model split ratio of 1:3:6.

[0111] Afterwards, in one or more instances of this specification, the function scheduler generates a model execution strategy for each target computing node based on the indication information of the exit point that meets the task indicator in the sub-model corresponding to the target computing node and the description information of the sub-model corresponding to the target computing node. For example, only the information of some layers of the DNN model that the target computing node needs to be responsible for is sent to reduce memory usage and parameter transmission.

[0112] It is worth noting that when it is determined that there are multiple target computing nodes, the following model segmentation modes may be used:

[0113] 1. The terminal and edge server work together to run the model.

[0114] 2. The terminal and cloud server work together to run the model.

[0115] 3. Edge server and cloud server collaborative operation model.

[0116] 4. The terminal, edge server and cloud server work together to run the model.

[0117] 5. Terminal and terminal collaborative operation model.

[0118] 6. The collaborative operation model of terminals, terminals and edge servers.

[0119] 7. The terminal, terminal and cloud server work together to run the model.

[0120] For the first, second, third and fifth cases, the model is run by two target computing nodes. At this time, there is a split point between the two target computing nodes running the model. The part of the model (sub-model) before the split is run on one target computing node, and the rest of the model (sub-model) after the split point is run on another target computing node.

[0121] In the fourth, sixth and seventh cases, the model is run by three target computing nodes, and there are two split points when the three target computing nodes run the model. The two split points divide the model into three sub-models, and each sub-model is executed by one target computing node.

[0122] It is worth noting that the order in which each target computing node runs the model can be determined according to the model segmentation mode. For example, for the fifth, sixth and seventh types, which involve two terminals, the model segmentation mode can be used to indicate which terminal is the leading terminal and which terminal is the post-sequence terminal.

[0123] In one or more embodiments of the present specification, the function arranger may generate a model execution policy for the target computing node based on the indication information of the exit point that meets the task indicators, the description information, and the model execution policy update cycle. The model execution policy is sent to the target computing node so that the target computing node determines the model that matches the description information, executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicators, and returns the model execution information of the target computing node when executing the model task after determining that the model execution policy update cycle has been reached. The function arranger then updates the model execution policy based on the model execution information returned by the target computing node.

[0124] It is worth noting that the target computing node executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicators, and returns the policy acquisition request to the function orchestrator after determining that the model execution policy update cycle has been reached. After receiving the policy acquisition request, the function orchestrator sends the adjusted model execution policy to the target computing node.

[0125] In one or more embodiments of the present specification, when it is determined that there are multiple indication information of exit points that meet the task indicators, the model execution strategy is sent to the target computing node, so that the target computing node obtains the output result for each exit point that meets the task indicators through the model branch corresponding to the exit point, and determines the confidence corresponding to the output result of the model branch corresponding to the exit point, and judges whether the confidence is not lower than the confidence threshold corresponding to the exit point. If so, the output result of the model branch corresponding to the exit point is used as the output result of the model. If not, the output result is obtained through the model branch corresponding to the next exit point of the exit point, and it is judged whether the confidence of the output result of the model branch corresponding to the next exit point is not lower than the confidence threshold corresponding to the next exit point.

[0126] Figure 5 This is a first schematic diagram of an exit point provided in this specification. Figure 5 As shown in the figure, there are multiple model branches (edge ​​branches) on the main branch of the multi-branch network model. Figure 5 The black bold line in the middle indicates that the target computing node runs the model to the side branch where the exit point A that meets the task indicators is located, and then determines whether the confidence of the model running result reaches the confidence threshold T1 of the exit point. If so, the running result is output and the model running is exited. If not, it returns to the main branch and continues to execute in the main branch. Since multiple exit points that meet the task indicators are pointed out, such as exit point A and the exit points after it, if exit point A does not exit the model running, the model continues to run to the side branches corresponding to the remaining exit points that meet the task indicators, and threshold judgment is performed. If the model has not stopped running after passing all exit points that meet the task indicators, it runs to the end of the main branch and outputs the model output result at the end of the main branch.

[0127] Of course, in one or more embodiments of the present specification, when it is determined that there is only one exit point that meets the task indicators, the model execution strategy is sent to the target computing node, so that the target computing node obtains the output result for the exit point that meets the task indicators through the model branch corresponding to the exit point, and determines the confidence corresponding to the output result of the model branch corresponding to the exit point, and judges whether the confidence is not lower than the confidence threshold corresponding to the exit point. If so, the output result of the model branch corresponding to the exit point is used as the output result of the model. If not, the main branch of the model continues to run.

[0128] Figure 6 This is a second schematic diagram of an exit point provided in this specification. Figure 6As shown, the computing node runs to the model branch (side branch) where the exit point that meets the task indicators is located, and then determines whether the confidence of the model running result reaches the exit point threshold T. If so, the model output result is output and the running model is exited. Otherwise, it returns to the main branch and continues to execute on the main branch. Since the exit points of the remaining side branches are not pointed out, there is no need to execute the remaining side branches. It runs to the end of the main branch, outputs the model running result and exits the running model.

[0129] In one or more embodiments of the present specification, the function scheduler may determine the target computing node that last runs the model from each target computing node according to the model segmentation mode as the upload node. Obtain the model execution information returned by the upload node. Then adjust the model execution strategy according to the model execution information returned by the upload node. It should be noted that among each target computing node, according to the order represented by the model segmentation mode, the target computing node that is ranked before the target computing node that last runs the model can be used as the preceding target computing node. The preceding target computing node that runs the model can send the model execution information of the preceding target computing node to the target computing node that last runs the model (i.e., the upload node). The upload node processes and calculates the model execution information of all target computing nodes (including the preceding nodes and the upload node) to obtain the model execution performance and returns it to the function scheduler.

[0130] In one or more embodiments of the present specification, when multiple target computing nodes are determined, the function scheduler receives, for each target computing node, model execution information returned by the target computing node when the target computing node executes the model task corresponding to the task request, as the real-time status information corresponding to the target computing node. The real-time status information may be the actual reasoning time, actual computing power, actual energy consumption, actual transmission time of intermediate results, or processor utilization rate consumed by the target computing node when executing the model task corresponding to the task request.

[0131] If it is determined that the target computing node meets the preset execution conditions according to the real-time status information corresponding to the target computing node, then the node to be replaced is determined from the other target computing nodes according to the real-time status information corresponding to other target computing nodes, and the exit point corresponding to the node to be replaced is determined, and the model branch of the exit point corresponding to the node to be replaced includes the model branch corresponding to the exit point of the target computing node. Among them, the target computing node meets the preset execution conditions can be that the processor utilization rate of the target computing node when executing the model task corresponding to the task request is less than the preset value, then the computing power of the target computing node is not fully utilized, and there is still remaining computing power. Of course, the preset execution conditions are not limited in this specification. According to the real-time status information corresponding to other target computing nodes, the node to be replaced that does not meet the preset execution conditions (such as the processor utilization rate is greater than the preset value) can be determined from other target computing nodes.

[0132] According to the exit point corresponding to the node to be replaced, the model deployed in the target computing node is adjusted to obtain an adjusted model branch including the model branch of the exit point corresponding to the node to be replaced.

[0133] Through the adjusted model branch, the model task required by the node to be replaced is executed, and the intermediate results in the execution process are obtained, and the resource data consumed to obtain the intermediate results are determined. The resource data may include hardware resource data, such as processor utilization, etc., and may also include software resource data, such as the calculation time for calculating the intermediate results, etc.

[0134] If it is determined based on the resource data that the resources consumed by the target computing node to obtain the intermediate result are less than the resources consumed by the node to be replaced, the model task executed by the node to be replaced is stopped, and the target computing node replaces the node to be replaced to continue executing the model task required by the node to be replaced.

[0135] It can make the model tasks of other computing nodes with insufficient computing power continue to run when the computing power of the target computing node remains surplus, so that the model tasks can run efficiently and the model running performance is improved.

[0136] In one or more embodiments of the present specification, the model scheduler may further include a model description information distribution component and a model performance monitoring component. The model description information distribution component is used to distribute the DNN model description information to the corresponding computing nodes. The model description information distribution component interacts with each computing node (i.e., a visual IoT terminal, and / or an edge server, and / or a cloud server) through an interface to implement the model distribution function.

[0137] Therefore, the model description information distribution component can share the same interface with the model execution strategy distribution component to achieve synchronization of DNN model description information and model execution strategy between various computing nodes (i.e., visual IoT terminals, and / or edge servers, and / or cloud servers).

[0138] The model performance monitoring component is used to monitor the model execution performance. It obtains real-time operating status information and model execution information from the computing nodes, such as actual inference time, actual computing power consumed, actual energy consumption, actual transmission time of intermediate results, etc. When the model execution performance decreases, the model execution strategy generation component regenerates a new model execution strategy. Model execution performance can be measured by AI service quality (Quality of AI Service, QoAIS) indicators, such as the execution percentage that meets the target SLA requirements (in terms of energy consumption, accuracy and latency) over a period of time. After processing the information obtained from the computing nodes, the model performance monitoring component obtains the model execution performance (QoAIS indicators, etc.), and feeds back the model execution performance to the model manager, so that when the model execution performance decreases, the model manager can perform corresponding model reselection, optimization, regeneration, and divide new split points and exit points.

[0139] In one or more embodiments of the present specification, the function scheduler receives the model execution information returned by each computing node. According to the model execution information, the model execution strategy is adjusted. By dynamically adjusting the model execution strategy, the performance of the entire system can be improved, the response is faster, and it is more flexible.

[0140] In one or more embodiments of this specification, the correspondence between DNN model description information and model execution strategy is one-to-many, and the DNN model description information ID and the model execution strategy ID can be bound. The DNN model description information is bound to which model execution strategy, and the computing node that subsequently receives the model execution strategy can execute tasks according to the bound DNN model description information and the model execution strategy.

[0141] In one or more embodiments of the present specification, when the model execution strategy is issued, it can be issued layer by layer from the cloud server to the edge server and then to the visual IoT terminal. For example, the function orchestrator can be deployed in the cloud, that is, the model manager and the model scheduler are also deployed in the cloud, and the model execution strategy can be transmitted from the cloud server to the edge server through the network between the cloud and the edge, and then transmitted from the edge server to the visual IoT terminal through the network between the edge and the end. In addition, there may also be end-to-end transmission to achieve synchronization of DNN model description information and model execution strategies between computing nodes. Wireless communication networks for data transmission between edge and end include Zigbee protocol (Zigbee), 4th Generation Mobile Communication Technology (4G) and 5th Generation Mobile Communication Technology (5G), etc. Wired transmission is usually used between cloud-edge and edge-edge.

[0142] In one or more embodiments of the present specification, the model manager, model scheduler, model repository, and data manager can be integrated into the function orchestrator and thus deployed in the visual IoT system. In actual deployment, the function orchestrator can be deployed entirely on the cloud server. Or, all on the edge server. Or, part of it on the cloud server and part on the edge server. Or, part of it on the cloud server, part of it on the edge server, and the rest on the visual IoT terminal. There is an interface between the function orchestrator and the cloud server, edge server, and visual IoT terminal for interaction. Even on network computing devices with computing-network fusion functions (such as smart network cards and smart base stations), signaling interaction can be performed with the model scheduler to perform intelligent analysis tasks of the video IoT such as in-network computing offloading and diversion.

[0143] The above is a task execution method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding system structure diagram for model control in the visual Internet of Things. Figure 7 shown.

[0144] Figure 7 This is a system structure diagram for model control in the visual Internet of Things provided in this specification. The system includes a function arranger and a computing node, wherein:

[0145] In one or more embodiments of the present specification, the computing node is used to send a task request and the operating status information to the function scheduler, and the task request includes the task indicator required to represent the model task. Of course, the computing node includes multiple computing nodes such as cloud servers, edge servers, and terminal devices.

[0146] In one or more embodiments of the present specification, the function scheduler is used to receive a task request and operating status information sent by a computing node. Determine the description information of the model that matches the task request, and determine the exit point set corresponding to the model that matches the description information based on the description information. According to the task indicator in the task request, determine the indication information of the exit point that meets the task indicator from the exit point set, and determine the target computing node for running the model from the computing node based on the operating status information. Generate a model execution strategy for the target computing node based on the indication information and description information of the exit point that meets the task indicator. Send the model execution strategy and description information to the target computing node.

[0147] The computing node, when determined as a target computing node, receives the model execution strategy and description information for the target computing node, executes the model task corresponding to the task request according to the model execution strategy and description information and the indication information of the exit point that meets the task indicator, and returns the model execution information of the target computing node when executing the model task.

[0148] The function orchestrator is used to receive model execution information and adjust the model execution strategy according to the model execution information, and send the adjusted model execution strategy to the target computing node.

[0149] The target computing node receives the adjusted model execution strategy and continues to execute the model task according to the adjusted model execution strategy.

[0150] The specific process of the above steps can be referred to the content of the aforementioned method, which will not be repeated here.

[0151] Figure 7In the system for model control in the visual Internet of Things, a function scheduler obtains a task request, and the task request includes a task indicator required for representing the model task. In response to the task request, the real-time operating status information of each computing node is determined. The description information of the model matching the task request is determined, and the exit point set corresponding to the model matching the description information is determined according to the description information. According to the task indicator, the indication information of the exit point that meets the task indicator is determined from the exit point set, and according to the operating status information, the target computing node for running the model is determined from each computing node. According to the indication information and description information of the exit point that meets the task indicator, a model execution strategy for the target computing node is generated. The model execution strategy is sent to the target computing node so that the target computing node executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicator and returns the model execution information of the target computing node when executing the model task. The model execution information is received, and the model execution strategy is adjusted according to the model execution information. The adjusted model execution strategy is sent to the target computing node so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

[0152] Each computing node in the visual Internet of Things system can run models according to its own computing power for the acquired image data or video stream data, process these data, and improve the efficiency of data processing. In this solution, first, through the task request, determine the model that matches the task request, and determine the exit point of the model branch that meets the task indicators in the task request. After that, select the target computing node for running the model according to the running status information of each computing node. For the selected target computing node, in this solution, according to its running status information, it is assigned a suitable model or sub-model to ensure that the target computing node fully utilizes the computing power of the target computing node during the task execution process to achieve efficient execution of the model task. In addition, the target computing node can output the model output result from the exit point corresponding to the model branch through the model branch that meets the task indicator. Execute the model task while taking into account the task indicator. According to the model branch that meets the task indicator, the target computing node runs the model assigned to it, which can not only achieve the accuracy required by the task request, but also reduce the computing power consumption and energy consumption of the computing node, and improve the model operation efficiency. At the same time, the model execution strategy can be adjusted according to the model execution information returned by the target computing node to further improve the resource utilization and model operation efficiency of the target computing node.

[0153] In one or more embodiments of the present specification, the function scheduler includes a model manager and a model scheduler. The model scheduler is used to receive task requests and operation status information sent by the computing node and send the task request to the model manager.

[0154] The model manager is used to receive the task request and obtain the model matching the task request according to the task request to determine the description information of the model and return the description information to the model scheduler.

[0155] In one or more embodiments of the present specification, the function composer further includes a model repository.

[0156] The model manager is used to send the task request to the model management library, receive the model returned by the model repository, receive model execution information, and adjust the model according to the model execution information, and return the description information of the adjusted model to the model scheduler.

[0157] The model repository is used to receive task requests, determine the models that match the task requests, and return the models to the model manager.

[0158] The model scheduler is used to receive the model execution information and send the model execution information to the model manager. The model description information after adjustment is received to adjust the model execution strategy of the target computing node according to the description information after adjustment, so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

[0159] In one or more embodiments of the present specification, the function arranger further includes a data manager.

[0160] The model manager is used to send a data acquisition request to the data manager according to the received model execution information. The model manager receives the training data set, adjusts the model according to the training data set and the model execution information, and returns the description information of the adjusted model to the model scheduler, so that the model scheduler adjusts the model execution strategy of the target computing node according to the description information of the adjusted model.

[0161] The data manager is used to return the training data set to the model manager based on the data acquisition request.

[0162] In one or more embodiments of the present specification, the model scheduler includes a model execution strategy generation component, a model execution strategy distribution component, a model description information distribution component, and a model performance monitoring component, wherein:

[0163] The model execution strategy generation component is used to receive the task request and operation status information sent by the computing node. Send the task request to the model manager. Receive the description information returned by the model manager. According to the description information, determine the exit point set corresponding to the model matching the description information. According to the task indicator in the task request, determine the indication information of the exit point that meets the task indicator from the exit point set, and according to the operation status information, determine the target computing node for running the model from the computing nodes. Generate a model execution strategy for the target computing node based on the indication information and description information of the exit point that meets the task indicator. Send the model execution strategy to the model execution strategy distribution component, and send the description information to the model description information distribution component. Receive the model execution information, and adjust the model execution strategy according to the model execution information. Send the adjusted model execution strategy to the model execution strategy distribution component.

[0164] The model execution strategy distribution component is used to receive the model execution strategy and send the model execution strategy to the target computing node. The model execution strategy after adjustment is received and sent to the target computing node.

[0165] The model description information distribution component is used to receive the description information and send the description information to the target computing node.

[0166] The model performance monitoring component is used to receive the model execution information returned by the target computing node and send the model execution information to the model execution strategy generation component. Of course, the model execution information can also be processed by calculation to obtain the model execution performance and send it to the model execution strategy generation component and the model manager.

[0167] For the specific contents about the model scheduler, model manager, model repository and data manager, please refer to the contents in the aforementioned method and will not be repeated here.

[0168] Figure 8 This is a schematic diagram of the interaction of a visual Internet of Things system provided in this manual. Figure 8In the pre-step shown, the terminal device, edge server and cloud server send the task request and real-time operation status information to the model scheduler. The model scheduler can obtain the description information of the DNN model matching the task request through the model manager according to the task request. Then generate the model execution strategy according to the description information and the operation status of each computing node. The model scheduler sends the generated model execution strategy to the computing node, synchronizes the DNN model description information and the model execution strategy. The computing node can execute the model task according to the synchronized DNN model description information and model execution strategy, run the model to the split point or exit point, and output the intermediate result. Then the next computing node continues to run the model, execute the model task, and output the model output result. After that, the computing node can report the model execution information to the model scheduler, and the model scheduler can adjust the model execution strategy. The model scheduler can also feedback the model execution information to the model manager, so that the model manager can optimize, generate or split the DNN model according to the model execution information, obtain a new DNN model and store the new DNN model in the model repository. The model manager then returns the description information of the new DNN model to the model scheduler so that the model scheduler can adjust the model execution strategy. The computing node can report the model execution information in real time, for example, while outputting the intermediate results or the final model output results, or immediately afterwards. The model execution information can also be reported after reaching the reporting period (for example, the reporting period can be equal to the model execution strategy update period). The model execution information can also be reported after the trigger condition is met, for example, when the computing node enters a low-power state. Similarly, the model scheduler can also update the model execution strategy in real time or periodically or when the trigger condition (model execution performance decreases) is met.

[0169] Fig. 9 This is a schematic diagram of a visual Internet of Things system provided in this manual. Fig. 9 As shown, the model manager includes a model selection component, a model optimization component, a model generation component, and a model segmentation component, and the model scheduler includes a model execution strategy generation component, a model execution strategy distribution component, a model description information distribution component, and a model performance monitoring component. The computing nodes include cloud servers, edge servers, and terminal devices. The description of the above components can refer to the content of the above method, which will not be repeated here. Fig. 9 The model manager can obtain DNN models from the model repository and store DNN models. The model manager can also obtain training data sets from the data manager to facilitate model training. The model scheduler can obtain DNN model description information from the model manager, feedback model execution information to the model manager, and send DNN model description information and model execution strategy to the computing nodes, and receive real-time operating status information and model execution information reported by the computing nodes.

[0170] This specification also provides a schematic diagram of a device for model control in a visual Internet of Things, such as Fig.10 shown.

[0171] Fig.10 A schematic diagram of a device for model control in a visual Internet of Things provided in this specification specifically includes:

[0172] An acquisition module 1000 is used to acquire a task request, wherein the task request includes a task indicator required to represent a model task;

[0173] A first determining module 1002 is used to determine the real-time operating status information of each computing node in response to the task request;

[0174] The second determination module 1004 is used to determine the description information of the model matching the task request, and determine the exit point set corresponding to the model matching the description information according to the description information;

[0175] A third determination module 1006 is used to determine, according to the task indicator, from the exit point set, indication information of an exit point that meets the task indicator, and to determine, according to the running status information, a target computing node for running the model from the computing nodes, wherein the indication information of the exit point is used to determine the exit point from the exit point set;

[0176] A first generating module 1008, configured to generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information;

[0177] The first sending module 1010 is used to send the model execution strategy and the description information to the target computing node, so that the target computing node executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicator and returns the model execution information of the target computing node when executing the model task;

[0178] A second generating module 1012 is used to receive the model execution information and adjust the model execution strategy according to the model execution information;

[0179] The second sending module 1014 is used to send the adjusted model execution strategy to the target computing node, so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

[0180] Optionally, the first generation module 1008 is used to determine the model segmentation ratio and the model segmentation mode according to the operating status information of each target computing node when it is determined that there are multiple target computing nodes; determine the indication information of each segmentation point that meets the model segmentation ratio from the segmentation point set according to the model segmentation ratio; and determine the order in which each target computing node runs the model according to the model segmentation mode, wherein the indication information of the segmentation point is used to determine the segmentation point from the segmentation point set, and for each target computing node, determine the sub-model corresponding to the target computing node in order according to the determined segmentation points, and the description information also includes a segmentation point set corresponding to the model matching the description information, and different segmentation points in the segmentation point set are used to segment the model into multiple sub-models.

[0181] Optionally, the first generation module 1008 is used to generate a model execution strategy for each target computing node based on the indication information of the exit point that meets the task indicators in the sub-model corresponding to the target computing node and the description information of the sub-model corresponding to the target computing node.

[0182] Optionally, the first generating module 1008 is used to generate the model execution strategy of the target computing node according to the indication information of the exit point that meets the task indicator, the description information and the model execution strategy update cycle;

[0183] The first sending module 1010 is used to send the model execution strategy to the target computing node so that the target computing node determines the model that matches the description information, executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicators, and returns the model execution information of the target computing node when executing the model task after determining that the model execution strategy update cycle has been reached.

[0184] Optionally, the second generation module 1012 is used to determine the target computing node that finally runs the model from each target computing node according to the model segmentation mode, as the upload node, obtain the model execution information returned by the upload node, and adjust the model execution strategy according to the model execution information returned by the upload node.

[0185] Optionally, the device further includes a replacement module 1014;

[0186] The replacement module 1014 is used for, when multiple target computing nodes are determined, receiving, for each target computing node, model execution information returned by the target computing node when the target computing node executes the model task corresponding to the task request, as the real-time status information corresponding to the target computing node; if it is determined that the target computing node meets the preset execution conditions according to the real-time status information corresponding to the target computing node, then according to the real-time status information corresponding to other target computing nodes, determining the node to be replaced from other target computing nodes, and determining the exit point corresponding to the node to be replaced, the model branch corresponding to the exit point of the node to be replaced contains the model branch corresponding to the exit point of the target computing node, according to the real-time status information corresponding to the target computing node The exit point corresponding to the node to be replaced is used to adjust the model deployed in the target computing node to obtain an adjusted model branch including the model branch corresponding to the exit point of the node to be replaced; the model task required by the node to be replaced is executed through the adjusted model branch, and the intermediate result in the execution process is obtained, and the resource data consumed for obtaining the intermediate result is determined; if it is determined according to the resource data that the resources consumed by the target computing node to obtain the intermediate result are less than the resources consumed by the node to be replaced to obtain the intermediate result, the model task executed by the node to be replaced is stopped, and the target computing node is used to replace the node to be replaced to continue to execute the model task required by the node to be replaced.

[0187] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A method for model control in a visual Internet of Things is provided.

[0188] This manual also provides Fig.11 The schematic structure diagram of the electronic device shown in FIG. Fig.11 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The method for model control in a visual Internet of Things.

[0189] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0190] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0191] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0192] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0193] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0194] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0195] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0196] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0197] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0198] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0199] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0200] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0201] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0202] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0204] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0205] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for model control in a visual Internet of Things, characterized in that: include: Obtaining a task request, wherein the task request includes a task indicator required to represent the model task; In response to the task request, determining real-time operating status information of each computing node; Determine description information of a model that matches the task request, and determine, based on the description information, a set of exit points corresponding to the model that matches the description information; According to the task indicator, determining indication information of an exit point that meets the task indicator from the exit point set, and according to the running status information, determining a target computing node for running the model from the computing nodes, wherein the indication information of the exit point is used to determine the exit point from the exit point set; Generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information; Sending the model execution strategy and the description information to the target computing node, so that the target computing node executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicator and returns the model execution information of the target computing node when executing the model task; receiving the model execution information, and adjusting the model execution strategy according to the model execution information; The adjusted model execution strategy is sent to the target computing node, so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

2. The method according to claim 1, characterized in that The description information also includes a segmentation point set corresponding to the model matching the description information, and different segmentation points in the segmentation point set are used to segment the model into multiple sub-models.

3. The method according to claim 2, characterized in that Before generating a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information, the method further includes: When it is determined that there are multiple target computing nodes, determining a model segmentation ratio and a model segmentation mode according to operation status information of each target computing node; According to the model segmentation ratio, determining, from the segmentation point set, indication information of each segmentation point that meets the model segmentation ratio, and according to the model segmentation mode, determining the order in which each target computing node runs the model, wherein the indication information of the segmentation point is used to determine the segmentation point from the segmentation point set; For each target computing node, the sub-model corresponding to the target computing node is determined according to the determined segmentation points in the order.

4. The method according to claim 3, characterized in that Generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information, specifically including: For each target computing node, a model execution strategy for the target computing node is generated according to the indication information of the exit point that meets the task indicator in the sub-model corresponding to the target computing node and the description information of the sub-model corresponding to the target computing node.

5. The method according to claim 1 or 4, characterized in that Generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information, specifically including: Generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator, the description information, and the model execution strategy update cycle; The model execution strategy is sent to the target computing node, so that the target computing node executes the model task corresponding to the task request according to the indication information of the exit point that meets the task indicator, specifically including: The model execution strategy is sent to the target computing node so that the target computing node determines the model that matches the description information, executes the model task corresponding to the task request according to the instruction information of the exit point that meets the task indicators, and returns the model execution information of the target computing node when executing the model task after determining that the model execution strategy update cycle has been reached.

6. The method according to claim 3, characterized in that Receiving the model execution information and adjusting the model execution strategy according to the model execution information specifically includes: According to the model segmentation mode, determining, from each target computing node, a target computing node that last runs the model as an uploading node; Obtaining model execution information returned by the upload node; The model execution strategy is adjusted according to the model execution information returned by the uploading node.

7. The method according to claim 1, characterized in that The method further comprises: In the case where multiple target computing nodes are determined, for each target computing node, receiving model execution information returned by the target computing node when the target computing node executes the model task corresponding to the task request as real-time status information corresponding to the target computing node; If it is determined that the target computing node meets the preset execution condition according to the real-time status information corresponding to the target computing node, a node to be replaced is determined from other target computing nodes according to the real-time status information corresponding to other target computing nodes, and an exit point corresponding to the node to be replaced is determined, and the model branch of the exit point corresponding to the node to be replaced includes the model branch of the exit point corresponding to the target computing node; According to the exit point corresponding to the node to be replaced, the model deployed in the target computing node is adjusted to obtain an adjusted model branch including the model branch of the exit point corresponding to the node to be replaced; Executing the model task required by the node to be replaced through the adjusted model branch, obtaining an intermediate result during the execution, and determining resource data consumed to obtain the intermediate result; If it is determined based on the resource data that the resources consumed by the target computing node to obtain the intermediate result are less than the resources consumed by the node to be replaced to obtain the intermediate result, the model task executed by the node to be replaced is stopped, and the target computing node is used to replace the node to be replaced to continue executing the model task that needs to be executed by the node to be replaced.

8. A system for model control in a visual Internet of Things, characterized in that: The system includes a function orchestrator and a computing node, wherein: The function scheduler is used to receive the task request and operation status information sent by the computing node; determine the description information of the model matching the task request, and determine the exit point set corresponding to the model matching the description information based on the description information; determine the indication information of the exit point that meets the task indicator from the exit point set based on the task indicator in the task request, and determine the target computing node that runs the model from the computing nodes based on the operation status information; generate a model execution strategy for the target computing node based on the indication information of the exit point that meets the task indicator and the description information; send the model execution strategy and the description information to the target computing node; receive model execution information, and adjust the model execution strategy based on the model execution information; send the adjusted model execution strategy to the target computing node; The computing node is used to send the task request and the operating status information to the function orchestrator, wherein the task request includes the task indicators required to represent the model task; when being determined as the target computing node, receiving the model execution strategy and the description information for the target computing node, and executing the model task corresponding to the task request according to the indication information of the exit point that meets the task indicators based on the model execution strategy and the description information, and returning the model execution information of the target computing node when executing the model task; receiving the adjusted model execution strategy, and continuing to execute the model task according to the adjusted model execution strategy.

9. The system according to claim 8, characterized in that The function composer includes a model manager and a model scheduler, wherein: The model scheduler is used to receive the task request and the operation status information sent by the computing node; and send the task request to the model manager; The model manager is used to receive the task request and, based on the task request, obtain a model that matches the task request to determine the description information of the model; and return the description information to the model scheduler.

10. The system according to claim 9, characterized in that The function orchestrator also includes a model repository, wherein: The model manager is used to send the task request to the model management library; receive the model returned by the model repository; receive the model execution information, and adjust the model according to the model execution information, and return the description information of the adjusted model to the model scheduler; The model repository is configured to receive the task request, determine a model matching the task request, and return the model to the model manager; The model scheduler is used to receive the model execution information and send the model execution information to the model manager; receive the description information of the adjusted model to adjust the model execution strategy of the target computing node according to the description information of the adjusted model, so that the target computing node continues to execute the model task according to the adjusted model execution strategy.

11. The system according to claim 10, characterized in that The function composer also includes a data manager, wherein: The model manager is used to send a data acquisition request to the data manager according to the received model execution information; receive a training data set, and adjust the model according to the training data set and the model execution information, and return the description information of the adjusted model to the model scheduler, so that the model scheduler adjusts the model execution strategy of the target computing node according to the description information of the adjusted model; The data manager is used to return the training data set to the model manager according to the data acquisition request.

12. The system according to claim 9, characterized in that The model scheduler includes a model execution strategy generation component, a model execution strategy distribution component, a model description information distribution component and a model performance monitoring component, wherein: The model execution strategy generation component is used to receive the task request and the operating status information sent by the computing node; send the task request to the model manager; receive the description information returned by the model manager; determine the exit point set corresponding to the model matching the description information according to the description information; determine the indication information of the exit point that meets the task indicator from the exit point set according to the task indicator in the task request, and determine the target computing node that runs the model from the computing nodes according to the operating status information; generate a model execution strategy for the target computing node according to the indication information of the exit point that meets the task indicator and the description information; send the model execution strategy to the model execution strategy distribution component, and send the description information to the model description information distribution component; receive the model execution information, and adjust the model execution strategy according to the model execution information; send the adjusted model execution strategy to the model execution strategy distribution component; The model execution strategy distribution component is used to receive the model execution strategy and send the model execution strategy to the target computing node; receive the adjusted model execution strategy and send the adjusted model execution strategy to the target computing node; The model description information distribution component is used to receive the description information and send the description information to the target computing node; The model performance monitoring component is used to receive the model execution information returned by the target computing node; and send the model execution information to the model execution strategy generation component.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images