Distributed model reasoning method and device, equipment and storage medium

Abstract

The invention relates to a distributed model reasoning method, device and equipment and a storage medium, the method is applied to center equipment of a distributed network system, and the distributed network system comprises one piece of center equipment and a plurality of pieces of edge equipment; comprising the following steps: dividing a to-be-distributed model according to structure information of the to-be-distributed model to obtain a plurality of sub-models, and distributing each sub-model to at least one edge device; wherein the structure information represents a hierarchical structure between the model layers, and the plurality of sub-models form a to-be-distributed model; if the model reasoning request is responded, obtaining resource information of each edge device; wherein the resource information represents the operation state of the edge device; determining a target device from the at least one edge device allocated by the sub-model according to the resource information of each edge device; and controlling the target equipment to execute the corresponding sub-model according to the execution sequence among the sub-models. The operation efficiency of the model is improved, and full utilization of resources is realized.

Description

[0001] The present application claims priority to the Chinese patent application No. 202510522071.6, filed on April 24, 2025, and entitled "Distributed-based model inference method, device, equipment and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, in particular to a distributed-based model inference method, device, equipment and storage medium. BACKGROUND

[0003] Deep Neural Networks (DNN) as an important basis for contemporary artificial intelligence technology has been widely used in face recognition, video analysis, natural language processing and other fields. Through the training of complex multi-layer network structure and massive parameters, the structure of DNN model is becoming more and more complex. However, the improvement of DNN model performance also brings huge computing and resource demand.

[0004] When running DNN tasks on devices with limited computing power, high latency and high energy consumption problems often occur. Therefore, it is necessary to deploy DNN model specifically to realize the full use of resources and meet the requirements of model real-time and efficient processing. SUMMARY

[0005] The purpose of the present application is to provide a distributed-based model inference method, device, equipment and storage medium to realize the full use of resources and improve the inference efficiency of the model.

[0006] In a first aspect, the present application provides a distributed-based model inference method, which is applied to a center device of a distributed network system, and the distributed network system includes a center device and a plurality of edge devices; the method includes:

[0007] According to the structure information of the to-be-assigned model, the to-be-assigned model is divided to obtain a plurality of sub-models, and each sub-model is assigned to at least one edge device; wherein the structure information represents the hierarchical structure between model layers, and the plurality of sub-models constitute the to-be-assigned model;

[0008] If a model inference request is responded to, the resource information of each edge device is obtained; wherein the resource information represents the running state of the edge device;

[0009] According to the resource information of each edge device, a target device is determined from at least one edge device to which the sub-model is assigned;

[0010] The target device is controlled to execute corresponding sub-models according to an execution order between the sub-models.

[0011] In a second aspect, the present application provides a distributed model inference device, which is applied to a center device of a distributed network system, and the distributed network system comprises a center device and a plurality of edge devices; the device comprises:

[0012] a model division unit, configured to perform division processing on a to-be-allocated model according to structure information of the to-be-allocated model, to obtain a plurality of sub-models, and allocate each sub-model to at least one edge device; wherein the structure information represents a hierarchical structure between model layers, and the plurality of sub-models constitute the to-be-allocated model;

[0013] a resource acquisition unit, configured to acquire resource information of each edge device in response to a model inference request; wherein the resource information represents a running state of the edge device;

[0014] a device determination unit, configured to determine a target device from the at least one edge device to which the sub-model is allocated according to the resource information of each edge device;

[0015] a model execution unit, configured to control the target device to execute corresponding sub-models according to an execution order between the sub-models.

[0016] In a third aspect, the present application provides an electronic device, comprising a processor and a memory connected with the processor in communication;

[0017] The memory stores computer execution instructions;

[0018] The processor executes the computer execution instructions stored in the memory to realize the method of the first aspect.

[0019] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer execution instructions, and the computer execution instructions are executed by a processor to realize the method of the first aspect.

[0020] In a fifth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the method of the first aspect.

[0021] The application provides a distributed model reasoning method, device, equipment and storage medium, by determining the structure information of a to-be-distributed model, the to-be-distributed model can be divided into multiple sub-models, and the to-be-distributed model is split. Each sub-model is distributed to at least one edge device, that is, the same sub-model can be loaded in multiple edge devices, and the normal operation of the subsequent sub-model is ensured. After responding to the model reasoning request, the resource information of each edge device is obtained, and according to the resource information of each edge device, one edge device is determined from the multiple edge devices to which the sub-model is distributed, as the target device of the sub-model, that is, the best device for executing the sub-model, and the effective use of the computing resource is realized. According to the execution order between the sub-models, the target devices are controlled in sequence to execute the corresponding sub-models, and the reasoning of the to-be-distributed model is realized. The application can improve the operation efficiency of the model by splitting and distributing the model and selecting the edge device, effectively reduces the communication overhead, and realizes the dual improvement of resource utilization and performance. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0023] Figure 1 A flowchart of a distributed model reasoning method provided by an embodiment of the application is shown in the figure.

[0024] Figure 2 A structure diagram of a distributed network system provided by an embodiment of the application is shown in the figure.

[0025] Figure 3 A flowchart of a distributed model reasoning method provided by an embodiment of the application is shown in the figure.

[0026] Figure 4 A structure topology diagram provided by an embodiment of the application is shown in the figure.

[0027] Figure 5 A subgraph division result diagram provided by an embodiment of the application is shown in the figure.

[0028] Figure 6 A flowchart of a distributed model reasoning method provided by an embodiment of the application is shown in the figure.

[0029] Figure 7 A structure block diagram of a distributed model reasoning device provided by an embodiment of the application is shown in the figure.

[0030] Figure 8 A structure block diagram of a distributed model reasoning device provided by an embodiment of the application is shown in the figure.

[0031] Figure 9 A structural block diagram of an electronic device according to an embodiment of the present application;

[0032] Figure 10 A structural block diagram of an electronic device according to an embodiment of the present application.

[0033] The above-described embodiments of the present application have been shown and described, and it will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments without departing from the spirit and scope of the application. Thus, it is intended that the present application covers the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents. DETAILED DESCRIPTION

[0034] The present application can be implemented in various ways, and numerous modifications and variations can be made to the preferred embodiments without departing from the spirit and scope of the present application. Therefore, it is to be understood that the preferred embodiments are only for illustrating the present application and are not for limiting the scope of the present application.

[0035] It is to be understood that the above-mentioned embodiments are only for illustrating the basic concept of the present application, and thus the drawings only show the components related to the present application, not the components in the actual implementation, shapes, and sizes. The shapes, numbers, and proportions of the components in the actual implementation can be changed arbitrarily, and the layout of the components can be more complex.

[0036] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, "multiple" means two or more, unless otherwise specified. "And / or" describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects.

[0037] It should be noted that due to the limitations of the length of the specification, this specification does not exhaust all optional embodiments, and those skilled in the art should be able to think of any combination of technical features as long as the technical features do not contradict each other, which can constitute optional embodiments. The following will describe each embodiment in detail.

[0038] As an important basis of contemporary artificial intelligence technology, DNN models have been widely used in face recognition, video analysis, natural language processing and other fields. Through the training of complex multi-layer network structure and massive parameters, the structure of DNN model is becoming more and more complex. The network structure between the internal levels of the model can be represented by a directed acyclic graph.

[0039] DNN models can achieve high-precision calculation and significantly improve task performance. However, these performance improvements also bring huge computing and resource requirements. When running DNN tasks on devices with limited computing power, high latency and high energy consumption problems often occur, especially in real-time inference scenarios. The limited computing power of the device makes it difficult to achieve efficient inference. Therefore, in order to meet the requirements of real-time and efficient processing, traditional DNN inference tasks are usually deployed on high-performance servers to complete computationally intensive tasks with the help of their powerful computing power and stability.

[0040] With the rapid development of mobile devices and Internet of Things technology, more and more smart devices have become potential computing carriers. However, these devices are limited by hardware performance and energy consumption constraints, and often encounter the bottleneck of insufficient computing power when running complex DNN tasks. How to efficiently run DNN tasks on these resource-limited devices has become a research hotspot in the current academic and industrial circles.

[0041] The present application provides a distributed-based model inference method, device, equipment and storage medium, aiming at solving the above technical problems of the prior art.

[0042] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0043] Figure 1 The flowchart of a distributed-based model inference method provided by an embodiment of the present application is shown in the figure. The method can be executed by a distributed-based model inference device. The method is deployed in a distributed network system, which includes a center device and a plurality of edge devices. As shown in the figure, the method includes the following steps: Figure 1

[0044] S101, according to the structure information of the model to be allocated, the model to be allocated is divided to obtain a plurality of sub-models, and each sub-model is allocated to at least one edge device; wherein the structure information represents the hierarchical structure between the model layers, and the plurality of sub-models constitute the model to be allocated.

[0045] ​Exemplarily, the distributed network system can include a plurality of nodes, wherein a central node and a plurality of edge nodes can be included, each node can be a computer device, the central node is a central device, and the edge node is an edge device. The distributed network system can also include a user node, i.e., a user's client device, referred to as a user device. The central device can be responsible for managing, scheduling and distributing inference tasks of a DNN model, i.e., a to-be-distributed model in the embodiment, which can be used for artificial intelligence tasks such as image processing, text processing and speech processing. The user device can send a task request for model inference to the central device, the central device receives the task request sent by the user device, analyzes the request content, and determines the scheduling strategy of the task according to the real-time load of the system and the resource status of the edge device, distributes the inference task of the model to a suitable edge device, and completes the inference of the DNN model by cooperation of a plurality of edge devices. That is, the edge device is responsible for executing a specific DNN inference task.

[0046] Figure 2 A structural schematic diagram of the distributed network system is shown. Client represents a user device, Master represents a central device, and Worker represents an edge device.

[0047] The to-be-distributed model is an AI model of a deep neural network architecture, and includes a plurality of model layers, such as convolutional layers, fully connected layers, etc. In the embodiment, the model architecture of the to-be-distributed model is not specifically limited. The to-be-distributed model is sent to the central device, the central device receives the to-be-distributed model, and determines the structure information of the to-be-distributed model. The structure information can represent the hierarchical structure between the model layers in the to-be-distributed model, i.e., the dependency relationship between the model layers. For example, the structure information of the to-be-distributed model is first a 3x3 convolutional layer, then a 5x5 convolutional layer, and the output of the 3x3 convolutional layer is the input of the 5x5 convolutional layer, i.e., the 5x5 convolutional layer depends on the 3x3 convolutional layer. Different to-be-distributed models can correspond to different structure information.

[0048] The central device can split the to-be-distributed model into a plurality of sub-models according to the structure information of the to-be-distributed model after receiving each to-be-distributed model. These sub-models can constitute a complete to-be-distributed model. For example, the model layers can be evenly distributed according to the number of model layers in the to-be-distributed device, so that the number of model layers in each sub-model is consistent. For another example, the model layers with close dependency relationship can be divided into a sub-model to avoid the need for data transmission between these closely connected model layers across devices.

[0049] After obtaining multiple sub-models, for each sub-model, the sub-model can be assigned to one or more edge devices, that is, the same sub-model can be assigned to multiple edge devices. The assignment can be made in the order of the number of edge devices, for example, there are 8 edge devices in total, and there are 4 sub-models, then the first two edge devices are assigned the first sub-model, and the third and fourth edge devices are assigned the second sub-model. The sub-models can also be assigned to appropriate edge devices according to the computing power and resources of the edge devices, and according to the model size of the sub-models.

[0050] In S102, if the model inference request is responded to, resource information of each edge device is obtained; the resource information represents the running state of the edge device.

[0051] For example, the center device can receive multiple to-be-assigned models, each to-be-assigned model can be split into multiple sub-models, and the sub-models are loaded into different edge devices. Each to-be-assigned model corresponds to its own model identifier, and each sub-model can also correspond to the model identifier of the to-be-assigned model, indicating which to-be-assigned model the sub-model belongs to. The center device issues the sub-models to each corresponding edge device, and the edge device waits for the instruction of the center device to execute the sub-model and perform model inference.

[0052] The user device can send a model inference request to the center device, that is, a request to control the to-be-assigned model to perform inference. The model inference request can include a model identifier, indicating which model needs to be inferred. The model inference request can also include the input data of the model, so as to perform model inference according to the input data given by the user.

[0053] After responding to the model inference request, in addition to obtaining the model identifier and the input data, the resource information of each edge device can also be obtained in real time. The resource information can represent the running state of the edge device. For example, the resource information can include the computing power of the device, the memory usage, the network bandwidth, and the queue load, etc. That is, the state of the edge device is monitored in real time. The monitored information can include the computing power resources of the edge device, such as CPU and GPU usage, idle and busy state, hardware computing power of each edge device, number of tasks in the queue of the edge device, load pressure, etc.; it can also include memory usage, such as memory usage of each device, real-time memory occupancy data, etc., to avoid task failure due to insufficient resources; it can also include the transmission bandwidth of the network, such as measuring the actual bandwidth and delay, to master the data transmission speed between devices in real time and avoid transmission bottlenecks.

[0054] The resource information of all edge devices can be determined, or according to the model identifier in the model inference request, it can be determined which edge devices the sub-models of the to-be-assigned model are assigned to, and only the resource information of these edge devices is obtained, to improve the efficiency of model inference.

[0055] S103, determining a target device from the at least one edge device allocated by the sub-model according to the resource information of each edge device.

[0056] For each sub-model, the resource information of each edge device corresponding to the sub-model is determined. According to the resource information of each edge device corresponding to the sub-model, one edge device is determined from the edge devices corresponding to the sub-model as the target device of the sub-model. Each sub-model can correspond to one target device.

[0057] The edge devices corresponding to the sub-model can be ranked according to the resource information. For example, the edge devices can be ranked from high to low according to the CPU usage, and the edge device ranked first is determined as the target device of the sub-model. The edge devices can also be ranked in combination with multiple information in the resource information, and the best edge device is determined as the target device. The determination rule of the target device can be set in advance, and the determination rule of the target device is not limited in the embodiment.

[0058] S104, controlling the target devices to execute the corresponding sub-models according to the execution order between the sub-models.

[0059] For example, the execution order between the sub-models can be determined according to the dependency relationship between the model layers, for example, the output of the first sub-model is the input of the second sub-model, and the first sub-model is executed before the second sub-model. After the target devices corresponding to the sub-models are determined, the order of the target devices can be determined according to the execution order between the sub-models, so as to control the target devices to execute the corresponding sub-models in turn. For example, the first target device can send the output data to the second target device after executing the corresponding sub-model, and the second target device can execute the corresponding sub-model after receiving the output data.

[0060] In the embodiment, the target devices corresponding to the sub-models are ranked according to the execution order between the sub-models, and the current target device to be run is determined according to the ranking result. The current target device to be run is controlled to execute the corresponding sub-model, and the output result of the sub-model by the current target device to be run is received.

[0061] Specifically, the execution order between different sub-models is determined, the target devices corresponding to the sub-models are ranked according to the execution order between the sub-models, and the ranking result of the target devices is obtained. The current target device to be run is determined according to the ranking result of the target devices. That is, the current target device to be run can be determined in turn from front to back according to the ranking result.

[0062] The center device controls the target device currently to be executed to execute the corresponding sub-model. For the first target device to be executed, the center device can send the input data in the model inference request to the first target device. The first target device executes the sub-model according to the input data, obtains the output data of the first sub-model, and sends the output data of the first sub-model to the center device. The center device sends the output data of the first sub-model to the second target device and receives the output data of the second sub-model sent by the second target device. That is, the center device can receive the output result of each target device currently to be executed on the sub-model, and send the received output result to the next target device until the last target device. The output data sent back by the last target device is the inference result of the to-be-assigned model.

[0063] The beneficial effect of such a setting is that the center device can distribute the sub-tasks of the DNN model, that is, the sub-models, to the corresponding edge devices and perform inference execution. After the execution of the sub-models is completed, the processing results are summarized, and the final result of the inference task is obtained. The center device collects the calculation results of all sub-tasks, activates the next target device according to the task dependency relationship, and outputs the final inference result to the user device. The output results of each sub-model are integrated, data confusion is avoided, and the efficiency and accuracy of model inference are improved.

[0064] The embodiment of the application provides a distributed-based model inference method. By determining the structure information of the to-be-assigned model, the to-be-assigned model can be divided into multiple sub-models, and the splitting of the to-be-assigned model is realized. Each sub-model is assigned to at least one edge device, that is, the same sub-model can be loaded in multiple edge devices, and the normal operation of subsequent sub-models is ensured. After responding to the model inference request, the resource information of each edge device is obtained, and according to the resource information of each edge device, an edge device is determined from the multiple edge devices to which the sub-model is assigned as the target device of the sub-model, that is, the best device for executing the sub-model, and the effective use of computing resources is realized. According to the execution order between the sub-models, each target device is controlled to execute the corresponding sub-model in turn, and the inference of the to-be-assigned model is realized. The model is split and distributed, and the edge device is selected, so that the running efficiency of the model can be improved, the communication overhead is effectively reduced, and the resource utilization rate and performance are improved.

[0065] Figure 3 A flowchart of a distributed-based model inference method is provided for the embodiment of the application. The embodiment is an optional embodiment based on the above-mentioned embodiment.

[0066] In this embodiment, the to-be-assigned model is divided according to the structure information of the to-be-assigned model, and a plurality of sub-models are obtained, including: obtaining the structure information of the to-be-assigned model, and converting the structure information into a structure topology graph; wherein the structure topology graph includes a plurality of nodes, and the nodes represent model layers, and the structure topology graph is a directed acyclic graph representing the hierarchical relationship between the model layers; and the to-be-assigned model is divided according to the dependency relationship between the nodes in the structure topology graph, and a plurality of sub-models are obtained.

[0067] As shown in the method shown in the figure, the method comprises the following steps: Figure 3

[0068] S301, obtaining the structure information of the to-be-assigned model, and converting the structure information into a structure topology graph; wherein the structure topology graph includes a plurality of nodes, and the nodes represent model layers, and the structure topology graph is a directed acyclic graph representing the hierarchical relationship between the model layers.

[0069] Exemplarily, a to-be-assigned model is received, and the model structure of the to-be-assigned model is parsed to obtain the structure information of the to-be-assigned model. According to the structure information, the structure of the to-be-assigned model is converted into a directed acyclic graph (DAG) form, i.e. a structure topology graph is obtained. The structure topology graph includes a plurality of nodes, and one node represents one model layer, and the edges between the nodes represent the hierarchical relationship between the model layers, i.e. the dependency relationship.

[0070] Figure 4 The structure topology graph is a schematic diagram. Figure 4 The represented model is an Inception V4 model, DAG={L1, L2, …, L13}, L1 to L13 are the first to thirteenth layers of the model. That is, each model layer is a node, and the data dependency relationship between the layers is an edge. The more complex the network model corresponding to the inference task is, the more complex the structure topology graph is.

[0071] p() and s() can be used to represent the predecessor node and the successor node of the node in the structure topology graph, respectively, the predecessor node has an influence on the calculation of the node, and the successor node is affected by the calculation of the node. That is, the input of Li is the output of p(Li), and the output of Li is the input of s(Li). For example, for L3, p(L3) is L1, and s(L3) is L6 and L7.

[0072] S302, according to the dependency relationship between the nodes in the structure topology graph, the to-be-assigned model is divided, and a plurality of sub-models are obtained, and each sub-model is assigned to at least one edge device.

[0073] ​Exemplarily, the dependency relationship between nodes is determined from the structural topology graph, for example, there is an edge connection between two nodes, and there is a dependency relationship between the two nodes. According to the dependency relationship between the nodes in the structural topology graph, the model to be allocated is split, that is, the structural topology graph is split, and the structural topology graph is split into multiple subgraphs, and each subgraph represents a submodel. For example, the model layers with close dependency relationship can be divided into a submodel according to the structural topology graph.

[0074] The model splitting method based on subgraph segmentation can significantly improve the parallelism of the DNN inference task, optimize the resource utilization rate, and shorten the task execution time. Compared with the traditional hierarchical splitting method, the method can reduce the model data transmission amount by nearly 20% and the task execution time by 15%, and improve the overall system performance.

[0075] In this embodiment, the model to be allocated is divided and processed according to the dependency relationship between the nodes in the structural topology graph, and multiple submodels are obtained, including: obtaining parameter information of each node in the structural topology graph; wherein the parameter information represents the model parameters of the model layer corresponding to the node; according to the dependency relationship between the nodes in the structural topology graph and the parameter information of each node, the structural topology graph is divided and processed to obtain multiple subgraphs; for each subgraph, the model layers contained in the subgraph are combined into a submodel.

[0076] Specifically, for each node in the structural topology graph, there is corresponding parameter information of its own, and the parameter information of the node is the related information of the model parameters of the model layer. That is, for each node, the model parameters of the model layer corresponding to the node are obtained as the parameter information of the node. The parameter information can include the input tensor size, the output tensor size, the calculation cost, the memory consumption, etc. of the model layer.

[0077] According to the dependency relationship between the nodes in the structural topology graph and the parameter information of each node, the structural topology graph is divided and processed, and the structural topology graph is split into multiple subgraphs. For example, according to the parameter information, the similarity between nodes can be determined, and the nodes with high similarity and dependency relationship with edge connection are divided into a subgraph. In this embodiment, the calculation method of the similarity is not specifically limited. For example, the similarity can be calculated according to the output tensor size of the node and the input tensor size of another node.

[0078] Each subgraph can contain multiple nodes, that is, multiple model layers. For each subgraph, the model layers contained in the subgraph are combined into a submodel. The hierarchical relationship between the model layers in the submodel can be determined according to the connection relationship of the nodes in the subgraph.

[0079] The beneficial effect of such an arrangement is that subgraph partitioning is performed on the basis of the directed acyclic graph to obtain multiple subgraphs, i.e., multiple submodels, thereby improving the efficiency and accuracy of model splitting and further improving the efficiency and accuracy of subsequent model inference.

[0080] In this embodiment, the structure topology graph is divided and processed according to the dependency relationship between the nodes in the structure topology graph and the parameter information of each node to obtain multiple subgraphs, including: determining the weight information of the node according to the parameter information of the node in the structure topology graph; wherein the weight information represents the operation consumption of the model layer corresponding to the node; and dividing and processing the structure topology graph according to the dependency relationship between the nodes in the structure topology graph, the weight information of each node, and the preset number of subgraphs to obtain multiple subgraphs.

[0081] Specifically, for each node in the structure topology graph, the weight information of the node is determined according to the parameter information of the node. The weight information can represent the operation consumption of the model layer corresponding to the node, i.e., quantifying the operation consumption of each model layer. For example, the FLOPs (Floating Point Operations Per Second) of the model layer can be calculated according to the parameter information, and the FLOPs are used as the weight information.

[0082] The number of subgraphs is preset to determine how many subgraphs are to be divided, i.e., how many submodels are needed. For example, if the preset number of subgraphs is K, then K subgraphs, i.e., K submodels, can be divided. According to the weight information of each node, these nodes are divided into K subgraphs, so that each subgraph can include one or more nodes, and the nodes in different subgraphs can be different. The weight information of each node can be determined in sequence according to the dependency relationship between the nodes in the structure topology graph, and the node belongs to which subgraph is determined according to the weight information of the node. For example, the nodes can be allocated according to the weight information, so that the sum of the weights of the nodes in each subgraph is consistent. After the last node is determined, all the subgraphs are obtained.

[0083] The beneficial effect of such an arrangement is that the weight of each model layer is determined, and subgraph partitioning is performed through the dependency relationship between the model layers, so that the mutually dependent model layers can be allocated to the same subgraph, and important model layers can be avoided from being allocated to the same subgraph, thereby improving the partitioning accuracy of the subgraphs.

[0084] In this embodiment, the structural topology graph is divided according to the dependency relationship between nodes in the structural topology graph, the weight information of each node, and the preset number of subgraphs, to obtain a plurality of subgraphs, including: according to the preset number of subgraphs, a plurality of node spaces are constructed, and the current load of each node space is determined; wherein the node space is used to store nodes, and the current load represents the running load of the nodes in the node space; the nodes in the structural topology graph are iterated according to the dependency relationship between the nodes in the structural topology graph, to obtain the currently iterated node; the node space to which the currently iterated node belongs is determined according to the weight information of the currently iterated node and the current load of each node space, and the current load of the node space is updated; if it is determined that the structural topology graph is iterated, the nodes allocated in each node space are determined to obtain the subgraph.

[0085] Specifically, according to the preset number of subgraphs, a plurality of node spaces are constructed, and the number of node spaces is K. The initial node space is empty, and the node space can store nodes. Each node space can correspond to a subgraph, that is, the nodes in the node space can correspond to a subgraph. The current load of each node space can be determined in real time, and the current load represents the running load of the nodes in the node space. For the initial node space, since there is no node, the current load is 0.

[0086] According to the dependency relationship between the nodes in the structural topology graph, the nodes in the structural topology graph are iterated, that is, starting from the first model layer and iterating from front to back. Each iterated node is determined as the currently iterated node, and the weight information of the currently iterated node is determined.

[0087] For each node space, whether the currently iterated node belongs to the node space is determined according to the weight information of the currently iterated node and the current load of the node space. If yes, the node is stored in the node space, and the current load of the node space is updated; if no, whether the currently iterated node belongs to another node space is determined according to the weight information of the currently iterated node and the current load of another node space, until the node space to which the node belongs is found. For example, the maximum load of each node space can be preset. If the current load of the node space exceeds the preset maximum load after the node is added to the node space, it is determined that the node does not belong to the node space.

[0088] If it is determined that the structural topology graph is iterated, the nodes allocated in each node space are determined, and each node space corresponds to obtain a subgraph; if it is determined that the structural topology graph is not iterated, the node space to which the currently iterated node belongs is determined according to the weight information of the currently iterated node and the current load of each node space, and the current load of the node space is updated, until the iteration of the node is completed.

[0089] The beneficial effect of such arrangement is that the preset number of subgraphs is K, and the set of nodes in the initialized subgraph, i.e., the initialized node space, is empty. For each node, the node is assigned to a subgraph space, improving the rationality of subgraph division.

[0090] In this embodiment, the algorithm for assigning nodes is not specifically limited, for example, a greedy algorithm can be used to determine the node space to which the node belongs. Using the greedy strategy can improve the parallelism of subgraph division, reduce the dependency between subgraphs, enable the subgraphs to run simultaneously, and thus maximize the parallel efficiency. Each time the division is selected, the nodes with the least dependency are prioritized, the serial dependency between subgraphs is reduced, and the nodes with high dependency are assigned to the same subgraph as much as possible, reducing the communication and synchronization overhead across subgraphs.

[0091] In this embodiment, the node space to which the currently traversed node belongs is determined according to the weight information of the currently traversed node and the current load of each node space, including: determining the cost information corresponding to the node space according to the weight information of the currently traversed node and the current load of the node space; wherein the cost information represents the computation cost and communication cost required by the node space after the node is assigned to the node space; determining the node space to which the currently traversed node belongs according to the cost information corresponding to each node space.

[0092] Specifically, the weight information of the currently traversed node is determined, and for each node space, the cost information corresponding to the node space is determined according to the weight information of the currently traversed node and the current load of the node space. The cost information refers to the computation cost and communication cost required by the node space after the node is assigned to the node space. The new working load of the node space after the node is assigned to the node space can be determined according to the current load of the node space. The communication overhead and space depth of the node space after the node is assigned to the node space can also be determined, and the space depth refers to the hierarchical relationship of the nodes in the node space. According to the communication overhead, the space depth, and the new working load, the cost information can be determined. The calculation formula of the cost information can be:

[0093] F = α × ΔD + β × C + γ × W;

[0094] Wherein, F represents the cost information, ΔD represents the space depth, C represents the communication overhead, W represents the working load, and α, β, γ are preset parameters.

[0095] For the currently traversed node, a cost information can be calculated for each node space, and the node space to which the currently traversed node belongs is determined according to the cost information corresponding to each node space. For example, the node space with the minimum cost information can be determined as the node space to which the currently traversed node belongs.

[0096] The beneficial effect of such an arrangement is that the value of the generation of each node space is determined, the node space with the minimum value of the generation is selected, the communication and synchronization overhead across the subgraph is reduced, different edge scenarios and load environments can be adapted to, and the efficiency of subsequent model inference is improved.

[0097] In this embodiment, the parameter information includes at least one of a calculation cost, a memory consumption, an output data tensor size, and a transmission bandwidth, and the weight information includes a calculation weight and a communication weight, the calculation weight representing a length of a calculation time required by the model layer, and the communication weight representing a length of a transmission time required by the model layer; the weight information of the node is determined according to the parameter information of the node in the structure topology graph, including: the calculation weight of the node is determined according to the calculation cost and the memory consumption of the node, and the communication weight of the node is determined according to the output data tensor size and the transmission bandwidth of the node.

[0098] Specifically, the parameter information can include a calculation cost, a memory consumption, an input data tensor size, an output data tensor size, a transmission bandwidth, etc., and the weight information of the node can be calculated according to the parameter information. The weight information can include a calculation weight and a communication weight, the calculation weight representing a length of a calculation time required by the model layer, and the communication weight representing a length of a transmission time required by the model layer. That is, the calculation weight can be determined according to the parameter information, and the communication weight can be determined according to the parameter information.

[0099] The EFT (Earliest Finish Time) algorithm can be used to determine the weight information, and the calculation formula of the EFT algorithm can be:

[0100] EFT(Li)=EST(Li)+ET(Li)+CT(Li);

[0101] Wherein, EFT(Li) represents the earliest finish time of the calculation of the Li layer. EST(Li) represents the earliest start time of the calculation of the Li layer, which can be determined by the data transmission time. ET(Li) represents the calculation time required by the Li layer, which is determined by the calculation demand and the device computing power. CT(Li) represents the transmission time required by the Li layer, which is determined by the data output and the transmission bandwidth. In this embodiment, ET(Li) and CT(Li) are used to represent the calculation weight and the communication weight of the node Li respectively, ET(Li) is determined by the calculation demand and the device computing power, and CT(Li) is determined by the data output and the transmission bandwidth, that is, the calculation weight can be determined according to the calculation cost and the memory consumption of the node, and the communication weight can be determined according to the output data tensor size and the transmission bandwidth of the node.

[0102] The beneficial effect of such an arrangement is that by determining the calculation weight and the communication weight, the node can be accurately divided, and the division accuracy of the subgraph is improved.

[0103] Figure 5 A schematic diagram of the subgraph division result. Figure 5 is divided into five subgraphs, i.e., five submodels, T1 to T5, on the structure of Figure 4 The five subgraphs, i.e., five submodels, T1 to T5, are obtained by dividing the subgraph on the structure of

[0104] S303, if the model inference request is responded to, resource information of each edge device is acquired; the resource information represents the running state of the edge device.

[0105] Exemplarily, this step can refer to the above step S102, and will not be repeated here.

[0106] S304, according to the resource information of each edge device, a target device is determined from at least one edge device to which a submodel is allocated.

[0107] Exemplarily, this step can refer to the above step S103, and will not be repeated here.

[0108] S305, according to the execution order between each submodel, the target device is controlled to execute the corresponding submodel.

[0109] Exemplarily, this step can refer to the above step S104, and will not be repeated here.

[0110] The embodiment of the application provides a distributed-based model inference method. By determining the structure information of a to-be-allocated model, the to-be-allocated model can be divided into multiple submodels, so that the to-be-allocated model is split. Each submodel is allocated to at least one edge device, i.e., the same submodel can be loaded in multiple edge devices, so that the normal operation of the subsequent submodel is ensured. After responding to the model inference request, the resource information of each edge device is acquired, and according to the resource information of each edge device, one edge device is determined from the multiple edge devices to which the submodel is allocated as the target device of the submodel, i.e., the best device to execute the submodel, so that the effective use of the computing resource is realized. According to the execution order between each submodel, each target device is controlled to execute the corresponding submodel in turn, so that the inference of the to-be-allocated model is realized. By splitting and allocating the model and selecting the edge device, the running efficiency of the model can be improved, the communication overhead is effectively reduced, and the dual improvement of the resource utilization rate and the performance is realized.

[0111] Figure 6 A flowchart of a distributed-based model inference method provided by the embodiment of the application is shown in the figure, which is an optional embodiment based on the above embodiment.

[0112] In this embodiment, the target device is determined from the at least one edge device allocated by the sub-model according to the resource information of each edge device, including: for each sub-model, determining the reward value of the edge device allocated by the sub-model based on the preset deep reinforcement learning algorithm according to the resource information of the edge device allocated by the sub-model; wherein the reward value represents the communication overhead and delay of the edge device executing the sub-model; and determining the target device of the sub-model according to the reward value of each edge device allocated by the sub-model.

[0113] As shown in Figure 6 the method comprises the following steps:

[0114] S601, according to the structure information of the to-be-allocated model, performing division processing on the to-be-allocated model to obtain a plurality of sub-models, and allocating each sub-model to at least one edge device; wherein the structure information represents the hierarchical structure between model layers, and the plurality of sub-models constitute the to-be-allocated model.

[0115] Exemplarily, this step can refer to the above-mentioned step S101, and will not be repeated here.

[0116] S602, if a model inference request is responded to, obtaining the resource information of each edge device; wherein the resource information represents the running state of the edge device.

[0117] Exemplarily, this step can refer to the above-mentioned step S102, and will not be repeated here.

[0118] S603, for each sub-model, determining the reward value of the edge device allocated by the sub-model based on the preset deep reinforcement learning algorithm according to the resource information of the edge device allocated by the sub-model; wherein the reward value represents the communication overhead and delay of the edge device executing the sub-model.

[0119] Exemplarily, the deep reinforcement learning algorithm is preset, for example, the deep reinforcement learning algorithm can be a DQN (Deep Q-Network) algorithm. When model inference is needed, each sub-model corresponds to a plurality of edge devices, therefore, a target device needs to be determined from the plurality of edge devices to execute the sub-model. The target device can be selected by the preset deep reinforcement learning algorithm, in this embodiment, the task offloading strategy of the deep reinforcement learning algorithm is used to optimize the offloading decision of the model in a dynamic environment, and the optimal edge device is selected for each sub-model.

[0120] For each sub-model, resource information of all edge devices corresponding to the sub-model can be obtained, for example, device status, network condition and the like of the edge devices can be obtained, and dependency characteristics of the sub-model can also be obtained, the dependency characteristics can represent the dependency relationship between model layers in the sub-model. The device status, the network condition, the dependency characteristics of the sub-model and the like are determined as S (State, state) parameters in the deep reinforcement learning algorithm. For each edge device corresponding to the sub-model, there is also an A (Action, action) parameter, A represents whether to offload, offload to which edge device and the like. For example, if the sub-model is offloaded to a certain edge device, A is 1; if the sub-model is not offloaded to a certain edge device, A is 0. The deep reinforcement learning algorithm also includes a P (Policy, policy) parameter, P represents the rule of transition from state to action, that is, the task offloading policy.

[0121] Suppose that the sub-model corresponds to N edge devices, a set of S parameters corresponding to the sub-model is determined as a state space. The state space can be represented as: S t =[Ns1, …, Ns N , Task_info], wherein S t represents the state space, Ns i represents resource information of the edge device i, i is 1 to N. Task_info represents related information of the sub-model, including size of the sub-model, calculation complexity of the sub-model, execution positions and states of pre-task and post-task of the sub-model, quality of service requirements of the sub-model and the like. Ns i =[f i , Mu i , Bw i , Q i , Av], wherein f i represents CPU utilization, Mu i represents memory usage, Bw i represents current available network bandwidth, Q i represents the number of sub-tasks accumulated on the current worker node queue, and Av represents whether the edge device i supports execution of the sub-model.

[0122] A reward function in the deep reinforcement learning algorithm is designed in advance, the reward function needs to comprehensively consider performance indicators such as calculation delay, energy consumption, network transmission time and the like, and the goal is to maximize user experience while minimizing system resource consumption. The reward function can be designed in the following form:

[0123]

[0124] wherein R t is a reward value, representing performance indicators such as communication overhead and delay of the edge device executing the sub-model. ε and is a preset weight coefficient used to adjust the balance between communication overhead and delay. Com t is the communication overhead consumed by the execution of the sub-model, Delay t characterizes the total delay required for the completion of the sub-model, including the computation time and the network transmission time. In the reward function, the greater the delay or communication overhead, the greater the negative reward will be.

[0125] The preset deep reinforcement learning algorithm is adopted, and the resource information of each edge device is input, for example, the state space can be input. The preset reward function is calculated, and the reward value of each edge device is output. In this embodiment, the calculation process of the deep reinforcement learning algorithm is not specifically limited.

[0126] In this embodiment, the deep reinforcement learning algorithm can also be pre-trained. The training process can be initializing the DQN network parameters, the experience replay pool and the environment state; at each time, it is monitored whether there is a ready training task, the state information of each edge device is obtained, and the state space is generated; according to the current state space and the policy network, it is determined whether the offloading action needs to be executed, so as to execute the offloading of the sub-model; the system feedback after executing the action is obtained, including the new state space and the immediate reward value; the feedback data is stored in the experience pool, and samples are randomly extracted from the experience pool, and the network parameters are updated; the above steps are repeatedly executed through iterative updating, and the policy is constantly updated until the model converges. Each module of the whole system cooperates through a closed loop to realize real-time computation offloading optimization. Through this closed loop feedback and real-time optimization mechanism, different computing environments can be adapted, the offloading policy can be dynamically adjusted, and the optimal use of computing resources and the improvement of user experience can be ensured. In this embodiment, the training process of the deep reinforcement learning algorithm is not specifically limited.

[0127] S604, according to the reward value of each edge device allocated by the sub-model, determining the target device of the sub-model.

[0128] Exemplarily, for each sub-model, the reward value of each edge device corresponding to the sub-model can be determined. According to the reward value of each edge device, the target device is determined from these edge devices. For example, the edge device with the largest positive reward can be determined as the target device. For another example, a filtering condition of the reward value is set in advance, the edge device with the reward value meeting the condition is determined as the target device.

[0129] The embodiment of the present application provides a neural network offloading algorithm based on deep learning, that is, a deep reinforcement learning algorithm is introduced, and a model splitting structure is combined to dynamically optimize offloading of a submodel. Compared with a traditional offloading method based on rules or static allocation, the following key factors can be adaptively considered: a load state and a model loading condition of a work node, a network bandwidth and a communication delay, a task dependency relationship and a priority, and the like. Through deep reinforcement learning, the system can select optimal nodes participating in execution of the submodel in real time, reduce overall inference delay, and improve throughput. The intelligent scheduling method significantly improves resource utilization efficiency of a distributed cluster.

[0130] In the embodiment, the resource information includes current remaining memory; the method further includes: if a target device of the submodel is not determined based on the preset deep reinforcement learning algorithm, determining the target device from the edge devices to which the submodel is allocated according to current remaining memory of the edge devices to which the submodel is allocated.

[0131] Specifically, the target device of the submodel is first determined according to the deep reinforcement learning algorithm, and if each reward value corresponding to the submodel does not satisfy a preset condition, for example, each reward value is negative, it is indicated that the target device cannot be determined according to the deep reinforcement learning algorithm.

[0132] A bottomed scheduling strategy is preset, and if the target device of the submodel cannot be determined according to the deep reinforcement learning algorithm, the target device can be determined by using the bottomed scheduling strategy. The resource information can include current remaining memory of the edge devices, and the bottomed scheduling strategy can be to determine the target device from the edge devices to which the submodel is allocated according to current remaining memory of the edge devices to which the submodel is allocated. For example, the edge device with the largest current remaining memory can be determined as the target device.

[0133] The beneficial effect of such a setting is that the best target device is selected for the submodel by setting the bottomed scheduling strategy, the reliable execution of the model is ensured, the efficiency of model inference is improved, and the robustness of the distributed network system is improved.

[0134] In the embodiment, the neural network model is split into multiple submodels, efficient parallel inference can be realized in a distributed cluster environment, computing performance is improved, and communication overhead of inter-level information transmission is reduced. The split submodel is taken as a scheduling unit to participate in the offloading process, can reasonably allocate tasks to appropriate computing nodes according to different devices and network environments, and thus the load balancing and overall efficiency of the system are improved.

[0135] S605, according to the execution order between the submodels, control the target device to execute the corresponding submodel.

[0136] Exemplarily, the present step can refer to the above step S104, and will not be described herein again.

[0137] The embodiment of the present application provides a distributed model reasoning method, by determining structure information of a to-be-distributed model, the to-be-distributed model can be divided into a plurality of sub-models, and the splitting of the to-be-distributed model is realized. Each sub-model is distributed to at least one edge device, that is, the same sub-model can be loaded in a plurality of edge devices, and the normal operation of the subsequent sub-model is ensured. After responding to the model reasoning request, the resource information of each edge device is acquired, and according to the resource information of each edge device, an edge device is determined from the plurality of edge devices to which the sub-model is distributed, as the target device of the sub-model, that is, the best device for executing the sub-model, and the effective use of the computing resource is realized. According to the execution order between the sub-models, the target devices are controlled in sequence to execute the corresponding sub-models, and the reasoning of the to-be-distributed model is realized. The model is split, distributed, and the edge device is selected, so that the running efficiency of the model can be improved, the communication overhead is effectively reduced, and the dual improvement of resource utilization and performance is realized.

[0138] Figure 7 A structural block diagram of a distributed model reasoning device is provided for the embodiment of the present application. For the convenience of description, only the part related to the embodiment of the present application is shown. The device is applied to a center device of a distributed network system, and the distributed network system includes one center device and a plurality of edge devices. Referring to Figure 7 The distributed model reasoning device 700 includes a model division unit 701, a resource acquisition unit 702, a device determination unit 703 and a model execution unit 704.

[0139] The model division unit 701 is configured to divide the to-be-distributed model according to the structure information of the to-be-distributed model, obtain a plurality of sub-models, and distribute each sub-model to at least one edge device; wherein the structure information represents the hierarchical structure between model layers, and the plurality of sub-models constitute the to-be-distributed model.

[0140] The resource acquisition unit 702 is configured to acquire the resource information of each edge device if responding to the model reasoning request; wherein the resource information represents the running state of the edge device.

[0141] The device determination unit 703 is configured to determine a target device from at least one edge device to which the sub-model is distributed according to the resource information of each edge device.

[0142] The model execution unit 704 is configured to control the target device to execute the corresponding sub-model according to the execution order between the sub-models.

[0143] Figure 8 A structural block diagram of a distributed model reasoning device is provided for the embodiment of the present application, as shown inFigure 8 As shown, the distributed-based model inference apparatus 800 comprises a model partition unit 801, a resource acquisition unit 802, a device determination unit 803 and a model execution unit 804, wherein the model partition unit 801 comprises a graph conversion module 8011 and a model partition module 8012.

[0144] The graph conversion module 8011 is configured to acquire structure information of a to-be-distributed model, and convert the structure information into a structure topology graph; wherein the structure topology graph comprises a plurality of nodes, the nodes represent model layers, and the structure topology graph is a directed acyclic graph representing hierarchical relationships between the model layers.

[0145] The model partition module 8012 is configured to perform partition processing on the to-be-distributed model according to the dependency relationships between the nodes in the structure topology graph, to obtain a plurality of sub-models.

[0146] In one example, the model partition module 8012 comprises:

[0147] A parameter acquisition sub-module is configured to acquire parameter information of each node in the structure topology graph; wherein the parameter information represents model parameters of the model layer corresponding to the node.

[0148] A subgraph partition sub-module is configured to perform partition processing on the structure topology graph according to the dependency relationships between the nodes in the structure topology graph and the parameter information of each node, to obtain a plurality of subgraphs.

[0149] A sub-model determination sub-module is configured to, for each subgraph, combine the model layers contained in the subgraph into the sub-model.

[0150] In one example, the subgraph partition sub-module is specifically configured to:

[0151] Determine weight information of the node according to the parameter information of the node in the structure topology graph; wherein the weight information represents operation consumption of the model layer corresponding to the node.

[0152] Perform partition processing on the structure topology graph according to the dependency relationships between the nodes in the structure topology graph, the weight information of each node, and a preset number of subgraphs, to obtain a plurality of subgraphs.

[0153] In one example, the subgraph partition sub-module is specifically configured to:

[0154] According to the preset number of subgraphs, construct a plurality of node spaces, and determine current loads of the node spaces; wherein the node spaces are used to store nodes, and the current loads represent running loads of the nodes in the node spaces.

[0155] According to the dependency relationship between the nodes in the structure topology graph, the nodes in the structure topology graph are processed in a traversal manner to obtain a currently traversed node;

[0156] According to the weight information of the currently traversed node and the current load of each node space, a node space to which the currently traversed node belongs is determined, and the current load of the node space is updated.

[0157] If it is determined that the structure topology graph is traversed, the subgraph is determined according to the nodes allocated in each node space.

[0158] In one example, the subgraph division sub-module is specifically configured to:

[0159] According to the weight information of the currently traversed node and the current load of each node space, a node space to which the currently traversed node belongs is determined, and the current load of the node space is updated.

[0160] According to the weight information of the currently traversed node and the current load of each node space, a node space to which the currently traversed node belongs is determined, and the current load of the node space is updated.

[0161] In one example, the parameter information includes at least one of a calculation cost, a memory consumption, an output data tensor size, and a transmission bandwidth, the weight information includes a calculation weight and a communication weight, the calculation weight represents a length of a calculation time required by a model layer, and the communication weight represents a length of a transmission time required by the model layer; the subgraph division sub-module is specifically configured to:

[0162] According to the calculation cost and the memory consumption of the node, the calculation weight of the node is determined, and according to the output data tensor size and the transmission bandwidth of the node, the communication weight of the node is determined.

[0163] In one example, the device determination unit 803 includes:

[0164] The reward value determination module is configured to, for each sub-model, determine a reward value of an edge device to which the sub-model is allocated based on a preset deep reinforcement learning algorithm according to resource information of the edge device to which the sub-model is allocated; wherein the reward value represents a communication overhead and a time delay of the edge device executing the sub-model.

[0165] The device determination module is configured to determine a target device of the sub-model according to the reward values of the edge devices to which the sub-model is allocated.

[0166] In one example, the resource information includes a current remaining memory; and further includes:

[0167] The bottom unit is configured to determine the target device from the edge device assigned by the sub-model according to the current remaining memory of the edge device assigned by the sub-model if the target device of the sub-model is not determined based on the preset deep reinforcement learning algorithm.

[0168] Figure 9 A structural block diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1. The electronic device includes a memory 91 and a processor 92. Figure 9 The memory 91 is configured to store instructions executable by the processor 92.

[0169] The processor 92 is configured to perform the method provided by the above-described embodiment.

[0170] The electronic device further includes a receiver 93 and a transmitter 94. The receiver 93 is configured to receive instructions and data transmitted by other devices, and the transmitter 94 is configured to transmit instructions and data to external devices.

[0171] Figure 10 FIG. 1 is a block diagram of an electronic device according to an example embodiment. The electronic device can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a personal digital assistant, or the like.

[0172] The device 1000 can include one or more of the following components: a processing component 1002, a memory 1004, a power supply component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0173] The processing component 1002 usually controls overall operations of the device 1000, such as operations associated with display, phone call, data communication, camera operation, and recording operation. The processing component 1002 can include one or more processors 1020 to execute instructions to complete all or part of steps of the methods described above. In addition, the processing component 1002 can include one or more modules to facilitate the interaction between the processing component 1002 and other components. For example, the processing component 1002 can include a multimedia module to facilitate the interaction between the multimedia component 1008 and the processing component 1002.

[0174] The device 1000 can include one or more of the following components: a processing component 1002, a memory 1004, a power supply component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0175] The memory 1004 is configured to store various types of data to support operations of the device 1000. Examples of these data include instructions for any application or method operating on the device 1000, contact data, phonebook data, messages, pictures, videos, and the like. The memory 1004 can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0176] The power supply component 1006 supplies power for various components of the device 1000. The power supply component 1006 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1000.

[0177] The multimedia component 1008 includes a screen providing an output interface between the device 1000 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. The front and / or rear camera can receive external multimedia data when the device 1000 is in an operating mode, such as a shooting mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0178] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC) configured to receive external audio signals when the device 1000 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 1004 or transmitted via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio signals.

[0179] The I / O interface 1012 provides an interface between the processing component 1002 and peripheral interface modules, which can be a keyboard, a click wheel, a button, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0180] The sensor component 1014 includes one or more sensors for providing status assessments for various aspects of the device 1000. For example, the sensor component 1014 can detect an open / closed position of the device 1000, relative positioning of components of the device 1000, such as a display and keypad of the device 1000, a change in position of the device 1000 or a component of the device 1000, presence or absence of user contact with the device 1000, orientation or acceleration / deceleration of the device 1000, and temperature changes of the device 1000. The sensor component 1014 can include proximity sensor(s) configured to detect presence of nearby objects without any physical contact. The sensor component 1014 can further include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 1014 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0181] The communication component 1016 is configured to facilitate wired or wireless communication between the device 1000 and another device. The device 1000 can access a wireless network based on a corresponding communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an example embodiment, the communication component 1016 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 1016 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technology.

[0182] In an example embodiment, the device 1000 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing the above-described methods.

[0183] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 1004 including instructions, is also provided, which can be executed by the processor 1020 of the device 1000 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0184] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-described distributed-based model inference method.

[0185] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0186] The above embodiments are only preferred examples of the present application, and the protection scope of the present application is not limited thereto. Any equivalent replacement or transformation made by those skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. A distributed-based model inference method, characterized in that, The method is applied to a central device of a distributed network system, the distributed network system comprising a central device and a plurality of edge devices; the method comprises: performing division processing on the to-be-assigned model according to structure information of the to-be-assigned model to obtain a plurality of sub-models, and assigning each sub-model to at least one edge device; wherein the structure information represents a hierarchical structure between model layers, and the plurality of sub-models constitute the to-be-assigned model; if a model inference request is responded to, obtaining resource information of each edge device; wherein the resource information represents the running state of the edge device; determining a target device from at least one edge device to which the sub-model is assigned according to the resource information of each edge device; controlling the target device to execute the corresponding sub-model according to the execution order between the sub-models.

2. The method of claim 1, wherein, performing division processing on the to-be-assigned model according to structure information of the to-be-assigned model to obtain a plurality of sub-models, comprising: obtaining the structure information of the to-be-assigned model, and converting the structure information into a structure topology graph; wherein the structure topology graph comprises a plurality of nodes, the nodes represent model layers, and the structure topology graph is a directed acyclic graph representing the hierarchical relationship between the model layers; performing division processing on the to-be-assigned model according to the dependency relationship between the nodes in the structure topology graph to obtain a plurality of sub-models.

3. The method of claim 2, wherein, performing division processing on the to-be-assigned model according to the dependency relationship between the nodes in the structure topology graph to obtain a plurality of sub-models, comprising: obtaining parameter information of each node in the structure topology graph; wherein the parameter information represents the model parameters of the model layer corresponding to the node; performing division processing on the structure topology graph according to the dependency relationship between the nodes in the structure topology graph and the parameter information of each node to obtain a plurality of sub-graphs; for each sub-graph, combining the model layers contained in the sub-graph into the sub-model.

4. The method of claim 3, wherein, performing division processing on the structure topology graph according to the dependency relationship between the nodes in the structure topology graph and the parameter information of each node to obtain a plurality of sub-graphs, comprising: determining weight information of the nodes according to the parameter information of the nodes in the structure topology graph; wherein the weight information represents the operation consumption of the model layer corresponding to the node; performing division processing on the structure topology graph according to the dependency relationship between the nodes in the structure topology graph, the weight information of each node, and a preset number of sub-graphs to obtain a plurality of sub-graphs.

5. The method of claim 4, wherein, performing division processing on the structure topology graph according to the dependency relationship between the nodes in the structure topology graph, the weight information of each node, and a preset number of sub-graphs to obtain a plurality of sub-graphs, comprising: constructing a plurality of node spaces according to the preset number of sub-graphs, and determining the current load of each node space; wherein the node spaces are used to store nodes, and the current load represents the running load of the nodes in the node space; performing traversal processing on the nodes in the structure topology graph according to the dependency relationship between the nodes in the structure topology graph to obtain a currently traversed node; According to the weight information of the current traversed node and the current load of each node space, a node space to which the current traversed node belongs is determined, and the current load of the node space is updated; If it is determined that the structure topology graph is traversed, the sub-graph is determined according to the nodes allocated in each node space.

6. The method of claim 5, wherein, According to the weight information of the current traversed node and the current load of each node space, a node space to which the current traversed node belongs is determined, and the current load of the node space is updated; According to the weight information of the current traversed node and the current load of each node space, a node space to which the current traversed node belongs is determined, and the current load of the node space is updated; According to the weight information of the current traversed node and the current load of each node space, a node space to which the current traversed node belongs is determined, and the current load of the node space is updated; 7. The method of claim 4, wherein, The parameter information includes at least one of a calculation cost, a memory consumption, an output data tensor size, and a transmission bandwidth, the weight information includes a calculation weight and a communication weight, the calculation weight represents a length of a calculation time required by a model layer, and the communication weight represents a length of a transmission time required by the model layer; According to the parameter information of the node in the structure topology graph, the weight information of the node is determined, including: According to the calculation cost and the memory consumption of the node, the calculation weight of the node is determined, and according to the output data tensor size and the transmission bandwidth of the node, the communication weight of the node is determined.

8. The method of claim 1, wherein, According to the resource information of each edge device, a target device is determined from at least one edge device allocated by the sub-model, including: For each sub-model, according to the resource information of the edge device allocated by the sub-model, a reward value of the edge device allocated by the sub-model is determined based on a preset deep reinforcement learning algorithm; wherein the reward value represents a communication overhead and a time delay of the edge device executing the sub-model; According to the reward value of each edge device allocated by the sub-model, a target device of the sub-model is determined.

9. The method of claim 8, wherein, The resource information includes a current remaining memory; the method further includes: If no target device of the sub-model is determined based on the preset deep reinforcement learning algorithm, a target device is determined from the edge device allocated by the sub-model according to the current remaining memory of the edge device allocated by the sub-model.

10. The method according to any one of claims 1-9, characterized in that, According to the execution order between each sub-model, the target device is controlled to execute the corresponding sub-model, including: According to the execution order between each sub-model, the target devices corresponding to each sub-model are sorted, and a current target device to be run is determined according to a sorting result; The current target device to be run is controlled to execute the corresponding sub-model, and an output result of the sub-model by the current target device to be run is received.

11. A distributed-based model inference apparatus, comprising: The device is applied to a center device of a distributed network system, the distributed network system includes one center device and multiple edge devices; the device includes: The model division unit is configured to perform division processing on the to-be-allocated model according to structure information of the to-be-allocated model, to obtain a plurality of sub-models, and to allocate each of the sub-models to at least one edge device; wherein the structure information represents a hierarchical structure between model layers, and the plurality of sub-models constitute the to-be-allocated model. The resource acquisition unit is configured to acquire resource information of each edge device in response to a model inference request; wherein the resource information represents a running state of the edge device. The device determination unit is configured to determine a target device from the at least one edge device to which the sub-models are allocated, according to the resource information of each edge device. The model execution unit is configured to control the target device to execute a corresponding sub-model according to an execution order between the sub-models.

12. An electronic device, comprising: The processor and a memory connected in communication with the processor are included. The memory stores computer-executable instructions. The processor executes the computer-executable instructions stored in the memory to implement the method of any one of claims 1-10. The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are configured to implement the method of any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, A computer program is included, which, when executed by a processor, implements the method of any one of claims 1-10.

14. A computer program product, characterised in that, ​

Images