A collaborative reasoning method for convolutional neural network models in edge-cloud environments

Abstract

The present invention discloses a collaborative reasoning method for a convolutional neural network model in an edge-cloud environment, comprising: converting a convolutional neural network model into a directed acyclic graph, constructing an objective function and constraint conditions of an edge-cloud collaborative reasoning system, executing a given convolutional neural network model in the edge-cloud collaborative reasoning system, obtaining the execution delay of each layer of the convolutional neural network model on a heterogeneous device and output data of each layer, merging nodes of the directed acyclic graph to obtain a scale-optimized directed acyclic graph; pre-quantizing the output data of each layer of the convolutional neural network model, and constructing a pre-quantized delay graph based on the scale-optimized directed acyclic graph according to the execution delay of each layer on the heterogeneous device and the transmission delay of the pre-quantized output data of each layer, calculating the minimum cut from the source point to the sink point in the delay graph, performing actual model compression, and obtaining a collaborative reasoning decision of a given convolutional neural network model on a heterogeneous device.

Description

Technical Field

[0001] The present invention relates to a collaborative reasoning method for a convolutional neural network model in an edge-cloud environment, and belongs to the field of edge computing technology. Background Art

[0002] The Convolutional Neural Networks (CNN) model is a deep learning model commonly used in the field of computer vision. It has made significant progress in image recognition, target detection, image generation and many other fields, and has become an important part of computer vision and deep learning research.

[0003] The execution of traditional CNN models is usually divided into two stages: training and inference. The training stage is the process in which the CNN model learns to extract features from input data and make accurate predictions or classifications. By learning to identify patterns and features through a large amount of labeled data, the model can accurately extract important features from the input data and make correct predictions or classifications based on these features. In the inference stage, due to the huge amount of computation required by the CNN model, the traditional architecture is a cloud-centric paradigm, so that edge devices with limited computing resources can use such services. A large amount of data is generated in the edge device and transmitted to a powerful cloud server through the network. The model is inferred on the cloud server, and then the inference result is transmitted back to the edge device. However, this method is highly dependent on the network environment and has significant latency. For example, in the field of autonomous driving, the sensors used in current advanced autonomous driving technology can generate hundreds to thousands of gigabits of data per second, but the average uplink rate of the widely used 5g network is only tens of Mbps. Under the cloud-centric architecture, there is inevitably a problem of network bandwidth load, and the long model inference latency has a significant impact on applications such as autonomous driving and smart medical equipment that require real-time response.

[0004] In recent years, collaborative reasoning methods between edge and cloud devices have been widely studied. For example, through model segmentation technology, the CNN model can be divided into two parts according to the computing power and network bandwidth of edge devices and cloud devices, and then deployed on heterogeneous two-layer devices. For common CNN models, there are significant differences in the amount of computing required for each layer and the size of the output data. The computational requirements of the middle layer of the model are relatively high, and there are significant differences in the computational requirements between adjacent layers, while the output data size of the later layer is much smaller than the output data size of the previous layer. The existing technology uses model segmentation methods without considering the actual application of the algorithm to edge devices with limited computing resources, and the huge amount of computing required to make segmentation decisions. As a result, when the model has a large number of layers and a complex structure, the decision time for obtaining segmentation decisions is too long, which limits applications that require real-time response services.

[0005] Another widely studied collaborative reasoning method is model compression, which aims to reduce the complexity of the model through model compression technology and deploy the compressed complete model on resource-constrained end or edge devices. Currently, the widely used model compression technologies include pruning, quantization, early exit, knowledge distillation, low-rank decomposition, and sparse representation. However, model compression in existing technologies focuses on compressing the entire model, which will lead to a non-negligible loss of reasoning accuracy on resource-limited edge devices, especially for compact CNN models. In addition, the compression reasoning acceleration of the entire model may require specially designed hardware support, which is difficult for general edge devices to meet. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a collaborative reasoning method for a convolutional neural network model in an edge-cloud environment to improve collaborative reasoning decisions and optimize transmission delays.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] In a first aspect, the present invention provides a collaborative reasoning method for a convolutional neural network model in an edge-cloud environment. Based on an edge-cloud collaborative reasoning system composed of an edge device and a cloud server, the collaborative reasoning method for a convolutional neural network model includes:

[0009] Obtain the model structure and data flow direction of a given trained convolutional neural network model, convert the convolutional neural network model into a directed acyclic graph, and build the objective function and constraints of the edge-cloud collaborative reasoning system based on the latency model of collaborative reasoning between edge devices and cloud servers;

[0010] Based on the objective function and constraints of the edge-cloud collaborative reasoning system and the relevant data required for segmentation decision-making based on the minimum cut theory, a given convolutional neural network model is executed in the edge-cloud collaborative reasoning system to obtain the execution latency of each layer of the convolutional neural network model on heterogeneous devices and the output data of each layer;

[0011] Based on the execution delay of each layer of the convolutional neural network model on heterogeneous devices and the output data of each layer, the directed acyclic graph is merged with nodes to obtain a scale-optimized directed acyclic graph; the output data of each layer of the convolutional neural network model is pre-quantized to reduce the transmission delay between layers. Based on the scale-optimized directed acyclic graph, a pre-quantized delay graph is constructed according to the execution delay of each layer on heterogeneous devices and the transmission delay of the pre-quantized output data of each layer;

[0012] Based on the highest-numbered pre-push algorithm, the minimum cut from the source point to the sink point in the delay graph is calculated, and the actual model compression is performed on the cut points in the minimum cut set to obtain the collaborative reasoning decision of the given convolutional neural network model on heterogeneous devices.

[0013] In combination with the first aspect, further, the directed acyclic graph is expressed as:

[0014] ，

[0015] In the formula, G Represents a directed acyclic graph constructed according to the model structure and data flow direction of a given convolutional neural network model. , represents the layer of the convolutional neural network model, Represents the layers of the model; Represents the set of relationships between layers with data dependencies, i.e., the edges in a directed acyclic graph, An element in , presentation layer With layer There is data dependency between them, and data is divided into layers Flow layer ;

[0016] The delay model is expressed as:

[0017] ,

[0018] In the formula, represents the total latency of the collaborative reasoning of the convolutional neural network model on the edge device and the cloud server, T e represents the sum of the inference latency of the model layers deployed on the edge device in the convolutional neural network model, T c Indicates the transmission delay of the total amount of data that needs to be transmitted in the data layer. T t Represents the sum of the inference latency of the model layers in the convolutional neural network model deployed on the cloud server.

[0019] Furthermore, the sum of the inference latency of the model layer deployed on the edge device in the convolutional neural network model is expressed as:

[0020] ,

[0021] The transmission delay of the total amount of data that needs to be transmitted in the data layer is expressed as:

[0022] ,

[0023] The sum of the inference latency of the model layers in the convolutional neural network model deployed on the cloud server is expressed as:

[0024] ,

[0025] In the formula, Represents the convolutional neural network model The inference time of the layer on the edge device, v i Represents the model layer in the convolutional neural network model, Indicates the model layer in the convolutional neural network model that needs to be deployed on the edge device; Represents the convolutional neural network model The inference time of the layer on the cloud server, Indicates the model layer in the convolutional neural network model that needs to be deployed on the cloud server; express The set of layers that need to transmit data to the cloud server, Indicates the output data size that needs to be transmitted by the data layer. Indicates the data transmission rate between edge devices and cloud servers.

[0026] Furthermore, the objective function is:

[0027] ,

[0028] In the formula, Indicates the model layer in the convolutional neural network model that needs to be deployed on the edge device. Indicates the model layer in the convolutional neural network model that needs to be deployed on the cloud server. argmin T total Indicates that it can make Smallest , Represents the total latency of the collaborative inference of the convolutional neural network model on the edge device and the cloud server;

[0029] The constraints are:

[0030] ,

[0031] ,

[0032] In the formula, V Represents all layers of the convolutional neural network model. For a given convolutional neural network model's collaborative inference decision, the union of the model layers that need to be inferred on the edge device and the model layers that need to be inferred on the cloud device; This means that the layers that need to be deployed on the edge device and the layers that need to be deployed on the cloud server cannot have duplicate layers.

[0033] Furthermore, the execution latency of each layer of the convolutional neural network model on heterogeneous devices and the output data of each layer are obtained, including:

[0034] Step S1: register hooks on each layer of the convolutional neural network model, and obtain the time required for the hook to execute each layer and the output data size of each layer when the hook is activated;

[0035] Step S2: On the edge device and cloud server, the convolutional neural network model executes step S1 multiple times to obtain the average execution time of each layer of the convolutional neural network model on the heterogeneous device as the execution latency, as well as the output data of each layer.

[0036] Furthermore, a pre-quantized delay graph is constructed, including:

[0037] Based on the execution latency of each layer of the convolutional neural network model on heterogeneous devices and the output data of each layer, the directed acyclic graph is merged with layers with the same input data size to obtain a scale-optimized directed acyclic graph.

[0038] Based on the dynamic quantization technology, the output data of each layer of the convolutional neural network model is pre-quantized, and the size of the output data of each layer of the convolutional neural network model after pre-quantization to INT8 bits is counted;

[0039] Based on the execution latency of each layer of the convolutional neural network model on heterogeneous devices, the output data of each layer, the directed acyclic graph after scale optimization, and the size of the output data of each layer after pre-quantization to INT8 bits, a pre-quantized latency graph is constructed.

[0040] Furthermore, based on the execution latency of each layer of the convolutional neural network model on heterogeneous devices, the output data of each layer, the directed acyclic graph after scale optimization, and the size of the output data of each layer after pre-quantization to INT8 bits, a pre-quantized latency graph is constructed, including:

[0041] For the scale-optimized directed acyclic graph, add a special node " ”, “ "and" ", respectively represent the entry and exit points of the delay graph and the original data constraints;

[0042] According to the directed acyclic graph with special nodes added, traverse all nodes except " "and" ", set or update the node from " " to each node and from each node to " "The capacity attribute of the edge of " is used to obtain a directed acyclic graph with the execution delay of each layer set;

[0043] For all edges in the directed acyclic graph with execution delays of each layer set and without the capacity attribute set, the transmission delay is calculated according to the pre-quantized output data of the upstream node and the provided bandwidth; the transmission delay is allocated as the capacity of the edge, and a directed acyclic graph with execution delays of each layer set and pre-quantized transmission delays is obtained;

[0044] On the basis of the directed acyclic graph with the execution delay of each layer and the transmission delay after prequantization, for special nodes with multiple successors, remove the edge connecting the current special node to the successor node, add an intermediate node, connect the current node with the intermediate node, and set the capacity to the transmission delay of the current node under the network bandwidth and prequantize it to the INT8 level; connect the intermediate node and the successor node of the current node, set the capacity value of the edge to infinity, and obtain the directed acyclic graph after processing the special node;

[0045] Based on the directed acyclic graph after processing special nodes, connect " "arrive" o "," o "arrive" "and" o ” to the edge of the first layer of the calculation to obtain the final delay graph.

[0046] Furthermore, the special node " ”, “ "and" "The latency and output size of nodes are set to 0; the capacity attribute of edges is set to the cloud or edge latency value stored in the layer information of each node," e "arrive" o "The infinite capacity of " indicates that there is no constraint on entry," o "arrive" c " has a capacity of zero, indicating no delay, and " o ” to the first layer with a capacity of R, simulating the target constraint.

[0047] Furthermore, the collaborative reasoning decision of a given convolutional neural network model on heterogeneous devices is obtained, including:

[0048] Based on the highest number pre-push algorithm, the maximum flow from the source to the sink in the delay graph is calculated to obtain the residual network containing the maximum flow information;

[0049] Based on the residual network, traverse all reachable nodes from the source point and identify the node set on the source side ;

[0050] Based on the node set on the source side, obtain the node set on the sink side ;

[0051] According to the residual network arrive All edges of , accumulate the delay values, and obtain the minimum total delay that can be achieved by the current task allocation;

[0052] for The model layer that needs to transmit data to the cloud server for dynamic quantization completes the entire process of a given convolutional neural network model in the edge-cloud collaborative reasoning system.

[0053] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements any of the above-mentioned collaborative reasoning methods for a convolutional neural network model in an edge-cloud environment.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention merges nodes of a directed acyclic graph based on the execution delay of each layer of a convolutional neural network model on heterogeneous devices and the output data of each layer to obtain a directed acyclic graph with optimized scale; without changing the model structure, the scale of the original model graph converted from the CNN model is optimized, and the decision time required for obtaining a decision using the minimum cut for a pre-quantized delay graph constructed based on the graph is significantly optimized;

[0056] The present invention pre-quantizes the output data of each layer of the convolutional neural network model to reduce the transmission delay between layers. On the basis of the directed acyclic graph after scale optimization, a delay graph is constructed according to the execution delay of each layer on heterogeneous devices and the transmission delay of the output data of each layer after pre-quantization. By introducing model quantization technology, the present invention can make full use of heterogeneous system resources, making it less likely for the decision algorithm to tend to a single-end execution strategy, and reducing the transmission cost of collaborative reasoning after CNN model segmentation while maintaining model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of a collaborative reasoning method for a convolutional neural network model in an edge-cloud environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0059] Embodiment 1:

[0060] The embodiment of the present invention provides a collaborative reasoning method for a convolutional neural network model in an edge-cloud environment, and specifically relates to an efficient collaborative reasoning method for a convolutional neural network model based on model segmentation and model compression. Under the operating framework of the entire edge-cloud collaborative reasoning system, the collaborative reasoning scheme of joint model partitioning and compression is mainly divided into the following four stages: the model offline analysis stage, the stage of modeling the CNN model as a DAG graph and performing scale optimization, the construction of a pre-quantized delay graph based on the layer's reasoning delay and transmission delay, and the use of the maximum flow minimum cut algorithm to divide the graph and actually quantize the transmission data layer stage.

[0061] When making decisions in a specific edge-cloud environment and a pre-trained CNN model scenario, you first need to perform an offline analysis of the edge-cloud collaborative reasoning system to obtain a system configuration file, which contains the execution delay data of the model layer on the edge device and the cloud server obtained through layer delay measurement, as well as the size of the output data of each layer of the model. And obtain the model structure and data flow direction of the given trained convolutional neural network model, and convert the convolutional neural network model into a directed acyclic graph.

[0062] Then, according to the output data size of each layer in the system configuration file, the directed acyclic graph is merged with the layers with the same input data size to obtain the scale-optimized directed acyclic graph. Then, based on the scale-optimized directed acyclic graph, a pre-quantized delay graph is constructed. Finally, the optimal segmentation strategy is obtained based on the maximum flow minimum cut algorithm, and the layers that need to transmit data to the cloud server are dynamically quantized.

[0063] Based on the edge-cloud collaborative reasoning system composed of edge devices and cloud servers, the edge devices and cloud servers have heterogeneous computing resources. For the collaborative accelerated reasoning of a given convolutional neural network model in this environment, the edge-cloud collaborative system performs the following steps to obtain the distributed reasoning decision of the given model on devices with heterogeneous resources, and realize efficient reasoning for the given convolutional neural network model in the edge-cloud environment. The method flow is as follows: Figure 1 As shown, the specific steps include:

[0064] Step A: For the edge-cloud collaborative system, based on a given trained convolutional neural network model, analyze the model structure and data flow direction. According to the characteristic that data between model layers only flows in one direction, convert the convolutional neural network model into a directed acyclic graph. Based on the latency model of collaborative reasoning between edge devices and cloud servers, construct the objective function and constraints of the edge-cloud collaborative reasoning system.

[0065] In step A, when performing collaborative reasoning decision for a given convolutional neural network model, from step A1 to step A2, a directed acyclic graph of the given convolutional neural network model and a delay model when collaborative reasoning is performed using edge devices and cloud servers are obtained.

[0066] Step A1: For a given convolutional neural network , analyze the model structure and data flow direction, and use the nodes in the directed acyclic graph to represent the data flow direction according to the characteristic that data between model layers can only flow in one direction. The layers in are represented by edges in a directed acyclic graph. The dependencies between the middle layers, i.e. the direction of data flow, are represented by the constructed directed acyclic graph as follows:

[0067] ,

[0068] in, G Represents a directed acyclic graph constructed according to the model structure and data flow direction of a given convolutional neural network model. , represents the collection of layers of a convolutional neural network model, Represents the layers of the model, Represents the set of relationships between layers with data dependencies, i.e., the edges in a directed acyclic graph. An element in , presentation layer With layer There is data dependency between them, and data is divided into layers Flow layer .

[0069] Step A2: When executing the collaborative reasoning decision requirement delay model for a given convolutional neural network model, the calculation process is as follows:

[0070] For a given convolutional neural network model After being split by the collaborative reasoning decision model, the model layer that requires reasoning on the edge device , whose inference delay is:

[0071] ,

[0072] in, Represents the model The inference time of each layer on the edge device.

[0073] Use express The collection of layers that need to transmit data to the cloud server.

[0074] The transmission delay of the total data transmission of the data layer required It can be expressed as:

[0075] ,

[0076] In the formula, Indicates the output data size that needs to be transmitted by the data layer. Indicates the data transmission rate between edge devices and cloud servers.

[0077] For a given convolutional neural network model After being split by the collaborative reasoning decision model, the model layer that requires reasoning on the cloud server , whose inference delay is:

[0078] ,

[0079] in, Represents the model Inference time of each layer on the cloud server.

[0080] In summary, for a given convolutional neural network Total inference latency when performing collaborative inference on edge cloud devices It is expressed as:

[0081] ,

[0082] Based on the directed acyclic graph of a given convolutional neural network model and the latency model when using collaborative reasoning between edge devices and cloud servers, the objective function and constraints of the edge-cloud collaborative system are constructed;

[0083] The objective function is:

[0084] ,

[0085] argmin T total Indicates that it can make Smallest ;

[0086] The constraints are as follows:

[0087] ,

[0088] Indicates that for a given convolutional neural network model's collaborative inference decision, the union of the model layers that need to be inferred on the edge device and the model layers that need to be inferred on the cloud device should be all the layers of the model;

[0089] ,

[0090] Indicates that for the collaborative inference decision of a given convolutional neural network model, the model layers that need to be inferred on the edge device and the model layers that need to be inferred on the cloud device cannot have duplicate layers.

[0091] Step B: Based on the objective function and constraints of the edge-cloud collaborative system, and for the relevant data required for segmentation decisions based on the minimum cut theory, by actually executing the given convolutional neural network model on the two devices, construct a model containing the execution delay of each layer on heterogeneous devices and the output data of each layer.

[0092] In step B, the relevant data required for segmentation decision making based on the minimum cut theory is constructed by executing a given convolutional neural network model on two devices, including the execution delay of each layer of the model on heterogeneous devices and the output data of each layer. The specific process is as follows:

[0093] In the offline state, hooks are registered on each layer of a given model. The hooks are used to measure and record the time required to execute the layer and the size of the transmitted data when each layer is activated. At the same time, since the data of a single layer execution is very short and difficult to record, the execution time of the layer is obtained by calculating the cumulative execution time of multiple layers and then dividing it by the number of times the layer is executed, and a report on the average execution delay is generated for each layer.

[0094] Execute a given model on different devices to obtain the profile of the model layer's latency on different devices And the size of the output data of each layer of the model .

[0095] Step C: Based on the system configuration file, the directed acyclic graph converted by the convolutional neural network model is scaled and optimized by utilizing its features. The scale of the graph is reduced by merging target nodes to construct a scale-optimized directed acyclic graph. Then, dynamic quantization technology is used to pre-quantize the output data of each layer of the model to reduce the transmission delay between layers. Based on the scale-optimized directed acyclic graph, a pre-quantized delay graph is constructed according to the execution delay of each layer on heterogeneous devices and the transmission delay of the pre-quantized output data of each layer.

[0096] In step C, when executing collaborative reasoning decision for a given convolutional neural network model, from step C1 to step C3, a pre-quantized delay graph is obtained:

[0097] Step C1: Based on the fact that there are a large number of layers with the same input data size in the convolutional neural network model, and the total inference delay of segmentation between these adjacent layers is greater than the total inference delay of segmentation before the initial layers of these adjacent layers, these layers with the same input data size are merged;

[0098] If , , If the input data of three adjacent layers have the same size, they are regarded as inseparable nodes. In the constructed directed acyclic graph, these special nodes are merged into new nodes. , reduce the number of nodes and links in the graph and optimize the graph size, such as Figure 1 In and ,as well as and Merge into new nodes and ;

[0099] Step C2: For the layer that needs to transmit data, such as Figure 1 In and , by using the quantization technology in model compression to pre-quantize its output data, that is, for the intermediate results that need to be transmitted between CNN model inferences, the range of the floating-point number of the value is , pre-quantized to the range , the size of the output data after each layer of the statistical model is pre-quantized to INT8 bits ;

[0100] Step C3: Based on the system configuration file, the directed acyclic graph after scale optimization, and the pre-quantized output data size of each layer, the process of constructing the delay graph is as follows:

[0101] Step C3.1: Add a special node " ”, “ "and" ", respectively represent the entry and exit points of the delay graph and the original data constraints. The delay and output size of these nodes are set to 0 because they are not actual computation layers.

[0102] Step C3.2: Exclude "and" ", traverse all nodes, set or update the representative from the cloud" "to each node, and from each node to the edge" The capacity attribute of these edges is set to the cloud or edge latency value stored in the layer_info of each node.

[0103] Step C3.3: For all edges that do not have a capacity attribute set, i.e., do not point to or come from " "or" "The edge is based on the size of the pre-quantized output data of the upstream node. The transmission delay is calculated based on the provided bandwidth. Then, this delay is allocated as the capacity of the edge.

[0104] Step C3.4: Special treatment for nodes with multiple successors: remove the edge connecting the current node to the successor node, add an intermediate node that is not the actual computing layer, connect the current node with the intermediate node, set the capacity to the transmission delay of the current node under the network bandwidth and pre-quantize it to the INT8 level, connect the intermediate node and the successor node of the current node, and set the capacity value of the edge to infinity.

[0105] Step C3.5: Connect e "arrive" o "," o "arrive" c "and" o ” to the first layer of the computation. The capacities of these edges are set to represent the target constraint or latency: e "arrive" o "The infinite capacity indicates that there is no constraint on entry from " o "arrive" c "'s zero capacity means no delay, and o ” to the first layer with a capacity of R to simulate the target constraint.

[0106] Step D: Based on the pre-quantized delay graph, the highest-numbered pre-push algorithm is used to calculate the minimum cut from the source point to the sink point of the graph, and the actual model compression is performed on the cut points in the minimum cut set to obtain the collaborative reasoning decision of the given convolutional neural network model on heterogeneous resource devices under the current network bandwidth.

[0107] In step D, the specific process is as follows:

[0108] For the pre-quantized delay graph, initialize a pre-flow from the source point to the directly connected node, and manage the excess flow and height of each node. Operate on the residual graph of the network, initialize the pre-flow by pushing the flow from the source point to the adjacent node as much as possible, continuously adjust the height of the node, and push the excess flow of the node to the node with a lower height, so as to ensure the effective transmission of the flow. The global restandard periodically updates the height of all nodes, optimizes the flow path, and speeds up the solution. When the excess flow of all nodes is properly processed, the excess flow of the sink represents the maximum flow value from the source point to the sink. Through this process, the maximum flow is efficiently calculated to help determine the maximum carrying capacity of the network. The residual network contains the required minimum cut value of the delay graph, that is, the minimum , and distributed reasoning decisions in the current scenario and Then, according to the segmentation strategy r The model layer needs to transmit data to the cloud server for dynamic quantization.

[0109] The core idea of ​​the present invention is to convert the convolutional neural network model segmentation problem into a maximum flow minimum cut problem, construct a directed acyclic graph according to the model structure, and then scale-optimize the constructed directed acyclic graph. On the basis of the optimized graph, a pre-quantized delay graph is established, and the impact of quantization on the total inference delay of different decisions is considered. The model segmentation decision is obtained for the constructed pre-quantized delay graph using the minimum cut theory that is more suitable for the current scenario. After the decision is obtained, the actual quantization is performed on the layer that needs to transmit data.

[0110] Embodiment 2:

[0111] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned collaborative reasoning methods for a convolutional neural network model in an edge-cloud environment.

[0112] The technical solution of the present invention significantly reduces the decision complexity of complex CNN models in collaborative reasoning in edge-cloud environments, and can quickly obtain decisions for models with different numbers of layers and structures. In addition, the introduction of quantization also makes it less likely for decision algorithms to tend to single-end execution strategies in general scenarios, which improves the overall resource utilization of the system.

[0113] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A collaborative reasoning method for a convolutional neural network model in an edge-cloud environment, characterized in that: Based on the edge-cloud collaborative reasoning system composed of edge devices and cloud servers, the collaborative reasoning method of the convolutional neural network model includes: Obtain the model structure and data flow direction of a given trained convolutional neural network model, convert the convolutional neural network model into a directed acyclic graph, and build the objective function and constraints of the edge-cloud collaborative reasoning system based on the latency model of collaborative reasoning between edge devices and cloud servers; Based on the objective function and constraints of the edge-cloud collaborative reasoning system and the minimum cut theory, a given convolutional neural network model is executed in the edge-cloud collaborative reasoning system to obtain the execution latency of each layer of the convolutional neural network model on heterogeneous devices and the output data of each layer; Based on the execution delay of each layer of the convolutional neural network model on heterogeneous devices and the output data of each layer, the directed acyclic graph is merged with nodes to obtain a directed acyclic graph after scale optimization; the output data of each layer of the convolutional neural network model is pre-quantized to reduce the transmission delay between layers. Based on the directed acyclic graph after scale optimization, a pre-quantized delay graph is constructed according to the execution delay of each layer on heterogeneous devices and the transmission delay of the pre-quantized output data of each layer; Based on the highest-numbered pre-push algorithm, the minimum cut from the source point to the sink point in the delay graph is calculated, and the actual model compression is performed on the cut points in the minimum cut set to obtain the collaborative reasoning decision of the given convolutional neural network model on heterogeneous devices.

2. According to the collaborative reasoning method of a convolutional neural network model in an edge-cloud environment according to claim 1, it is characterized in that: The directed acyclic graph is represented as: ， In the formula, G Represents a directed acyclic graph constructed according to the model structure and data flow direction of a given convolutional neural network model. , represents the layer of the convolutional neural network model, Represents the layers of the model; Represents the set of relationships between layers with data dependencies, i.e., the edges in a directed acyclic graph, An element in , presentation layer With layer There is data dependency between them, and data is divided into layers Flow layer ; The delay model is expressed as: ， In the formula, represents the total latency of the collaborative reasoning of the convolutional neural network model on the edge device and the cloud server, T e represents the sum of the inference latency of the model layers deployed on the edge device in the convolutional neural network model, T c Indicates the transmission delay of the total amount of data that needs to be transmitted in the data layer. T t Represents the sum of the inference latency of the model layers in the convolutional neural network model deployed on the cloud server.

3. According to the collaborative reasoning method of the convolutional neural network model in the edge cloud environment of claim 2, it is characterized in that: The sum of the inference latency of the model layers deployed on the edge device in the convolutional neural network model is expressed as: ， The transmission delay of the total amount of data that needs to be transmitted in the data layer is expressed as: ， The sum of the inference latency of the model layers in the convolutional neural network model deployed on the cloud server is expressed as: ， In the formula, Represents the convolutional neural network model The inference time of the layer on the edge device, v i Represents the model layer in the convolutional neural network model, Indicates the model layer in the convolutional neural network model that needs to be deployed on the edge device; Represents the convolutional neural network model The inference time of the layer on the cloud server, Indicates the model layer in the convolutional neural network model that needs to be deployed on the cloud server; express The set of layers that need to transmit data to the cloud server, Indicates the output data size that needs to be transmitted by the data layer. Indicates the data transmission rate between edge devices and cloud servers.

4. According to the collaborative reasoning method of a convolutional neural network model in an edge-cloud environment according to claim 1, it is characterized in that: The objective function is: ， In the formula, Indicates the model layer in the convolutional neural network model that needs to be deployed on the edge device. Indicates the model layer in the convolutional neural network model that needs to be deployed on the cloud server. argmin T total Indicates that it can make Smallest , Represents the total latency of the collaborative reasoning of the convolutional neural network model on the edge device and the cloud server; The constraints are: ， ， In the formula, V Represents all layers of the convolutional neural network model. For a given convolutional neural network model's collaborative inference decision, the union of the model layers that need to be inferred on the edge device and the model layers that need to be inferred on the cloud device; This means that the layers that need to be deployed on the edge device and the layers that need to be deployed on the cloud server cannot have duplicate layers.

5. According to the collaborative reasoning method of a convolutional neural network model in an edge-cloud environment according to claim 1, it is characterized in that: Obtain the execution latency of each layer of the convolutional neural network model on heterogeneous devices and the output data of each layer, including: Step S1: register hooks on each layer of the convolutional neural network model, and obtain the time required for the hook to execute each layer and the output data size of each layer when the hook is activated; Step S2: On the edge device and cloud server, the convolutional neural network model executes step S1 multiple times to obtain the average execution time of each layer of the convolutional neural network model on the heterogeneous device as the execution latency, as well as the output data of each layer.

6. The collaborative reasoning method of a convolutional neural network model in an edge-cloud environment according to claim 1, characterized in that: Construct a pre-quantized delay graph, including: Based on the execution latency of each layer of the convolutional neural network model on heterogeneous devices and the output data of each layer, the directed acyclic graph is merged with layers with the same input data size to obtain a scale-optimized directed acyclic graph. Based on the dynamic quantization technology, the output data of each layer of the convolutional neural network model is pre-quantized, and the size of the output data of each layer of the convolutional neural network model after pre-quantization to INT8 bits is counted; Based on the execution latency of each layer of the convolutional neural network model on heterogeneous devices, the output data of each layer, the directed acyclic graph after scale optimization, and the size of the output data of each layer after pre-quantization to INT8 bits, a pre-quantized latency graph is constructed.

7. The collaborative reasoning method of a convolutional neural network model in an edge-cloud environment according to claim 6 is characterized in that: Based on the execution latency of each layer of the convolutional neural network model on heterogeneous devices, the output data of each layer, the directed acyclic graph after scale optimization, and the size of the output data of each layer after pre-quantization to INT8 bits, a pre-quantized latency graph is constructed, including: Add special nodes to the scale-optimized directed acyclic graph" ”, " "and" ", respectively represent the entry and exit points of the delay graph and the original data constraints; According to the directed acyclic graph with special nodes added, traverse all nodes except " "and" ", set or update the node from " " to each node and from each node to" "The capacity attribute of the edge of " is used to obtain a directed acyclic graph with the execution delay of each layer set; For all edges in the directed acyclic graph with execution delays of each layer set and without the capacity attribute set, the transmission delay is calculated according to the pre-quantized output data of the upstream node and the provided bandwidth; the transmission delay is allocated as the capacity of the edge, and a directed acyclic graph with execution delays of each layer set and pre-quantized transmission delays is obtained; On the basis of the directed acyclic graph with the execution delay of each layer and the transmission delay after prequantization, for special nodes with multiple successors, remove the edge connecting the current special node to the successor node, add an intermediate node, connect the current node with the intermediate node, and set the capacity to the transmission delay of the current node under the network bandwidth and prequantize it to the INT8 level; connect the intermediate node and the successor node of the current node, set the capacity value of the edge to infinity, and obtain the directed acyclic graph after processing the special node; Based on the directed acyclic graph after processing special nodes, connect" "arrive" o ”、" o "arrive" "and" o ” to the edge of the first layer of the calculation to obtain the final delay graph.

8. The collaborative reasoning method of a convolutional neural network model in an edge-cloud environment according to claim 7, characterized in that: The special node" ”, " "and" "The latency and output size of nodes are set to 0; the capacity attribute of edges is set to the cloud or edge latency value stored in the layer information of each node," e "arrive" o "The infinite capacity of " means that there is no constraint on entry, " o "arrive" c "A capacity of zero means no delay, and " o ” to the first layer with a capacity of R, simulating the target constraint.

9. The collaborative reasoning method of a convolutional neural network model in an edge-cloud environment according to claim 1, characterized in that: Obtain collaborative reasoning decisions for a given convolutional neural network model on heterogeneous devices, including: Based on the highest-number pre-push algorithm, the maximum flow from the source to the sink in the delay graph is calculated to obtain the residual network containing the maximum flow information; Based on the residual network, traverse all reachable nodes from the source point and identify the node set on the source side ; Based on the node set on the source side, obtain the node set on the sink side ; According to the residual network arrive All edges of , accumulate the delay values, and obtain the minimum total delay that can be achieved by the current task allocation; for The model layer that needs to transmit data to the cloud server for dynamic quantization completes the entire process of a given convolutional neural network model in the edge-cloud collaborative reasoning system.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements a collaborative reasoning method for a convolutional neural network model in an edge-cloud environment as described in any one of claims 1-9.

Images