Deep learning model collaborative inference method, device and equipment based on knowledge distillation

Abstract

The application provides a kind of deep learning model collaborative inference method, device and equipment based on knowledge distillation, comprising: obtaining the FLOPS corresponding to each edge node, storage capacity, data successful transmission probability;Based on FLOPS, storage capacity and data successful transmission probability, each edge node is clustered and processed, and K target node clusters are obtained for redundant backup;Multiple convolution filters of the last convolution layer of the preset teacher model are set partitioning processing, and K filter sets are obtained;Based on K target node clusters, K filter sets and multiple preset models, determine the to-be-trained model adapted to the capacity of each target node cluster;Based on multiple preset sample data, the to-be-trained model of each target node cluster is jointly trained using knowledge distillation technology, and the student model of each target node cluster is obtained;For each target node cluster, the student model of the target node cluster is deployed to each edge node in the target node cluster, to improve inference performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge intelligence, and in particular to a high-robustness deep learning model collaborative inference method and device based on knowledge distillation. BACKGROUND

[0002] With the computing processing function of Internet of Things devices shifting to the network edge, the intelligent service demand of edge devices at the network edge is increasing, and deep neural networks (DNN) play an important role in the intelligent service demand.

[0003] At present, in the distributed DNN inference technology, one type of method based on dividing the input within the layers of the DNN model divides the model computing load and distributes it to multiple edge devices for collaborative inference; another type of method directly divides the original DNN model to obtain multiple sub-models, each of which is deployed on a single edge device for distributed DNN collaborative inference. Although the above methods can effectively divide the computing load of the DNN model, due to the characteristics of convolution operation, frequent communication and synchronization of intermediate results are required between edge devices, and the communication overhead generated thereby limits the performance improvement brought by distributed inference. In addition, the method of dividing the input requires each device to accommodate the complete DNN model, which is a strong assumption for resource-constrained edge devices. In order to reduce the communication overhead, existing work has proposed a distributed inference method in which the models are independent of each other, and multiple lightweight models with the same structure and independent functions are derived from the original DNN model through knowledge distillation technology to collaboratively implement distributed inference.

[0004] However, due to the heterogeneity of the storage and computing capabilities of each edge device, uniform division of the computing load by the method can lead to a mismatch between the model and the device capability, which is not conducive to improving resource utilization and inference performance. On the other hand, due to the instability of the edge device state and wireless communication link, the inference process faces the risk of loss of results due to device failure or communication failure, thereby reducing the accuracy. The method lacks robustness design, and thus lacks flexibility in the face of failure. SUMMARY

[0005] The present application provides a high-robustness deep learning model collaborative inference method, device and equipment based on knowledge distillation, to solve the problem of low collaborative inference performance due to the synchronization communication between models, the mismatch between the model and the device capability, or node failure in the prior art, and to achieve the purpose of improving the collaborative inference performance.

[0006] In a first aspect, the present application provides a deep learning model collaborative inference method based on knowledge distillation, comprising:

[0007] obtain floating point operations per second (FLOPS), storage capacity and data successful transmission probability corresponding to each edge node;

[0008] perform clustering processing on the edge nodes based on the FLOPS, the storage capacity and the data successful transmission probability, to obtain K target node clusters;

[0009] perform set partitioning processing on a plurality of convolution filters of a last convolution layer in a preset teacher model, to obtain K filter sets;

[0010] determine a to-be-trained model of each target node cluster based on the K target node clusters, the K filter sets and a plurality of preset models;

[0011] perform joint training on the to-be-trained models of the target node clusters based on a plurality of preset sample data and using a knowledge distillation technology, to obtain student models of the target node clusters;

[0012] deploy the student models of the target node clusters to each edge node in the target node cluster respectively, and each edge node in the target node cluster is used to perform cooperative inference when running the corresponding student model.

[0013] In a second aspect, the present application further provides a deep learning model cooperative inference device based on knowledge distillation, comprising:

[0014] an obtaining module configured to obtain floating point operations per second (FLOPS), storage capacity and data successful transmission probability corresponding to each edge node;

[0015] a clustering module configured to perform clustering processing on the edge nodes based on the FLOPS, the storage capacity and the data successful transmission probability, to obtain K target node clusters;

[0016] a partitioning module configured to perform set partitioning processing on a plurality of convolution filters of a last convolution layer in a preset teacher model, to obtain K filter sets;

[0017] a determining module configured to determine a to-be-trained model of each target node cluster based on the K target node clusters, the K filter sets and a plurality of preset models;

[0018] a training module configured to perform joint training on the to-be-trained models of the target node clusters based on a plurality of preset sample data and using a knowledge distillation technology, to obtain student models of the target node clusters;

[0019] a deployment module configured to deploy, for each target node cluster, a student model of the target node cluster to each edge node in the target node cluster, and each edge node in the target node cluster is configured to perform collaborative inference when running the corresponding student model.

[0020] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the method for collaborative inference of the deep learning model based on knowledge distillation according to any one of the above aspects when running the program.

[0021] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the method for collaborative inference of the deep learning model based on knowledge distillation according to any one of the above aspects.

[0022] In a fifth aspect, the present application also provides a computer program product comprising a computer program, and the computer program, when executed by a processor, implements the method for collaborative inference of the deep learning model based on knowledge distillation according to any one of the above aspects.

[0023] The present application provides a method, device and equipment for collaborative inference of a deep learning model based on knowledge distillation. In the method, each edge node is clustered based on FLOPS, storage capacity and data successful transmission probability to obtain K target node clusters, the purpose of which is to redundantly backup student models in the node clusters, so that other edge nodes can continue to infer in the case of failure of a certain edge node in the node cluster, and the cumulative successful transmission probability of each node cluster is ensured to meet the requirements, thereby improving the resilience of collaborative inference to failure. In addition, a plurality of convolution filters of the last convolution layer in the preset teacher model are subjected to set partitioning processing to obtain K filter sets, which ensures that important filters are uniformly distributed in the K filter sets, so that each filter set has equal importance to the inference result. Further, when determining the model to be trained, the FLOPS, storage capacity and data transmission rate of each edge node are considered comprehensively, and the preset model and filter set that are adapted to the capability of each node cluster are allocated to each node cluster, so that the inference delay of each node cluster is close and the resources are fully utilized, thereby improving the inference performance. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0025] Figure 1 is an application scenario schematic diagram provided by the present application;

[0026] Figure 2 is a flow schematic diagram of the deep learning model collaborative deduction method based on knowledge distillation provided by the present application;

[0027] Figure 3 is a method flow schematic diagram for obtaining K target node clusters provided by the present application;

[0028] Figure 4 is a method flow schematic diagram for obtaining K filter sets provided by the present application;

[0029] Figure 5 is a method flow schematic diagram for determining the to-be-trained model of each target node cluster provided by the present application;

[0030] Figure 6 is a method flow schematic diagram for optimizing the to-be-trained model of the target node cluster provided by the present application;

[0031] Figure 7 is one of the collaborative deduction performance simulation result comparison schematic diagrams of the deep learning model collaborative deduction method based on knowledge distillation provided by the present application;

[0032] Figure 8 is the second of the collaborative deduction performance simulation result comparison schematic diagrams of the deep learning model collaborative deduction method based on knowledge distillation provided by the present application;

[0033] Figure 9 is the third of the collaborative deduction performance simulation result comparison schematic diagrams of the deep learning model collaborative deduction method based on knowledge distillation provided by the present application;

[0034] Figure 10 is a structural schematic diagram of the deep learning model collaborative deduction device based on knowledge distillation provided by the present application;

[0035] Figure 11 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0036] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0037] In the present application, the term "comprising" and its variants can refer to non-limiting inclusion; the term "or" and its variants can refer to "and / or". In the present application, the terms "first", "second", etc. are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. In the present application, "at least one" refers to two or more. "And / or", which describes the association relationship of the associated objects, means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the front and rear associated objects.

[0038] In the following, the application scenario of the technical solutions shown in the present application is described. Figure 1 The application scenario of the technical solutions shown in the present application is described.

[0039] Figure 1 The present application provides an application scenario diagram. As shown in Figure 1 , for example, the application scenario includes: a plurality of node clusters, a plurality of student models, a geographic area, and a deduction result receiving end.

[0040] For example, the plurality of node clusters includes node cluster 1, node cluster 2, node cluster 3, and node cluster 4. Each node cluster includes at least one edge node. Each edge node in the node cluster can deploy the student model corresponding to the node cluster. Optionally, the edge node can be a drone, an intelligent camera, or a vibration sensor, etc.

[0041] For example, the plurality of student models includes student model 1, student model 2, student model 3, and student model 4.

[0042] Among them, the node cluster and the student model one-to-one correspond, for example: node cluster 1 corresponds to student model 1.

[0043] Optionally, the deduction result receiving end can be a user terminal, which can be a mobile phone, a tablet computer, or a notebook computer, for example.

[0044] In the case that the edge node in node cluster 1 obtains the to-be-processed data from the geographic area, the edge node sends the to-be-processed data to all edge nodes in other node clusters (such as node cluster 2, node cluster 3, and node cluster 4).

[0045] For each edge node of the other node cluster, the edge node runs the corresponding student model, performs deduction on the to-be-processed data through the student model, obtains a partition result, and returns the partition result to the edge node in node cluster 1.

[0046] In the case that the edge nodes in the node cluster 1 receive multiple partition results, the multiple partition results are spliced by the edge nodes in the node cluster 1 to obtain a spliced result, and the spliced result is calculated to obtain a deduction result, and the deduction result is sent to a deduction result receiving end.

[0047] It should be noted that deduction means the process of running the model of the node and processing the to-be-processed data.

[0048] The following will be described in combination with Figure 2 The specific embodiment describes the method for collaborative deduction of the deep learning model based on knowledge distillation provided by the present application.

[0049] Figure 2 is a flowchart of the method for collaborative deduction of the deep learning model based on knowledge distillation provided by the present application. As Figure 2 indicated, the method for collaborative deduction of the deep learning model based on knowledge distillation provided by the present embodiment includes:

[0050] Step 201, obtaining the floating-point operations per second (FLOPS), storage capacity and data successful transmission probability corresponding to each edge node.

[0051] Optionally, the execution subject of the method for collaborative deduction of the deep learning model based on knowledge distillation provided by the present application can be an electronic device, or a device for collaborative deduction of the deep learning model based on knowledge distillation arranged in the electronic device. The device for collaborative deduction of the deep learning model based on knowledge distillation can be realized by the combination of software and / or hardware.

[0052] For example, the electronic device can be a server, a desktop computer or the like.

[0053] Optionally, the floating-point operations per second is the floating-point operations per second (FLOPS) performed by the edge node.

[0054] Optionally, the data successful transmission probability is the probability of successful data transmission between the edge node and the data node (a preset node of each edge node, and the data node is used to obtain the to-be-processed data).

[0055] Step 202, clustering each edge node based on the FLOPS, the storage capacity and the data successful transmission probability to obtain K target node clusters.

[0056] Optionally, each target node cluster includes at least one edge node.

[0057] Optionally, the K target node clusters satisfy the following clustering processing rule:

[0058]

[0059] wherein, denotes a set containing multiple target node clusters, M k denotes a target node cluster, m i denotes an edge node, denotes m i storage capacity, denotes M k average storage capacity, denotes m i FLOPS, denotes M k average FLOPS, denotes M k cumulative transmission success probability, p th denotes a preset probability threshold, M i denotes an i-th target node cluster, M j denotes a j-th target node cluster, and ∩ denotes an intersection.

[0060] Optionally, the cumulative transmission success probability of the node cluster can be obtained by formula 1.

[0061] wherein, I is the number of edge nodes included in the node cluster M k .

[0062] Optionally, the cumulative transmission success probability of the node cluster can be obtained by formula 2.

[0063] Optionally, the data success transmission probability of each edge node in the node cluster is processed by the probability calculation model to obtain the cumulative transmission success probability of the node cluster.

[0064] The probability calculation model is:

[0065] wherein, denotes the cumulative transmission success probability of M k , denotes the data success transmission probability of m i , and ∏(·) denotes a cumulative operation.

[0066] It should be noted that, in step 202, the clustering processing of the edge nodes is performed based on the FLOPS, the storage capacity and the data success transmission probability to obtain K target node clusters, so that the edge nodes in the target node clusters can satisfy a preset distance threshold, and the cumulative success transmission probability can satisfy a preset probability threshold.

[0067] Specifically, the detailed description of the K target node clusters can be found in Figure 3 the embodiments.

[0068] In step 203, the multiple convolution filters of the last convolution layer in the preset teacher model are subjected to set partitioning processing to obtain K filter sets.

[0069] Optionally, the preset teacher model is a pre-trained DNN model.

[0070] Optionally, each filter set in the K filter sets includes at least one convolution filter.

[0071] For detailed description of obtaining the K filter sets, please refer to Figure 4 Embodiments.

[0072] In step 203, the multiple convolution filters of the last convolution layer in the preset teacher model are subjected to set partitioning processing to obtain K filter sets, which can make important filters uniformly distributed in each filter set.

[0073] In step 204, based on the K target node clusters, the K filter sets and the multiple preset models, a to-be-trained model of each target node cluster is determined.

[0074] For detailed description of determining the to-be-trained model of each target node cluster, please refer to Figure 5 Embodiments.

[0075] In step 204, based on the K target node clusters, the K filter sets and the multiple preset models, a to-be-trained model of each target node cluster is determined, which can make the capability of each target node cluster adapt to the to-be-trained model.

[0076] In step 205, based on the multiple preset sample data, a student model of each target node cluster is obtained by jointly training the to-be-trained model of each target node cluster using a knowledge distillation technology.

[0077] Optionally, the student model of each target node cluster is obtained by jointly training the to-be-trained model of each target node cluster based on the multiple preset sample data using the knowledge distillation and a classic stochastic gradient descent method.

[0078] In the process of joint training, the calculation model of the loss function value is as follows:

[0079]

[0080] wherein, L(θ S ) represents the loss function value, θ s represents a parameter vector (including parameters of the to-be-trained model of each target node cluster), represents a standard cross-entropy operation, represents a hard label loss in the knowledge distillation loss function, y represents a real label vector (including real labels of a plurality of preset sample data), P S represents a predicted label vector (a predicted label output by a to-be-trained model of each target node cluster), represents a soft label loss in the knowledge distillation loss function, represents a probability distribution of a label output by a softmax layer of the preset teacher model, P S τ represents a probability distribution of a label output by a softmax layer of a to-be-trained model of each target node cluster, P represents one filter set in the K filter sets, represents a set including K filter sets, represents a convolutional layer activation value vector of P in the preset teacher model, represents a convolutional layer activation value vector of P in the to-be-trained student model, represents an activation migration loss of migrating the knowledge of a convolutional filter from the teacher model to the to-be-trained model, a represents a weight of the hard label loss, and β represents a weight of the soft label loss.

[0081] Optionally, the to-be-trained models of each target node cluster satisfy the following construction and allocation rules:

[0082]

[0083] wherein S represents a set including to-be-trained models of each target node cluster, S i represents an i-th to-be-trained model in S, S j represents a j-th to-be-trained model in S, f(S i ) represents a corresponding relationship between the i-th to-be-trained model and the target node cluster, g(S i ) represents a corresponding relationship between the i-th to-be-trained model and its corresponding partition, T f represents a set of a plurality of convolutional filters in the last convolutional layer of the preset teacher model, Rule R represents a preset rule that needs to be satisfied to obtain K filter sets, and ∩ represents a set intersection operation, represents a storage capacity requirement of the i-th to-be-trained model, represents a target storage capacity of the node cluster corresponding to the i-th to-be-trained model, t(S i ,f(S i )) represents a deduction delay of the target node cluster when executing deduction of the corresponding S i .

[0084] Specifically, detailed description of Rule R can be referred to Figure 4 Embodiments.

[0085] Step 206, respectively deploying the student model of each target node cluster to each edge node in the target node cluster; each edge node in each target node cluster is used to execute cooperative inference when running the corresponding student model.

[0086] In Figure 2 In the embodiment, the K target node clusters are obtained by clustering the edge nodes based on FLOPS, storage capacity and data successful transmission probability, the purpose is to redundantly backup the student model in the node cluster, so that other edge nodes can continue to infer in the case of failure of a certain edge node in the node cluster, and ensure that the cumulative successful transmission probability of each node cluster meets the requirements, thereby improving the flexibility of cooperative inference facing failure. In addition, the multiple convolution filters of the last convolution layer in the preset teacher model are subjected to set partitioning processing to obtain K filter sets, which ensures that important filters are uniformly distributed in the K filter sets, so that each filter set has equal importance to the inference result. Further, when determining the model to be trained, the FLOPS and storage capacity of each edge node are considered comprehensively, and the preset model and filter set suitable for the ability of each node cluster are allocated to each node cluster, so that the inference delay of each node cluster is close and the resources are fully utilized, thereby improving the inference performance.

[0087] Figure 3 is a method flowchart provided by the application for obtaining K target node clusters. As shown in Figure 3 , the method comprises:

[0088] Step 301, clustering the edge nodes based on FLOPS, storage capacity and data successful transmission probability to obtain multiple initial node clusters.

[0089] In some embodiments, step 301 specifically comprises:

[0090] arranging the multiple edge nodes in order from small to large FLOPS, and in the case of the same FLOPS, from small to large storage capacity, to obtain a node set;

[0091] determining the first edge node in the node set as the cluster head node of the preset node cluster;

[0092] performing a node division operation on each edge node in the node set except the first edge node: obtaining an ith node cluster set; determining distances between the edge node and cluster head nodes of each node cluster in the ith node cluster set based on FLOPS and storage capacity of the edge node and FLOPS and storage capacity of the cluster head nodes; arranging the node clusters in ascending order of the distances to obtain a target node cluster set; determining cumulative transmission success probabilities of the node clusters in the target node cluster set based on data successful transmission probabilities; if there is a first node cluster in the target node cluster set whose cumulative transmission success probability is less than a preset probability threshold and whose distance between the cluster head node and the edge node is less than a preset distance threshold, dividing the edge node into the first node cluster to obtain an (i+1)th node cluster set and updating the cluster head node of the first node cluster; otherwise, creating a new node cluster and dividing the edge node into the new node cluster to obtain the (i+1)th node cluster set, wherein the (i+1)th node cluster set includes each node cluster in the ith node cluster set and the new node cluster;

[0093] updating the ith node cluster set to the (i+1)th node cluster set, repeating the node division operation N times to obtain a plurality of initial node clusters, wherein N is a total number of the edge nodes in the node set except the first edge node;

[0094] Initially, i is equal to 1, and the ith node cluster set includes a preset node cluster.

[0095] The node set is described below in combination with a specific example 1.

[0096] In example 1, in a case where the edge nodes include m1, m2 and m3, if FLOPS of m1 is 30 million per second (M), storage capacity of m1 is 32 gigabytes (GB), FLOPS of m2 is 40 M, storage capacity of m2 is 128 GB, and FLOPS of m3 is 40 M, storage capacity of m3 is 64 GB, the node set obtained in ascending order of FLOPS and in descending order of storage capacity when FLOPS is the same is {m1, m3, m2}.

[0097] Next, the (i+1)th node cluster set is described below in combination with a specific example 2.

[0098] In Example 2, for an edge node m1 in the node set except the first edge node, when the ith node cluster set is {M1, M2, M3}, the distance between the cluster center node of M1 and m1 is 30, the distance between the cluster center node of M2 and m1 is 20, the distance between the cluster center node of M3 and m1 is 21, and the preset distance threshold is 25, the target node cluster set is {M2, M3, M1}; when the cumulative transmission success probability of M1 is 90%, the cumulative transmission success probability of M2 is 91%, the cumulative transmission success probability of M3 is 91%, and the preset probability threshold is 95%, the edge node m1 is divided into M2, and the ith+1 node cluster set is obtained, which includes M3, M1 and M2 with m1.

[0099] In some embodiments, based on the FLOPS and storage capacity of the edge node and the FLOPS and storage capacity of the cluster center nodes of the nodes in the ith node cluster set, the distance between the edge node and the cluster center nodes of the nodes in each node cluster is determined, including:

[0100] The FLOPS and storage capacity of the edge node and the FLOPS and storage capacity of the cluster center nodes of the nodes in each node cluster are processed by the distance calculation model to obtain the distance between the edge node and the cluster center nodes of the nodes in each node cluster.

[0101] The distance calculation model is:

[0102] wherein, m i represents the edge node, M k represents the node cluster in the ith node cluster set, represents the cluster center node of M k , d represents the distance between m i and the cluster center node of M k , represents the storage capacity of m i , represents the storage capacity of the cluster center node of M k , represents the FLOPS of m i , represents the FLOPS of the cluster center node of M k .

[0103] In some embodiments, the cumulative transmission success probability of each node cluster in the target node cluster set is determined based on the data successful transmission probability by the above probability calculation model, which is not described herein.

[0104] Optionally, step 301 can also be implemented by the following code:

[0105]

[0106] Step 302, adjust the edge nodes in the plurality of initial node clusters to obtain K target node clusters.

[0107] In some embodiments, step 302 specifically comprises:

[0108] performing a node adjustment operation: obtaining a plurality of i-th node clusters; determining, in the plurality of i-th node clusters, a target i-th node cluster with the smallest cumulative successful transmission probability; in the case that the cumulative successful transmission probability of the target i-th node cluster is less than a preset probability threshold, determining, for each edge node in the target i-th node cluster, a distance between the edge node and the cluster head node of each other node cluster; dividing the edge node into the other node cluster corresponding to the smallest distance to obtain a plurality of (i+1)-th node clusters; wherein each other node cluster is a node cluster other than the target i-th node cluster in the plurality of i-th node clusters, and the cluster head node of the other node cluster is a node determined based on the FLOPS and storage capacity of each edge node in the other node cluster;

[0109] updating the plurality of i-th node clusters to the plurality of (i+1)-th node clusters, repeatedly performing the node adjustment operation until the cumulative successful transmission probabilities of the final plurality of node clusters are all greater than or equal to the preset probability threshold, and determining the final plurality of node clusters as the K target node clusters;

[0110] Initially, i is equal to 1, and the plurality of i-th node clusters are the plurality of initial node clusters.

[0111] The following will be described in conjunction with specific example 3 to obtain the plurality of (i+1)-th node clusters.

[0112] Example 3, the plurality of i-th node clusters include M1, M2, and M3, wherein the cumulative successful transmission probability of M3 is the smallest, i.e., M3 is the target i-th node cluster. In the case that the cumulative successful transmission probability of M3 is less than the preset probability threshold, M3 includes edge nodes m1 and m2, the distance from m1 to the cluster head node of M1 is 20, the distance from m1 to the cluster head node of M2 is 25, and since the distance from m1 to the cluster head node of M1 is the smallest, m1 is divided into M1. The distance from m2 to the cluster head node of M1 is 21, the distance from m2 to the cluster head node of M2 is 19, and since the distance from m2 to the cluster head node of M2 is the smallest, m2 is divided into M2.

[0113] Optionally, step 302 can also be implemented by the following code:

[0114] / / Sort the plurality of node clusters in ascending order of cumulative successful transmission probability;

[0115] / / In the case that the cumulative successful transmission probability of the first node cluster M1 after sorting is less than the preset probability threshold, perform the following code;

[0116] The edge node in M1 is divided into the other node cluster corresponding to the minimum distance between the edge node in M1 and the cluster center node of each other node cluster

[0117]

[0118] / / Sort the multiple node clusters in ascending order according to the cumulative successful transmission probability;

[0119] end while

[0120] Return / / Output K target node clusters.

[0121] In Figure 3 In the embodiment, the edge nodes are clustered based on FLOPS, storage capacity and data successful transmission probability, a plurality of initial node clusters are obtained, and the edge nodes in the plurality of initial node clusters are adjusted to obtain K target node clusters, so that the intra-cluster similarity of each target node cluster is maximized, the resources are fully utilized when the same model is deployed, and the inference completion delay of each node in the same cluster is similar. In the case that some edge nodes in the target node cluster run out of battery or fail to communicate and cannot run, other normally running edge nodes in each target node cluster can run the student model as redundant nodes without interference, thereby improving the inference robustness.

[0122] Figure 4 It is a method flow diagram for obtaining K filter sets provided by the application. As Figure 4 shown, the method comprises:

[0123] Step 401, obtaining the average activation value of each convolution filter.

[0124] The average activation value is an important evaluation index of the convolution filter.

[0125] Optionally, the average activation value of each convolution filter can be obtained by the following method: for each convolution filter, input the preset verification data into each channel of the convolution filter to obtain the verification value output on each channel, and determine the average scalar value of each verification value as the average activation value of the convolution filter.

[0126] Step 402, determining the adjacency weight matrix of the target graph based on the average activation value; wherein the target graph is a complete graph constructed based on the plurality of convolution filters.

[0127] Optionally, a complete graph G = G(V, E) is constructed based on multiple convolution filters in the preset teacher model through the graph partitioning method, where V represents the vertex set and V = T f , E represents the edge set, E={e i,j}, e i,j represents the i-th convolution filter T fi and the jth convolution filter T fj There are edges between .

[0128] Optionally, the elements in the adjacency weight matrix of the target graph are edge weights between convolution filters in the target graph.

[0129] Alternatively, the edge weights between convolution filters in the target graph can be obtained by the following formula 3:

[0130] w ij =w ji =∑ val a i a j |a i -a j | Formula 3;

[0131] Among them, w ij Represents the convolution filter T in the target graph fi and T fj The edge weight between them, w ji Represents the convolution filter T in the target graph fj and T fi The edge weight between i Denotes the convolution filter T fi The average activation value, a j Denotes the convolution filter T fj The average activation value of .

[0132] Optionally, steps 401 to 402 may be implemented through the following code.

[0133]

[0134] Step 403: Segment the target graph using a canonical segmentation algorithm of spectral clustering based on the adjacency weight matrix and K to obtain K filter sets.

[0135] From formula 3, we can see that at T fi and T fJ The closer the average activation value is to IJThe smaller, since the average activation value is an importance evaluation index of the convolution filter, in the complete graph G=G(V, E), the importance of the convolution filter and the distribution of the convolution filter in the complete graph is: the distribution of the convolution filter with high importance and the convolution filter with general importance is relatively close, and the distribution of two convolution filters with close importance is relatively distant.

[0136] Based on the above distribution relationship, under the condition of ensuring that the total sum of edge weights of the complete graph is minimum and the total sum of edge weights in each subgraph is maximum, the complete graph G=G(V, E) is segmented by the standard segmentation algorithm of spectral clustering, K subgraphs are obtained, and all convolution filters included in a subgraph are confirmed as a filter set, so as to obtain K filter sets.

[0137] Optionally, the step 403 can also be implemented by the following code:

[0138] / / w represents the adjacent weight matrix of the complete graph constructed based on the plurality of convolution filters (composed of w Ij The standard segmentation algorithm of spectral clustering is used to divide the target graph to obtain P, and P represents a set including K filter sets.

[0139] It should be noted that the above Rule R is the steps 401 to 403.

[0140] In Figure 4 the embodiment, the standard segmentation algorithm of spectral clustering is used to segment the target graph based on the adjacent weight matrix and K, to obtain K filter sets, so that the important convolution filters in the filter set can be uniformly distributed among the K filter sets, avoiding the defect that the importance difference of the convolution filter set greatly affects the deduction accuracy, thereby improving the deduction accuracy.

[0141] Figure 5 is a method flow diagram provided by the application for determining the trained model of each target node cluster.

[0142] As Figure 5 shown, the method comprises:

[0143] Step 501, determining K target preset models in a plurality of preset models based on the target storage capacity of each target node cluster and the storage capacity requirement of each preset model; the target storage capacity of the target node cluster is the minimum storage capacity of at least one edge node in the corresponding target node cluster.

[0144] Optionally, for each target node cluster, based on the target storage capacity of the target node cluster and the storage capacity requirements of each preset model, a plurality of candidate models are determined from the plurality of preset models; the storage capacity requirement of each of the plurality of candidate models is less than the target storage capacity of the target node cluster.

[0145] One model randomly selected from the plurality of candidate models is determined as a target preset model.

[0146] For example, in the case where the target node cluster is M1 and M1 includes edge nodes m1, m2 and m3, if the storage capacity of m1 is 32 GB, the storage capacity of m2 is 64 GB, and the storage capacity of m3 is 16 GB, the target storage capacity of M1 is 16 GB.

[0147] Step 502, based on the K target preset models and the K filter sets, K to-be-trained models are constructed.

[0148] Optionally, the K target preset models are sorted in ascending order of floating point operations (FLOPs), to obtain sorted K target preset models;

[0149] The K filter sets are sorted in ascending order of the size of the filter set, to obtain sorted K filter sets;

[0150] The sorted K target preset models and the sorted K filter sets are one-to-one corresponding in the ascending order, to obtain the filter set corresponding to each target preset model;

[0151] For each target preset model, based on the filter set corresponding to the target preset model and the target preset model, a to-be-trained model is constructed.

[0152] The size of the filter set is the number of convolution filters included in the filter set.

[0153] For example, if there are sorted 3 target preset models {Y1, Y3, Y2} and sorted 3 filter sets {P3, P1, P2}, the 3 to-be-trained models constructed are S1 = buildmodel(Y1, P3), S2 = buildmodel(Y3, P1) and S3 = buildmodel(Y2, P2). Wherein, buildmodel means replacing the filter of the last convolution layer in the target preset model with the corresponding filter set. For example, buildmodel(Y3, P1) means replacing the filter of the last convolution layer in Y3 with the filter set P1.

[0154] Optionally, steps 501 to 502 can also be implemented through the following code:

[0155]

[0156] for i=1→K do

[0157] / / Select a preset model based on the target storage capacity of the target node cluster M i ;

[0158]

[0159] end for

[0160] / / Construct a to-be-trained model based on the FLOPs of the preset model and the target filter set size.

[0161] Step 503: Determine the to-be-selected model set of each target node cluster based on the target storage capacity of each target node cluster and the K to-be-trained models.

[0162] Optionally, for each target node cluster, the to-be-trained models with storage capacity requirements less than the target storage capacity of the target node cluster are determined as the to-be-selected model set of the target node cluster.

[0163] For example, in the case where the target node cluster is M1 and the three to-be-trained models are S1, S2 and S3, if the target storage capacity of M1 is 16 GB, the storage capacity requirement of S1 is 2 GB, the storage capacity requirement of S2 is 32 GB, and the storage capacity requirement of S3 is 12 GB, then the to-be-selected model set of M1 is {S1, S3}.

[0164] Optionally, step 503 can also be implemented through the following code:

[0165] for i=1→K do

[0166] S j ∈S} / / Add the to-be-trained models with storage capacity requirements less than or equal to the target storage capacity of the target node cluster to the to-be-selected model set, where S represents a set including K

[0167] to-be-trained models, represents the storage capacity requirement of a to-be-trained model, i represents a target node cluster, represents the to-be-selected model set of the target node cluster M i , S j represents a to-be-trained

[0168] model ​M represents i a target storage capacity;

[0169] end for

[0170] Step 504, based on the number of the to-be-trained models included in the to-be-selected model set and the average FLOPS of each target node cluster, sorting each target node cluster to obtain a node cluster set.

[0171] Optionally, for each target node cluster, determining a sum value of FLOPS of the edge nodes in the target node cluster; and determining a ratio of the sum value to a total number of the edge nodes in the target node cluster as the average FLOPS of the target node cluster.

[0172] Optionally, in the order from small to large of the number of the to-be-trained models included in the to-be-selected model set, or in the order from small to large of the average FLOPS of the target node cluster when the number of the to-be-trained models included in the to-be-selected model set is same, sorting each target node cluster to obtain the node cluster set.

[0173] For example, in the case that the target node clusters are M1, M2 and M3, if the average FLOPS of M1 is 10M, the average FLOPS of M2 is 15M, the average FLOPS of M3 is 20M, the to-be-selected model set of M1 includes 4 to-be-trained models, the to-be-selected model set of M2 includes 3 to-be-trained models, and the to-be-selected model set of M3 includes 3 to-be-trained models, then the node cluster set obtained by sorting M1, M2 and M3 is

[0174] {M2, M3, M1}.

[0175] Step 505, for each target node cluster in the node cluster set, in the case that there is a to-be-trained model satisfying a preset condition in the to-be-selected model set of the target node cluster, determining the to-be-trained model satisfying the preset condition as the to-be-trained model of the target node cluster.

[0176] The preset condition includes at least one of the following:

[0177] The to-be-selected model set includes one to-be-trained model;

[0178] The to-be-trained model has the minimum number of floating point operations FLOPs;

[0179] The to-be-trained model is not allocated a corresponding target node cluster.

[0180] For example, in the case of the node cluster set being {M2, M3, M1}, the candidate model set of M2 is {S1, S2}, the candidate model set of M3 is {S4}, the candidate model set of M1 is {S1, S2, S3}, if the FLOPs of S1 is 20M, the FLOPs of S2 is 25M, and the FLOPs of S3 is 20M, the trained model of M2 is S1, the trained model of M3 is S4, and the trained model of M1 is S3.

[0181] Alternatively, steps 504 to 505 can also be implemented by the following code:

[0182] / / sort each target node cluster according to the number of trained models included in the candidate model set from small to large, and according to the average FLOPS of the target node cluster from small to large in the case of the number of trained models included in the candidate model set being the same;

[0183]

[0184]

[0185] In Figure 5 the embodiments, based on the target storage capacity of each target node cluster and the storage capacity requirement of each preset model, K target preset models are determined from the plurality of preset models, K trained models are constructed based on the K target preset models and the K filter sets, in the case that there is a trained model satisfying the preset condition in the candidate model set of each target node cluster in the node cluster set, the trained model satisfying the preset condition is determined as the trained model of the target node cluster, which meets the requirement that the storage capacity of each edge node is different, and the trained model is assigned to each target node cluster to minimize the inference delay, thereby reducing the inference delay, and further, determining the trained model satisfying the preset condition as the trained model of the target node cluster can also avoid the problem of wasting computing resources in the uniform division manner in the prior art, and thus the computing resource utilization rate is maximized.

[0186] In some embodiments, after determining the trained model of the target node cluster, the trained model can also be subjected to an optimization operation, which will be described below in combination with Figure 6 the embodiments.

[0187] Figure 6 is a method flow diagram provided by the present application for optimizing the trained model of the target node cluster. As Figure 6 shown, the method comprises:

[0188] Step 601, for each target node cluster, determining the inference delay of the trained model of the target node cluster.

[0189] Optionally, the inference delay is determined by the following formula 4:

[0190] Wherein, t(S i , M k ) represents the inference delay of the target node cluster M k executing the to-be-trained model S i , represents the FLOPs of the to-be-trained model S i , represents the average FLOPS of the target node cluster M k , represents the result data volume of the corresponding filter set of the to-be-trained model S i , that is, the total number of bits of the output data of all convolution filters in the filter set, represents the average data transmission rate of each edge node in the target node cluster M k to the data node (represents the average value of the data transmission rate of each edge node to the data node, wherein the data transmission rate of the edge node to the data node is the pre-obtained transmission rate).

[0191] Step 602, in the to-be-trained model of each target node cluster, at least one second node cluster set including to-be-trained models with the same structure is determined, and partition adjustment processing is performed on each second node cluster set until the partition is stable, to obtain a first intermediate to-be-trained model of each target node cluster.

[0192] The to-be-trained models with the same structure are to-be-trained models whose layers except the last convolution layer are all the same.

[0193] For example, in the case where the target node cluster includes M1, M2, M3, and M4, if the to-be-trained models of M1 and M2 have the same structure, M1 and M2 are divided into one second node cluster set, and if the to-be-trained models of M3 and M4 have the same structure, M3 and M4 are divided into one second node cluster set.

[0194] Specifically, the partition adjustment processing includes:

[0195] Step 6021, for each second node cluster set, a first node cluster M maX corresponding to the maximum inference delay t I and a second node cluster M MIN corresponding to the minimum inference delay t J are determined.

[0196] Step 6022, it is judged whether there is a maximum integer k satisfying a preset adjustment condition:

[0197] MI the filter set P corresponding to the last convolution in the to-be-trained model i k convolution filters in P are partitioned into M j the filter set P corresponding to the last convolution in the to-be-trained model j M j the corresponding inference delay does not exceed M I the corresponding inference delay, and for any k'>k, partitioning k' convolution filters in P i into P j will result in M j the corresponding inference delay exceeds M i the corresponding inference delay.

[0198] Step 6023, if there is, based on the principle of spectral clustering, the k convolution filters with the smallest correlation in P i are partitioned into P j .

[0199] Step 6024, repeat steps 6021 to 6023 until the partition is stable, and obtain the first intermediate to-be-trained model of each target node cluster.

[0200] M j the corresponding inference delay is M j the inference delay of the to-be-trained module including the convolution filters in P i is executed.

[0201] M i the corresponding inference delay is M i the inference delay of the to-be-trained module excluding the convolution filters partitioned into P j is executed.

[0202] In the present application, it is determined whether there is a maximum integer k that satisfies the preset adjustment condition, which can minimize the influence of the above adjustment on the importance of the filter set, and the adjustment of the filter set can make the inference delays of the two clusters closest, thereby reducing t max .

[0203] Optionally, step 602 can also be implemented through the following code.

[0204]

[0205] Step 603, determine the inference delay of the first intermediate to-be-trained model of each target node cluster, and determine the target maximum inference delay t m in the inference delay.

[0206] Step 604, adjust the first intermediate to-be-trained model of each target node cluster to be larger in FLOPs and not exceed the target maximum inference delay tm The second intermediate to-be-trained model of each target node cluster.

[0207] Optionally, step 604 may be implemented by the following code.

[0208]

[0209] The model adjustment condition is satisfied when there are multiple preset models with FLOPs greater than the FLOPs of the first intermediate model to be trained, and the deduction delay does not exceed t m Preset model.

[0210] The second intermediate model to be trained of the target node cluster is a model with a larger FLOPs than the first intermediate model to be trained among multiple preset models, and the deduction delay does not exceed t m Preset model.

[0211] Step 605 : Perform partition adjustment processing on the second intermediate model to be trained of each target node cluster until the partition is stable, thereby obtaining the target model to be trained of each target node cluster.

[0212] Specifically, the specific process of step 605 is the same as the specific process of step 602, and will not be repeated here.

[0213] Furthermore, based on a plurality of preset sample data, the target models to be trained of each target node cluster are jointly trained to obtain student models of each target node cluster.

[0214] exist Figure 6 In an embodiment, through step 602, a portion of filters can be divided from the node cluster with the largest deduction delay to the node cluster with the smallest deduction delay, thereby averaging the maximum deduction delay and the minimum deduction delay to achieve the purpose of reducing the deduction delay. In addition, the partition adjustment processing based on the spectral clustering principle can make up for the defect of large size differences between each filter set, thereby improving the accuracy and robustness of the deduction. Furthermore, when determining the model to be trained, the FLOPS, storage capacity and data transmission rate of each edge node are comprehensively considered, and a preset model and filter set adapted to its capabilities is allocated to each node cluster, so that the deduction delay of each node cluster is close and the resources are fully utilized, thereby improving the deduction performance.

[0215] Furthermore, in step 604, a model with stronger learning ability can be adopted without affecting the maximum deduction delay, which is beneficial to improving the computing resource utilization of the target node cluster and maximizing the deduction accuracy.

[0216] Combined with Table 1 and Table 2, Figures 7 to 9The performance of the knowledge distillation based deep learning model collaborative inference method of the present application is described. Based on the teacher model trained on the CIFAR-10 dataset, the simulation inference results of the knowledge distillation based deep learning model collaborative inference method (i.e. my_NoNN), my_baseline, non_Redundancy and uniform_NoNN provided by the present application are compared.

[0217] my_baseline and my_NoNN are similar in thought, while considering the heterogeneity of edge networks and the robustness of inference. The differences between my_baseline and my_NoNN are as follows: (1) the average successful transmission probability is used to guide the node clustering process in my_baseline; (2) the construction and allocation process of the to-be-trained model strictly satisfies the constraint of inference delay. That is, given the communication delay threshold and the calculation delay threshold as constraints, the size of the filter set and the model structure allocated to each node cluster are determined according to formula 4, and then the multiple convolution filters are sorted by importance and allocated one by one.

[0218] non-Redundancy only considers the difference in storage capacity and FLOPS of edge nodes, does not consider setting redundant nodes to improve robustness within a node cluster, and does not consider the clustering process of edge nodes.

[0219] uniform_NoNN does not consider the difference in storage capacity and FLOPS of edge nodes, nor does it consider clustering edge nodes to obtain node clusters and then setting redundant nodes to improve robustness. Instead, each edge node runs a student model, the filter set size corresponding to the student model is balanced, and the preset model structure used is also the same.

[0220] my_NoNN, Teacher, non_Redundanc and uniform_NoNN are all implemented based on the Pytorch framework. For all simulation inferences, 8 edge nodes are used for inference.

[0221] Table 1 exemplarily shows the simulation collaborative inference performance comparison results of Teacher, my_NoNN, my_baseline, non_Redundancy and uniform_NoNN.

[0222] Table 1

[0223]

[0224] From Table 1, it can be seen that, compared with the teacher model, my_NoNN can obtain a student model with significantly reduced parameter quantity and FLOPs, thereby better adapting to the different storage capacity and FLOPS of the edge node, and the loss of inference accuracy is only less than 0.7%, and the inference accuracy of my_NoNN is the highest compared with the non_Redu ndancy method and the uniform_NoNN.

[0225] Figure 7 is one of the collaborative inference performance simulation result comparison diagrams of the deep learning model collaborative inference method based on knowledge distillation provided by the application. As shown in Figure 7 , it comprises: simulation results obtained by using my_NoNN, Teacher, non_Redundanc and uniform_NoNN.

[0226] In the simulation inference process Figure 7 , a faulty edge node is set to test the robustness of each scheme, and it can be seen from Figure 7 that as the number of faulty nodes increases, the lost inference result also increases, so the inference accuracy of my_NoNN, non_Redundanc and uniform_NoNN all show a significant downward trend. Compared with the uniform_NoNN and non_Redundanc schemes which do not introduce redundant student models, the inference accuracy decline curve of my_NoNN is obviously more gentle. When there are 4 faulty edge nodes, my_NoNN can still achieve an average inference accuracy of 88.08%, and the above results show that my_NoNN can effectively improve the inference accuracy and the robustness of inference.

[0227] Figure 8 is the second collaborative inference performance simulation result comparison diagram of the deep learning model collaborative inference method based on knowledge distillation provided by the application. As shown in Figure 8 , it comprises: simulation results obtained by using non_Redundanc, my_baseline, uniform_NoNN and my_NoNN based on 6 kinds of heterogeneity levels of edge nodes.

[0228] It should be noted that Figure 8 the results shown are obtained based on the 6 kinds of heterogeneity levels in Table 2.

[0229] Table 2

[0230] Heterogeneity level 0 1 2 3 4 5 FLOPS maximum difference 0 10 15 20 25 30 Data transfer rate maximum difference 0 100 200 300 400 500

[0231] From Table 2 and Figure 8It can be seen that when all devices in the cluster are homogeneous, the collaborative inference performance of my_NoNN is basically the same as that of the other three schemes. As the degree of heterogeneity of the edge node set increases, uniform_NoNN, which does not consider the heterogeneity of the edge node set, uses almost the same preset model for all edge nodes, resulting in a significant increase in inference delay; while the other three schemes can better adapt to the heterogeneity of the edge node set, and the inference delay presents a flat change trend. Among them, my_NoNN adjusts the filter set and model structure through iteration in the tuning stage to adapt to the heterogeneity of the edge node set, so it can obtain the minimum inference delay under different heterogeneity levels.

[0232] Figure 9 Figure 3 is a third comparison diagram of simulation results of the collaborative inference performance of the deep learning model collaborative inference method based on knowledge distillation provided by the application. As shown in Figure 9 , it includes simulation results obtained by using my_NoNN, Teacher, non_Redundanc and uniform_NoNN.

[0233] In the simulation inference of Figure 9 , the average transmission success probability of the edge node set and the preset probability threshold p th are changed.

[0234] As can be seen from Figure 9 , as the average transmission success probability of the edge node set increases, which means that the communication condition becomes better, the inference delay of the system gradually decreases under different p th . This is because the better the communication condition, the smaller the redundancy of the student model, so the number of node clusters increases, and the size of the filter set decreases, which reduces the inference delay. When the average successful transmission probability of the edge node set is determined, a larger p th can improve the robustness of the system, but usually leads to a larger inference delay. Therefore, the selection of p th needs to consider the trade-off between robustness and inference delay.

[0235] The deep learning model collaborative inference device based on knowledge distillation provided by the application is described below. The deep learning model collaborative inference device based on knowledge distillation described below can be mutually corresponding to the deep learning model collaborative inference method based on knowledge distillation described above.

[0236] Figure 10 Figure 1 is a structural schematic diagram of the deep learning model collaborative inference device based on knowledge distillation provided by the application. As shown in Figure 10 , the deep learning model collaborative inference device based on knowledge distillation comprises:

[0237] The acquisition module 1010 is configured to acquire floating point operations per second (FLOPS), storage capacity and data successful transmission probability corresponding to each edge node.

[0238] The clustering module 1020 is configured to perform clustering processing on the edge nodes based on the FLOPS, the storage capacity and the data successful transmission probability, to obtain K target node clusters.

[0239] The division module 1030 is configured to perform set division processing on a plurality of convolution filters of a last convolution layer in a preset teacher model, to obtain K filter sets.

[0240] The determination module 1040 is configured to determine a to-be-trained model of each target node cluster based on the K target node clusters, the K filter sets and a plurality of preset models.

[0241] The training module 1050 is configured to perform joint training on the to-be-trained models of the target node clusters based on a plurality of preset sample data and using a knowledge distillation technology, to obtain student models of the target node clusters.

[0242] The deployment module 1060 is configured to respectively deploy the student models of the target node clusters to each edge node in the target node cluster for each target node cluster, and each edge node in the target node cluster is configured to perform cooperative inference when running the corresponding student model.

[0243] According to the deep learning model cooperative inference device based on knowledge distillation provided by the application, the clustering module 1020 is specifically configured to:

[0244] Perform clustering processing on the edge nodes based on the FLOPS, the storage capacity and the data successful transmission probability, to obtain a plurality of initial node clusters.

[0245] Adjust the edge nodes in the plurality of initial node clusters, to obtain the K target node clusters.

[0246] According to the deep learning model cooperative inference device based on knowledge distillation provided by the application, the clustering module 1020 is specifically configured to:

[0247] Arrange the plurality of edge nodes in ascending order of the FLOPS, or in ascending order of the storage capacity when the FLOPS are the same, to obtain a node set.

[0248] Determine a first edge node in the node set as a cluster center node of a preset node cluster.

[0249] performing a node division operation on each edge node in the node set except the first edge node: obtaining an ith node cluster set; determining distances between the edge node and cluster center nodes of each node cluster in the ith node cluster set based on FLOPS and storage capacity of the edge node and FLOPS and storage capacity of the cluster center nodes of each node cluster; arranging the node clusters in ascending order of the distances to obtain a target node cluster set; determining cumulative transmission success probabilities of each node cluster in the target node cluster set based on data successful transmission probabilities; if there is a first node cluster in the target node cluster set whose cumulative transmission success probability is less than the preset probability threshold and whose distance between the cluster center node and the edge node is less than a preset distance threshold, dividing the edge node into the first node cluster to obtain an (i+1)th node cluster set and updating the cluster center node of the first node cluster; otherwise, creating a new node cluster, dividing the edge node into the new node cluster to obtain the (i+1)th node cluster set, wherein the (i+1)th node cluster set includes each node cluster in the ith node cluster set and the new node cluster;

[0250] updating the ith node cluster set to the (i+1)th node cluster set, repeating the node division operation N times to obtain the plurality of initial node clusters, wherein N is a total number of the edge nodes in the node set except the first edge node;

[0251] Initially, i is equal to 1, and the ith node cluster set includes the preset node cluster.

[0252] According to the deep learning model collaborative deduction device based on knowledge distillation provided by the application, the clustering module 1020 is specifically used for:

[0253] The distance calculation model is used to process FLOPS and storage capacity of the edge node and FLOPS and storage capacity of the cluster center nodes of each node cluster to obtain distances between the edge node and the cluster center nodes of each node cluster.

[0254] The distance calculation model is:

[0255] Wherein, m i represents the edge node, M k represents the node cluster, represents the cluster center node of the node cluster, and d represents the distance between the edge node and the cluster center node of the node cluster, represents storage capacity of the edge node, represents storage capacity of the cluster center node of the node cluster, represents FLOPS of the edge node, Indicates the FLOPS of the core node of the node cluster.

[0256] According to a deep learning model collaborative deduction device based on knowledge distillation provided by the present invention, the clustering module 1020 is specifically used to:

[0257] For each node cluster, the probability of successful data transmission of each edge node in the node cluster is processed using a probability calculation model to obtain a cumulative probability of successful data transmission of the node cluster;

[0258] The probability calculation model is:

[0259] in, represents the cumulative transmission success probability of the node cluster, M k Represents the node cluster, m i represents an edge node in the node cluster, represents the probability of successful data transmission of the edge node, and Π(·) represents the accumulation operation.

[0260] According to a deep learning model collaborative deduction device based on knowledge distillation provided by the present invention, the clustering module 1020 is specifically used to:

[0261] Performing a node adjustment operation: obtaining a plurality of i-th node clusters; determining a target i-th node cluster with the minimum cumulative probability of successful transmission among the plurality of i-th node clusters; when the cumulative probability of successful transmission of the target i-th node cluster is less than a preset probability threshold, determining, for each edge node in the target i-th node cluster, the distance between the edge node and the cluster center node of each other node cluster; dividing the edge node into other node clusters corresponding to the minimum distance to obtain a plurality of i+1-th node clusters; wherein each other node cluster is a node cluster other than the target i-th node cluster among the plurality of i-th node clusters, and the cluster center node of the other node cluster is a node determined based on the FLOPS and storage capacity of each edge node in the other node cluster;

[0262] Updating the plurality of i-th node clusters to the plurality of i+1-th node clusters, repeatedly performing the node adjustment operation until the cumulative successful transmission probability of each of the plurality of final node clusters is greater than or equal to the preset probability threshold, and determining the plurality of final node clusters as the K target node clusters;

[0263] Initially, i is equal to 1, and the multiple i-th node clusters are the multiple initial node clusters.

[0264] According to a deep learning model collaborative deduction device based on knowledge distillation provided by the present invention, the division module 1030 is specifically used to:

[0265] obtaining average activation values of the plurality of convolution filters respectively;

[0266] determining an adjacency weight matrix of a target graph based on the average activation values, wherein the target graph is a complete graph constructed based on the plurality of convolution filters;

[0267] performing segmentation on the target graph based on the adjacency weight matrix and K by a normalized cut algorithm of spectral clustering to obtain the K filter sets.

[0268] According to the deep learning model collaborative deduction device based on knowledge distillation provided by the application, the determination module 1040 is specifically used for:

[0269] determining K target preset models in the plurality of preset models based on target storage capacities of the target node clusters and storage capacity requirements of each preset model, wherein the target storage capacity of the target node cluster is the minimum storage capacity of at least one edge node in the corresponding target node cluster;

[0270] constructing K to-be-trained models based on the K target preset models and the K filter sets;

[0271] determining a to-be-selected model set of each target node cluster based on the target storage capacity of each target node cluster and the K to-be-trained models;

[0272] sorting the target node clusters based on the number of to-be-trained models included in the to-be-selected model set and the average FLOPS of each target node cluster to obtain a node cluster set;

[0273] in a case where there is a to-be-trained model satisfying a preset condition in the to-be-selected model set of each target node cluster in the node cluster set, determining the to-be-trained model satisfying the preset condition as a to-be-trained model of the target node cluster;

[0274] The preset condition includes at least one of the following:

[0275] the to-be-selected model set includes one to-be-trained model;

[0276] the to-be-trained model has the minimum number of floating point operations FLOPs;

[0277] the to-be-trained model is not allocated a corresponding target node cluster.

[0278] Figure 11 is a structural schematic diagram of an electronic device provided by the application. As Figure 11As shown, the electronic device can include a processor 1110, a communications interface 1120, a memory 1130, and a communications bus 1140. The processor 1110, the communications interface 1120, and the memory 1130 can communicate with each other through the communications bus 1140. The processor 1110 can invoke a logical instruction in the memory 1130 to execute the deep learning model collaborative inference method based on knowledge distillation.

[0279] In addition, the logical instruction in the memory 1130 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0280] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, and the computer can execute the deep learning model collaborative inference method based on knowledge distillation provided by the above-mentioned methods. In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the deep learning model collaborative inference method based on knowledge distillation provided by the above-mentioned methods.

[0281] The apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor. Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments. Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A deep learning model collaborative inference method based on knowledge distillation, characterized in that, The method comprises the following steps: obtaining the floating point operations per second (FLOPS), storage capacity and data successful transmission probability of each edge node; performing clustering processing on the edge nodes based on the FLOPS, the storage capacity and the data successful transmission probability to obtain K target node clusters; performing set partitioning processing on a plurality of convolution filters of a last convolution layer in a preset teacher model to obtain K filter sets; determining a to-be-trained model of each target node cluster based on the K target node clusters, the K filter sets and a plurality of preset models; performing joint training on the to-be-trained models of the target node clusters based on a plurality of preset sample data and using a knowledge distillation technology to obtain student models of the target node clusters; deploying the student models of the target node clusters to each edge node in the target node cluster respectively for each target node cluster, and each edge node in the target node cluster is used for performing cooperative inference when running the corresponding student model; wherein the K target node clusters satisfy the following clustering processing rules: wherein, denotes a set comprising a plurality of target node clusters, M k denotes a target node cluster, m i denotes an edge node, denotes m i a storage capacity, denotes M k an average storage capacity, denotes m i FLOPS, denotes M k an average FLOPS, denotes M k a cumulative transmission success probability, p th denotes a preset probability threshold, M i denotes an i-th target node cluster, M j denotes a j-th target node cluster, and ∩ denotes an intersection; In the process of joint training, the calculation model of the loss function value is: wherein L(θ S ) represents a loss function value, θ S represents a parameter vector, including parameters of the to-be-trained model of each target node cluster, represents a standard cross-entropy operation, represents a hard label loss in the knowledge distillation loss function, y represents a real label vector, including real labels of a plurality of preset sample data, P S represents a predicted label vector, a predicted label output by the to-be-trained model of each target node cluster, represents a soft label loss in the knowledge distillation loss function, represents a probability distribution of a label output by a softmax layer of the preset teacher model, P S τ represents a probability distribution of a label output by a softmax layer of the to-be-trained model of each target node cluster, P represents one filter set in the P, represents a set including K filter sets, represents a convolutional layer activation value vector of P in the preset teacher model, represents a convolutional layer activation value vector of P in the to-be-trained student model, represents an activation migration loss for migrating the knowledge of a convolutional filter from the teacher model to the to-be-trained model, α represents a weight of the hard label loss, and β represents a weight of the soft label loss.

2. The method of claim 1, wherein, The clustering processing on the edge nodes based on the FLOPS, the storage capacity and the data successful transmission probability to obtain K target node clusters comprises: performing clustering processing on the edge nodes based on the FLOPS, the storage capacity and the data successful transmission probability to obtain a plurality of initial node clusters; adjusting the edge nodes in the plurality of initial node clusters to obtain the K target node clusters.

3. The method of claim 2, wherein, The clustering processing on the edge nodes based on the FLOPS, the storage capacity and the data successful transmission probability to obtain a plurality of initial node clusters comprises: arranging the plurality of edge nodes in the order of the FLOPS from small to large, and in the case of the same FLOPS, the storage capacity from small to large to obtain a node set; determining the first edge node in the node set as the cluster head node of a preset node cluster; performing a node division operation on each edge node in the node set except the first edge node: obtaining an ith node cluster set; determining distances between the edge node and cluster center nodes of each node cluster in the ith node cluster set based on FLOPS and storage capacity of the edge node and FLOPS and storage capacity of the cluster center nodes; arranging the node clusters in ascending order of the distances to obtain a target node cluster set; determining cumulative transmission success probabilities of the node clusters in the target node cluster set based on data successful transmission probabilities; if there is a first node cluster in the target node cluster set whose cumulative transmission success probability is less than the preset probability threshold and whose distance between the cluster center node and the edge node is less than a preset distance threshold, dividing the edge node into the first node cluster to obtain an (i+1)th node cluster set and updating the cluster center node of the first node cluster; otherwise, creating a new node cluster, dividing the edge node into the new node cluster to obtain the (i+1)th node cluster set, wherein the (i+1)th node cluster set includes each node cluster in the ith node cluster set and the new node cluster; updating the ith node cluster set to the (i+1)th node cluster set, repeating the node division operation N times to obtain the plurality of initial node clusters, wherein N is a total number of edge nodes in the node set except the first edge node; initially, i is equal to 1, and the ith node cluster set includes the preset node cluster.

4. The method of claim 3, wherein, The determining of the distances between the edge node and the cluster center nodes of each node cluster based on FLOPS and storage capacity of the edge node and FLOPS and storage capacity of the cluster center nodes includes: processing the FLOPS and storage capacity of the edge node and the FLOPS and storage capacity of the cluster center nodes of each node cluster through a distance calculation model to obtain the distances between the edge node and the cluster center nodes of each node cluster; The distance calculation model is: wherein m i denotes the edge node, M k denotes the node cluster, denotes the cluster head node of the node cluster, d denotes the distance between the edge node and the cluster head node of the node cluster, denotes the storage capacity of the edge node, denotes the storage capacity of the cluster head node of the node cluster, denotes the FLOPS of the edge node, denotes the FLOPS of the cluster head node of the node cluster.

5. The method of claim 3, wherein, The determining of the cumulative transmission success probabilities of the node clusters in the target node cluster set based on data successful transmission probabilities includes: for each node cluster, processing data successful transmission probabilities of edge nodes in the node cluster through a probability calculation model to obtain the cumulative transmission success probability of the node cluster; The probability calculation model is: wherein, denotes the cumulative transmission success probability of the node cluster, M k denotes the node cluster, m i denotes an edge node in the node cluster, denotes the data success transmission probability of the edge node, denotes the cumulative operation.

6. The method of claim 2, wherein, The adjusting of the edge nodes in the plurality of initial node clusters to obtain the K target node clusters includes: Performing a node adjustment operation: obtaining a plurality of i-th node clusters; determining a target i-th node cluster with the smallest cumulative successful transmission probability among the plurality of i-th node clusters; in the case that the cumulative successful transmission probability of the target i-th node cluster is less than a preset probability threshold, determining, for each edge node in the target i-th node cluster, the distance between the edge node and the cluster head node of each other node cluster; dividing the edge node into the other node cluster corresponding to the smallest distance to obtain a plurality of i+1-th node clusters; wherein the other node cluster is a node cluster in the plurality of i-th node clusters except the target i-th node cluster, and the cluster head node of the other node cluster is a node determined based on the FLOPS and storage capacity of each edge node in the other node cluster; updating the plurality of i-th node clusters to the plurality of i+1-th node clusters, and repeatedly performing the node adjustment operation until the cumulative successful transmission probabilities of the final plurality of node clusters are all greater than or equal to the preset probability threshold, and determining the final plurality of node clusters as the K target node clusters; initially, i is equal to 1, and the plurality of i-th node clusters are the plurality of initial node clusters.

7. The method according to any one of claims 1 to 6, characterized in that, The set partitioning processing of the plurality of convolution filters of the last convolution layer in the preset teacher model obtains K filter sets, including: obtaining the average activation value of each of the plurality of convolution filters; determining the adjacency weight matrix of a target graph based on the average activation value; wherein the target graph is a complete graph constructed based on the plurality of convolution filters; segmenting the target graph based on the adjacency weight matrix and K by a normalized cut algorithm of spectral clustering to obtain the K filter sets.

8. The method according to any one of claims 1 to 6, characterized in that, The determination of the to-be-trained model of each target node cluster based on the K target node clusters, the K filter sets and a plurality of preset models includes: determining K target preset models from the plurality of preset models based on the target storage capacity of each target node cluster and the storage capacity requirement of each preset model; the target storage capacity of the target node cluster is the minimum storage capacity of at least one edge node in the corresponding target node cluster; constructing K to-be-trained models based on the K target preset models and the K filter sets; determining a to-be-selected model set of each target node cluster based on the target storage capacity of each target node cluster and the K to-be-trained models; sorting each target node cluster based on the number of to-be-trained models included in the to-be-selected model set and the average FLOPS of each target node cluster to obtain a node cluster set; in the case that there is a to-be-trained model satisfying a preset condition in the to-be-selected model set of each target node cluster in the node cluster set, determining the to-be-trained model satisfying the preset condition as the to-be-trained model of the target node cluster; the preset condition includes at least one of the following: the to-be-selected model set includes one to-be-trained model; the to-be-trained model has the minimum FLOPs; the corresponding target node cluster is not allocated.

9. A deep learning model collaborative inference device based on knowledge distillation, characterized in that, ​ An acquisition device is configured to acquire a floating point operations per second (FLOPS), a storage capacity, and a data successful transmission probability corresponding to each edge node; A clustering module is configured to perform clustering processing on the edge nodes based on the FLOPS, the storage capacity, and the data successful transmission probability, to obtain K target node clusters; A division module is configured to perform set division processing on a plurality of convolution filters of a last convolution layer in a preset teacher model, to obtain K filter sets; A determination module is configured to determine a to-be-trained model of each target node cluster based on the K target node clusters, the K filter sets, and a plurality of preset models; A training module is configured to perform joint training on the to-be-trained models of the target node clusters based on a plurality of preset sample data and a knowledge distillation technology, to obtain student models of the target node clusters; A deployment module is configured to deploy the student models of the target node clusters to each edge node in the target node cluster for each target node cluster, and each edge node in the target node cluster is configured to perform cooperative inference when running the corresponding student model. K target node clusters satisfy the following clustering processing rules: wherein, denotes a set comprising a plurality of target node clusters, M k denotes a target node cluster, m i denotes an edge node, denotes m i a storage capacity, denotes M k an average storage capacity, denotes m i a FLOPS, denotes M k an average FLOPS, denotes M k a cumulative transmission success probability, p th denotes a preset probability threshold, M i denotes an i-th target node cluster, M j denotes a j-th target node cluster, and ∩ denotes an intersection; In the process of joint training, the calculation model of the loss function value is: Among them, L(θ S ) represents the loss function value, θ S Represents the parameter vector, including the parameters of the model to be trained for each target node cluster, represents the standard cross entropy operation, represents the hard label loss in the knowledge distillation loss function, y represents the true label vector, including the true labels of multiple preset sample data, P S Represents the predicted label vector, the predicted label output by the trained model of each target node cluster, represents the soft label loss in the knowledge distillation loss function, P T τ Represents the probability distribution of the label output by the softmax layer of the preset teacher model, P S τ Represents the probability distribution of the labels output by the softmax layer of the training model for each target node cluster, P represents A filter set in represents a set of K filter sets, Represents the convolutional layer activation value vector of P in the preset teacher model, Represents the convolutional layer activation value vector of P in the student model to be trained, represents the activation transfer loss of transferring the knowledge of convolutional filters from the teacher model to the model to be trained, α represents the weight of the hard label loss, and β represents the weight of the soft label loss.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the deep learning model cooperative inference method based on knowledge distillation according to any one of claims 1 to 8.

Images