A multi-branch network collaborative reasoning method and system for the internet of things

Abstract

The disclosure provides a multi-branch network collaborative inference method and system for Internet of Things, and belongs to the field of computer vision algorithm acceleration of Internet of Things devices. The method comprises the following steps: inputting a to-be-predicted sample into a first branch of a preset multi-branch network on an Internet of Things device to obtain a corresponding initial prediction result and an uncertainty; obtaining an output branch corresponding to the sample in a preset distribution scheme of the multi-branch network according to the uncertainty; and obtaining a final prediction result of the sample by using the output branch according to a preset model division scheme of the multi-branch network; the model division scheme comprises a hierarchical calculation allocation result of each branch of the multi-branch network on the Internet of Things device and a corresponding server. The disclosure can realize on-demand adjustment of the multi-branch network collaborative inference, solve the challenge of distributed multi-branch network inference between cross-device servers, and ensure that the Internet of Things device stably provides services in a highly dynamic environment.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the field of computer vision algorithm acceleration of Internet of Things devices, and particularly relates to a multi-branch network collaborative inference method and system for Internet of Things. BACKGROUND

[0002] With the proliferation of computing and storage devices, from server clusters in cloud data centers to personal computers and smartphones, to wearable devices and other Internet of Things devices, we are now in an information-centric era, in which computing is ubiquitous and computing services are gradually moving from cloud servers to Internet of Things devices. However, the weak computing power of existing Internet of Things devices makes it difficult for them to process the data generated by the devices: 1) a large number of computing tasks need to be delivered to servers for processing, which undoubtedly poses a serious challenge to the communication capabilities of the network and the computing capabilities of the servers; 2) many new types of applications, such as cooperative autonomous driving and fault detection in smart factories, have strict delay requirements, and servers can be far from users, so it is difficult to meet these requirements. Therefore, how to enable Internet of Things devices to complete the processing of DNN (Deep Neural Network) models locally is a challenge that helps to alleviate the pressure brought about by the growth of data.

[0003] To solve the problem of Internet of Things devices executing computer vision models, existing solutions include server execution and device execution. In cloud server-centric solutions, data collected on Internet of Things devices is sent to cloud servers over the Internet, inference tasks are completed using accelerators on servers, and devices receive the results returned by servers. However, with the increasing capabilities of Internet of Things devices, the resolution of image data collected by devices is gradually increasing, and the frame rate of videos is also increasing. Moreover, as the center of servers, it is often necessary to process data from multiple devices, and transmitting raw data will bring a large communication and computing pressure to servers and networks. The main idea of edge computing is to migrate tasks from cloud servers and Internet of Things devices to servers at the edge of the network, which can reduce the impact of Internet fluctuations and reduce the pressure on the Internet, enabling devices to respond to image processing requirements in real time. However, edge computing is still affected by network volatility, and network deterioration can have a serious impact on the offloading of inference tasks.

[0004] The current deployment process of DNN models on IoT devices involves maintaining two models: a large, high-precision model on the server and a small, low-precision model on the device. However, this approach incurs significant deployment overhead. First, from a development time perspective, the dual-model approach requires training two models, resulting in two time- and resource-intensive phases. In the first phase, the design and training of the large model requires multiple GPUs to run for a long time. In the second phase, the large model is compressed using various techniques to obtain its lightweight counterpart, and selecting and tuning the compression method is itself a difficult task. Furthermore, to recover the accuracy loss caused by compression, the lightweight model must be fine-tuned through additional training steps.

[0005] While collaborative inference can achieve low-latency inference tasks compared to device-based and server-based execution, it still struggles to meet real-time requirements in certain scenarios and cannot adapt to dynamic changes in throughput. This is because the efficiency of collaborative inference is highly dependent on the available bandwidth between the server and IoT devices. Since communication latency accounts for a significant portion of the inference time, network unavailability can have disastrous consequences. In some traffic flow monitoring systems, vehicle numbers are correlated with time, with peak hour traffic significantly exceeding late-night traffic. This means that the data processed by the devices changes over time, requiring IoT devices to process the data in real-time. Summary of the Invention

[0006] The purpose of this disclosure is to overcome the shortcomings of existing technologies and propose a method and system for multi-branch network collaborative reasoning for the Internet of Things (IoT). This disclosure enables on-demand adjustable multi-branch network collaborative reasoning, solves the challenge of distributed multi-branch network reasoning across device servers, and ensures that IoT devices provide stable services in highly dynamic environments.

[0007] The first aspect of this disclosure proposes a method for collaborative reasoning in multi-branch networks for the Internet of Things, including:

[0008] On an IoT device, the sample to be predicted is input into the first branch of a preset multi-branch network to obtain the corresponding initial prediction result and uncertainty.

[0009] Based on the uncertainty, the output branch corresponding to the sample is obtained from the preset distribution scheme of the multi-branch network;

[0010] According to the preset model partitioning scheme of the multi-branch network, the final prediction result of the sample is obtained by using the output branch; the model partitioning scheme includes the hierarchical calculation allocation result of each branch of the multi-branch network on the IoT device and the corresponding server.

[0011] In one specific embodiment of this disclosure, obtaining the final prediction result of the sample using the output branch according to the preset model partitioning scheme of the multi-branch network includes:

[0012] 1) If the output branch corresponding to the sample is the first branch, then the initial prediction result is taken as the final prediction result of the sample;

[0013] 2) If the output branch corresponding to the sample is not the first branch, the final prediction result is obtained as follows:

[0014] 2-1) If the levels of the output branches corresponding to the samples are all assigned to the IoT device, then the final prediction result is calculated on the IoT device using the output branches;

[0015] 2-2) If all levels in the output branch corresponding to the sample are assigned to the server, the final prediction result is calculated on the server using the output branch and returned to the IoT device;

[0016] 2-3) If the level of the output branch corresponding to the sample is divided by the IoT device and the server respectively, then the intermediate result is first obtained through the level divided by the branch in the IoT device and sent to the server. Then the intermediate result is obtained through the level divided by the branch in the server to obtain the final prediction result and returned to the IoT device.

[0017] In one specific embodiment of this disclosure, the method further includes:

[0018] The initial prediction result includes the probability of each prediction category output by the sample via the first branch. The uncertainty of the sample is obtained by subtracting the second maximum value of the probability from the maximum value of the probability.

[0019] In one specific embodiment of this disclosure, the model partitioning scheme consists of model partitioning points for each branch of the multi-branch network, wherein the model partitioning points minimize the inference time of the branch.

[0020] In one specific embodiment of this disclosure, the method further includes:

[0021] If the output branch corresponding to the sample is not the first branch, then the output result of the backbone of the multi-branch network contained in the first branch is used to continue the calculation on the output branch to obtain the final prediction result.

[0022] In one specific embodiment of this disclosure, the method for determining the distribution scheme of the multi-branch network is as follows:

[0023] 1) Using the multi-branch network, calculate the uncertainty of each sample in the preset evaluation set, and determine the uncertainty distribution of the evaluation set; the evaluation set includes multiple samples and corresponding classification results;

[0024] 2) Based on the uncertainty distribution of the evaluation set, divide all samples in the evaluation set into M groups on an average basis to obtain the uncertainty level classification result, where M is the preset total number of uncertainty levels;

[0025] 3) Determine the initial distribution scheme, wherein the current output branch corresponding to each sample of uncertainty level in the evaluation set is the first branch of the multi-branch network;

[0026] 4) Let the current candidate branch corresponding to each uncertainty level be the next branch of the current output branch;

[0027] 5) Using the evaluation set, for each level of uncertainty, calculate the speedup ratio corresponding to the current candidate branch, whereby the speedup ratio is the ratio of the increase in prediction accuracy brought about by using the current candidate branch compared to the current output branch to the increase in inference time brought about by using the current candidate branch compared to the current output branch;

[0028] 6) Select the uncertainty level corresponding to the maximum speedup ratio among all current candidate branches, and use the current candidate branch of the uncertainty level as the new current output branch of the uncertainty level to obtain the updated current distribution scheme; update the current candidate branch of the uncertainty level to obtain the updated candidate branch set;

[0029] 7) Repeat steps 5) to 6) until all current candidate branches in the candidate branch set meet the set target requirements, then the current distribution scheme is taken as the final distribution scheme of the multi-branch network.

[0030] In a specific embodiment of this disclosure, the method for determining the model partitioning scheme is as follows:

[0031] 1) Update network bandwidth using the exponential moving average method, as shown in the following expression:

[0032] Band=(1-α)*Band+α*B_runtime

[0033] Where Band is the network bandwidth, B_runtime is the real-time network bandwidth; a is a hyperparameter, 0≤a≤1;

[0034] 2) Determine the optimization objective for the multi-branch network model partitioning:

[0035]

[0036] Where T represents the average inference time of the multi-branch network. p represents the reasoning time for the m-th branch. m This represents the probability that the m-th branch is selected.

[0037] 3) Determine the model partitioning points for each branch to obtain the model partitioning scheme for the multi-branch network;

[0038] The method for determining the model split point for any branch is as follows:

[0039] 3-1) Construct the directed acyclic graph corresponding to this branch;

[0040] Treat any branch as a DNN model and construct the corresponding directed acyclic graph G = (V, E); where V represents the set of nodes in graph G, each node is a layer in the DNN model corresponding to graph G; and E represents the set of edges in graph G, each edge is a link in the DNN model corresponding to graph G.

[0041] Let link l ij =(a i ,a j ) represents node a i The output is node a j The input, d i Representative node a i The output data size, then link l ij =(a i ,a j Network transmission time

[0042] Divide set V into two disjoint subsets Vi device and V edge V device V represents a subset of nodes that execute on IoT devices. edge Let L represent the subset of nodes executed on the server; let L represent the set of links between the two subsets, i.e., the model splitting points. Then the total latency of collaborative inference is the latency of executing subset V on the device. device Total execution time And executing subset V on the server edge Total execution time The sum of, among which, For node a i The execution time of the corresponding layer on the IoT device. For node a i The execution time of the corresponding layer on the server; the sum of the total data transmitted to the model partitioning point L. but:

[0043]

[0044] 3-2) Add two virtual nodes d and e to graph G; where d represents the IoT device and is the source node; and e represents the edge server node and is the destination node. Add new edges to graph G such that each edge corresponds to a delay, which includes network transmission time, execution time on the IoT device, and execution time on the edge server. After construction, a new directed acyclic graph is obtained, denoted as .

[0045] 3-3) Obtain the graph The minimum cut between the source node d and the destination node e is used as the model split point for this branch; with the cut as the boundary, in the graph... Nodes on the same side as the source node are assigned to perform computations on the IoT device, while nodes on the same side as the destination node are assigned to perform computations on the server.

[0046] A second aspect of this disclosure provides a multi-branch network collaborative reasoning system for the Internet of Things, comprising:

[0047] The initial prediction module, deployed on the IoT device, is used to input the sample to be predicted into the first branch of a preset multi-branch network to obtain the corresponding initial prediction result and uncertainty.

[0048] The output branch determination module is used to obtain the output branch corresponding to the sample from the preset distribution scheme of the multi-branch network based on the uncertainty.

[0049] The collaborative reasoning module is used to obtain the final prediction result of the sample by utilizing the output branch according to the preset model partitioning scheme of the multi-branch network; the model partitioning scheme includes the hierarchical calculation allocation results of each branch of the multi-branch network on the IoT device and the corresponding server.

[0050] A third aspect of this disclosure provides an electronic device, comprising:

[0051] At least one processor; and a memory communicatively connected to said at least one processor;

[0052] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-described multi-branch network collaborative reasoning method for the Internet of Things.

[0053] A fourth aspect of this disclosure provides a computer-readable storage medium storing computer instructions for causing the computer to execute the aforementioned multi-branch network collaborative reasoning method for the Internet of Things.

[0054] The features and beneficial effects of this disclosure are as follows:

[0055] 1) This disclosure addresses the challenges of distributed multi-branch network inference across devices and servers, enabling support for complex performance targets in highly dynamic environments while ensuring stable service delivery from IoT devices.

[0056] 2) This disclosure solves the model partitioning problem of multi-branch networks by optimizing the unified model partitioning scheme of multi-branch networks into finding a model partitioning scheme for a single branch, thus obtaining a more reasonable model partitioning scheme.

[0057] 3) This disclosure proposes a method for adaptive adjustment based on target requirements and network bandwidth changes. It can adaptively adjust the model partitioning and distribution schemes of a multi-branch network according to the current state to enhance the service experience of IoT devices and maintain their performance in edge computing environments. This disclosure can determine the optimal collaborative inference scheme in real time based on network bandwidth conditions without consuming excessive computing resources. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the structure of a multi-branch network in a specific embodiment of this disclosure.

[0059] Figure 2 This is an overall flowchart of a multi-branch network collaborative reasoning method for the Internet of Things in a specific embodiment of this disclosure.

[0060] Figure 3 This is a flowchart of the on-demand adjustment algorithm for the model partitioning scheme in a specific embodiment of this disclosure.

[0061] Figure 4 This is a schematic diagram of a DNN model in a specific embodiment of this disclosure.

[0062] Figure 5 This is a schematic diagram illustrating the principle of finding the minimum ST cut in an updated directed acyclic graph in a specific embodiment of this disclosure. Detailed Implementation

[0063] This disclosure presents a method and system for collaborative reasoning in multi-branch networks for the Internet of Things (IoT), which will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] The first aspect of this disclosure proposes a method for collaborative reasoning in multi-branch networks for the Internet of Things, including:

[0065] On an IoT device, the sample to be predicted is input into the first branch of a preset multi-branch network to obtain the corresponding initial prediction result and uncertainty.

[0066] Based on the uncertainty, the output branch corresponding to the sample is obtained from the preset distribution scheme of the multi-branch network;

[0067] According to the preset model partitioning scheme of the multi-branch network, the final prediction result of the sample is obtained by using the output branch; the model partitioning scheme includes the hierarchical calculation allocation result of each branch of the multi-branch network on the IoT device and the corresponding server.

[0068] In one specific embodiment of this disclosure, the multi-branch network structure is as follows: Figure 1 As shown, the backbone of this multi-branch network comprises five sequentially connected layers. Nodes v1, v2, v3, v4, and v5 represent the layers of the backbone, while nodes b1, b2, b3, and b4 represent branches extending from layers v1, v2, v3, and v4, respectively. The solid lines represent the data flow. Node (v1, b1) constitutes the first branch of the multi-branch network, which is the foundation of the network. The remaining branches form the remaining portion of the network, including: the second branch formed by nodes (v1, v2, b2); the third branch formed by nodes (v1, v2, v3, b3); the fourth branch formed by nodes (v1, v2, v3, v4, b4); and the fifth branch formed by nodes (v1, v2, v3, v4, v5).

[0069] This disclosure presents a specific embodiment of a multi-branch network collaborative reasoning method for the Internet of Things, the overall process of which is as follows: Figure 2 As shown, it includes the following steps:

[0070] 1) Input the sample to be predicted into the first branch of the preset multi-branch network to obtain the corresponding initial prediction result and uncertainty. The first branch is deployed on an IoT device.

[0071] In some specific embodiments of this disclosure, the sample to be predicted includes: images or video frames used for tasks such as image classification and object detection.

[0072] In some specific embodiments of this disclosure, the initial prediction result includes the probability of the sample corresponding to each prediction category output by the first branch, and the uncertainty of the sample is obtained by subtracting the second maximum value of the probability from the maximum value of the probability.

[0073] 2) Determine the uncertainty and obtain the output branch of the sample to be predicted in the multi-branch network in the preset multi-branch network distribution scheme.

[0074] The distribution scheme of the multi-branch network determines the output branch corresponding to each uncertainty level. The output branch can be the first branch, that is, the remaining branches are no longer used. In a specific implementation of this disclosure, if the output branch is the first branch, the prediction result of branch b1 is directly selected as the final classification result of the input sample.

[0075] The distribution scheme of the multi-branch network is determined after the multi-branch network has been trained. In a specific embodiment of this disclosure, the specific steps are as follows:

[0076] 2-1) Using the multi-branch network, calculate the uncertainty of each sample in the preset evaluation set and determine the uncertainty distribution of the evaluation set.

[0077] The evaluation set contains multiple samples and their classification results.

[0078] Specifically, the initial uncertainty distribution of all samples in the evaluation set is calculated using the initial prediction results obtained by passing the evaluation set through the first branch of the multi-branch network (i.e., the branch closest to the input of the multi-branch network, which is branch b1 in this embodiment).

[0079] In one specific embodiment of this disclosure, for any sample in the evaluation set, it is assumed that the output of branch b1 is y = (y1, y2, ..., y...). 10 ), where y i This represents the probability that the predicted sample belongs to class i. The final output will then show the probability for each class. for:

[0080]

[0081] Wherein, T is a hyperparameter that can be determined heuristically to make the uncertainty distribution close to a uniform distribution. In a specific embodiment of this disclosure, T = 1.5.

[0082] The uncertainty of this sample is determined by the final output. OK, the expression is as follows:

[0083]

[0084] Right now maximum value minus The second largest value The difference is the uncertainty of the sample.

[0085] 2-2) Classify the level of uncertainty.

[0086] The uncertainty distribution obtained from step 2-1) is used to divide the evaluation set samples into M equal parts based on the uncertainty of each sample in the evaluation set, so as to determine the M levels of uncertainty. M is an adjustable parameter. The larger M is, the finer the granularity of uncertainty division, but the calculation will be more complicated and the required number of evaluation set samples will be higher.

[0087] In one specific embodiment of this disclosure, M=10, and the classification boundaries for different levels are [0.000, 0.058, 0.130, 0.223, 0.343, 0.480, 0.625, 0.777, 0.894, 0.966, 1]. Samples with uncertainty close to 0 are considered difficult, and samples with uncertainty close to 1 are considered easy. The evaluation set is then divided into 10 sample sets according to the classification boundaries. The accuracy and inference latency of each branch for sample sets with different uncertainty levels are tested. Accuracy is the average prediction accuracy of each sample set output by each branch, and inference latency is the average execution time of each sample set output by each branch.

[0088] 2-3) Initialize the distribution scheme.

[0089] Based on the uncertainty level classification results, all samples in the evaluation set with uncertainty levels are initially output from the first branch. In a specific embodiment of this disclosure, the initial distribution scheme is [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], that is, the evaluation set samples divided into 10 uncertainty levels all select branch b1 to output the corresponding image prediction results.

[0090] Let the current candidate branch corresponding to each uncertainty level be the next branch of the current output branch. In a specific embodiment of this disclosure, the initial candidate branch for each uncertainty level is branch b2, and the initial set of candidate branches is [2, 2, 2, 2, 2, 2, 2, 2, 2].

[0091] For each level of uncertainty, calculate the speedup corresponding to the current candidate branch. The speedup is the ratio of the increase in accuracy brought by using the current candidate branch compared to the current output branch to the increase in inference time brought by using the current candidate branch compared to the current output branch, expressed as:

[0092]

[0093] Where, Δ acc =NewBranch acc OldBranch accNewBranch represents the increase in prediction accuracy resulting from replacing the current output branch with the current candidate branch; acc OldBranch represents the prediction accuracy corresponding to the current candidate branch. acc The prediction precision corresponding to the current output branch;

[0094] Δ time =NewBranch time OldBranch time This indicates the increase in inference time caused by replacing the current output branch with the current candidate branch; NewBranch time OldBranch represents the inference time corresponding to the current candidate branch. time This represents the inference time corresponding to the current output branch.

[0095] 2-4) Update the distribution scheme.

[0096] Select the uncertainty level corresponding to the maximum speedup ratio among all current candidate branches, and use the current candidate branch of that uncertainty level as the new current output branch for that uncertainty level to obtain the updated current distribution scheme; then update the candidate branch of that uncertainty level as the next branch of the current output branch to obtain the updated candidate branch set; use the updated current distribution scheme and candidate branch set to calculate the updated speedup ratio for each uncertainty level.

[0097] In one specific embodiment of this disclosure, the candidate branch with the largest speedup ratio after the first update corresponds to the first uncertainty level. The current distribution scheme is then updated to [2, 1, 1, 1, 1, 1, 1, 1, 1, 1], and the candidate branch set is updated to [3, 2, 2, 2, 2, 2, 2, 2, 2]. The speedup ratio of the candidate branch corresponding to the first uncertainty level is updated as the ratio of the accuracy improvement to the increase in inference latency brought by the sample at the first uncertainty level in branch 3 compared to branch 2.

[0098] 2-5) The DSGA algorithm (distribution scheme generation algorithm) is used to obtain the final output branch corresponding to each uncertainty level, which constitutes the final distribution scheme of the multi-branch network.

[0099] It should be noted that the core concept of the DSGA algorithm proposed in this embodiment is to greedily select the candidate branch with the largest speedup ratio each time the current distribution scheme is updated, until all current candidate branches in the candidate branch set no longer bring accuracy improvement or the current distribution scheme has met the target accuracy.

[0100] It's worth noting that multi-branch networks accelerate the inference process by inserting auxiliary classifiers into shallower layers of the model, improving the experience of running DNN models on IoT devices. Combining model partitioning with multi-branch networks allows for a trade-off between communication and computation; however, the unique characteristics of multi-branch networks make model partitioning more challenging than traditional model partitioning. In multi-branch networks, the execution of a sample depends on its uncertainty; simple samples can exit at the first branch, while difficult samples require exiting at deeper branches. During inference in a multi-branch network, the first branch calculates the uncertainty and initial prediction information of the input sample. The subsequent output branches are then determined by the multi-branch network's distribution scheme. For example, a sample might be output at the third branch or exit at the fifth branch. Deeper branches generally have higher accuracy than shallower branches. Adjusting the multi-branch network's distribution scheme can yield multi-branch networks with varying average inference latency and accuracy.

[0101] Furthermore, this embodiment can also dynamically adjust the distribution scheme of the multi-branch network according to the target requirements (accuracy requirements or throughput requirements), the current load level of IoT devices and servers, and the current network bandwidth, that is, by adjusting the proportion of output samples in different branches to the total samples to meet different target requirements.

[0102] 3) Based on the distribution scheme of the multi-branch network, the model partitioning scheme of the multi-branch network is used to obtain the final prediction result of the sample to be predicted using the output branch.

[0103] In a specific embodiment of this disclosure, the specific steps are as follows:

[0104] 3-1) Obtain the model partitioning scheme of the multi-branch network, wherein the model partitioning scheme includes the hierarchical processing allocation results of each branch of the multi-branch network on IoT devices and edge servers.

[0105] 3-2) Based on the corresponding output branch of the sample to be predicted, the final prediction result of the sample to be predicted is obtained using the model partitioning scheme. Specifically:

[0106] 3-2-1) If the output branch corresponding to the sample is the first branch, then the sample does not need to be processed further. The initial prediction result obtained in step 1) is taken as the final prediction result of the sample and directly output by the IoT device.

[0107] 3-2-2) If the output branch corresponding to the sample is not the first branch, the prediction result of the first branch is no longer used. According to the model partitioning scheme, the prediction result of the sample is obtained from the output branch corresponding to the sample. In subsequent processing, the calculation result of node v1 in the first branch in step 1) can be directly used for subsequent processing to improve computational efficiency.

[0108] In one specific embodiment of this disclosure, the processing method is as follows:

[0109] 3-2-2-1) If, in the model partitioning scheme, all levels of the output branch corresponding to the sample are assigned to IoT devices for processing, then the final prediction result of the sample can be directly calculated on the IoT device using the corresponding branch.

[0110] In a specific embodiment of this disclosure, for example, if the model split point corresponding to branch 2 is after the last layer of the branch, that is, all layers of the branch are assigned to the IoT device, then on the IoT device, the output of node v1 is used, and nodes v2 and b2 continue to infer to obtain the final prediction result of the input image.

[0111] 3-2-2-2) If, in the model partitioning scheme, all levels in the output branch corresponding to the sample are partitioned to the edge server, then the edge server uses the corresponding branch to calculate the final prediction result of the sample, wherein the input of the edge server is the output result of the backbone part of the multi-branch network contained in the first branch.

[0112] In a specific embodiment of this disclosure, for example, the model split point corresponding to branch 5 is after the last layer of the branch, that is, all layers of the branch are assigned to the edge server, and all unprocessed layers need to be completed by the edge server to complete the inference task. (wherein, the result of v1 can be reused, so v1 does not need to be executed again on the server). Therefore, the output of node v1 is sent to the edge server via Wi-Fi, and nodes (v2, v3, v4, v5) continue the inference. The final prediction result of the input image is returned to the IoT device via Wi-Fi.

[0113] 3-2-2-3) If, in the model partitioning scheme, the output branch corresponding to the sample is partitioned in both the IoT device and the edge server, then the intermediate result is first obtained through the part of the branch partitioned in the IoT device and sent to the edge server. Then, the final prediction result of the sample is obtained through the part of the branch partitioned in the edge server and returned to the IoT device. The input of the part of the branch partitioned in the IoT device is the output result of the backbone part of the multi-branch network contained in the first branch.

[0114] In one specific embodiment of this disclosure, for example, the model split point corresponding to the fourth branch is between nodes v2 and v3. Therefore, node v2, deployed on the IoT device, first processes the output of node v1, and then sends the output of node v2 to the edge server via Wi-Fi. Nodes (v3, v4, b4) continue inference to obtain the final prediction result of the input image, and then return the final prediction result to the IoT device via Wi-Fi.

[0115] Furthermore, the implementation method of the model partitioning scheme for the multi-branch network is as follows:

[0116] In one specific embodiment of this disclosure, considering the fluctuations in network bandwidth and the load fluctuations of IoT devices and edge servers during collaborative inference, an on-demand adjustment algorithm for the model partitioning scheme is proposed. The overall process is as follows: Figure 3 As shown, the on-demand adjustment algorithm runs once at fixed intervals or when network fluctuations are detected. The specific steps are as follows:

[0117] 3-1-1) Update network bandwidth using the EMA (Exponential Moving Average) method, as shown in the following expression:

[0118] Band=(1-α)*Band+α*B_runtime

[0119] Where Band is the network bandwidth used to calculate network transmission time, and B_runtime is the real-time network bandwidth; a is a hyperparameter set by the EMA method, 0≤a≤1; in a specific embodiment of this disclosure, a=0.1.

[0120] 3-1-2) Determine the optimization objective for partitioning the multi-branch network model:

[0121]

[0122] In this embodiment, the optimization objective considers the optimal model split point for each branch separately, eliminating the influence of the branch selection probability. Here, T represents the average inference time of the multi-branch network. p represents the reasoning time for the m-th branch. m This represents the probability that the m-th branch is selected.

[0123] 3-1-3) Determine the model partitioning point for each branch to obtain the model partitioning scheme for the multi-branch network.

[0124] In this embodiment, the method for determining the model split point for any branch is as follows:

[0125] 3-1-3-1) Construct the directed acyclic graph corresponding to this branch.

[0126] It should be noted that all branches in this embodiment can be regarded as a separate DNN model; therefore, the model partitioning method in this embodiment is also applicable to traditional DNN models. In a specific embodiment of this disclosure, using... Figure 4 The multi-branch network shown is used as an example to illustrate the DNN model partitioning method.

[0127] Treating any branch sub-network as an independent DNNN model, we construct the corresponding DAG (Directed Acyclic Graph) G = (V, E). In this embodiment, V = (a1, a2, a3, a4, a5) represents the set of nodes in graph G, where each node is a layer in the corresponding DNN model. E represents the set of edges in graph G, where each edge is a link in the corresponding DNN model, and each edge reflects the direction of data flow. Any link l ij =(a i ,a j ) represents node a i The output is node a j The input. And d i Representative node a i The output data size, where Band represents the network bandwidth. That is the link l ij =(a i ,a j Network transmission time.

[0128] Model partitioning involves dividing the nodes in graph G into two disjoint subsets V. device and V edge The sum of the two is V. Where V device V represents a subset of nodes that execute on IoT devices. edge L represents the subset of nodes executed on the edge server, while L represents the set of links between the two subsets, i.e., the model split points. Figure 4 (The part marked with a dashed line). Execute subset V on the device. device Total execution time For a i The execution time of the layer on IoT devices. Execution of a subset V on the edge server. edge Total execution time For a i The execution time of the layer on the edge server. The sum of the total data transmission time for model partitioning point L.

[0129] The total delay of collaborative inference is the sum of the three factors. Therefore, the optimization objective for any branch subnetwork is:

[0130] 3-1-3-2) Construct a new graph based on the original graph G

[0131] In this embodiment, the network partitioning problem is transformed into a minimum ST-cut problem of an equivalent DAG graph. A new graph is constructed based on the original graph G. Each edge in the new graph corresponds to a delay in step 3-1-3-1), which includes the data transmission time in step 3-1-3-1), the execution time on the IoT device, and the execution time on the edge server.

[0132] In one specific embodiment of this disclosure, the updated directed acyclic graph is as follows: Figure 5 As shown, two virtual nodes, d and e, are added to Figure G, where d represents an IoT device and is the source node; e represents an edge server and is the destination node. The minimum st cut is to find a split point between node d and node e. Figure 5 (A dashed line is used to minimize the sum of the weights of the links connected to this dashed line). The links between nodes and virtual nodes in the original graph G represent the execution time of that layer on IoT devices and edge servers. It's worth noting that the line connecting node e represents the execution time of the corresponding layer in G on IoT devices; for example, link l... 1e The weight of (a1,e) is the execution time of node a1 on the IoT device.

[0133] However, some nodes have multiple successor nodes. For example, node a1 has two nodes, a2 and a3. This can lead to the problem of communication latency being calculated repeatedly. Figure 5 In the partitioning method described above, the output data of node a1 only needs to be transmitted once, and the communication delay should only be calculated once. Therefore, this disclosure updates the weight of the corresponding link to one-tenth of the out-degree of the communication delay of the forward node. For example, if the out-degree of node a1 is 2, the link l with node a1 as the forward node... 12 =(a1,a2) and l 13 The weights of (a1, a3) are This update is based on the reality that links sharing the same forward node will all be connected to the dotted line at the split point, preventing partial connections. Assuming nodes a1 and a3 execute on the device, the output data of a1 still needs to be transmitted to the server. Therefore, link l... 12 The weights corresponding to (a1, a2) would be mismatched, but this is impossible. This is because placing node a3 on the server would result in faster inference time. The server processes nodes significantly faster than IoT devices, meaning that once a node's data is sent to the server, all its successor nodes can be executed on the server, resulting in shorter inference time.

[0134] 3-1-3-3) Request a new diagram The minimum cut between the source node d and the destination node e is the cut, which corresponds to the model splitting point. Using the cut as the boundary, in the new graph... The DNN model nodes on the same side as the source node are assigned to perform computations on the IoT device, while the DNN model nodes on the same side as the destination node are assigned to perform computations on the server.

[0135] It's important to note that model partitioning involves dividing the model into two parts: one deployed on IoT devices and the other on a server. In this partitioning scheme, the inference time consists of computation time and communication time. Communication time is related to the size of the transmitted data and network bandwidth. Generally, the output data of intermediate layers in a DNN model is less than the original data, meaning the communication latency from sending data from intermediate layers is less than the latency from sending the original data. Another advantage of having the device-executed part of the layer is reduced server load, allowing the server to serve more IoT devices. Model partitioning also addresses privacy concerns. Directly sending raw data easily leads to privacy leaks, while intermediate data, processed by the model, has already been encrypted, reducing the possibility of information leakage during network transmission.

[0136] After finding the model partitioning points for all branches, the distribution scheme of the multi-branch network is obtained.

[0137] Furthermore, this embodiment also includes:

[0138] 3-1-4) Update the distribution scheme of the multi-branch network according to the target requirements.

[0139] The collaborative inference time of each branch in the multi-branch network is estimated, and then the distribution scheme of the multi-branch network is updated. Depending on the actual application scenario, there are two target requirements: throughput and accuracy. The accuracy requirement demands that the accuracy of the multi-branch network be no less than the target requirement, while the throughput requirement requires that the multi-branch network can process a certain number of samples within a specified time. Deeper branches in the multi-branch network have longer inference times than shallower branches, but correspondingly higher accuracy.

[0140] 3-1-4-1) If the current target requirement is accuracy requirement, but the accuracy of the current distribution scheme is lower than the target accuracy requirement, then update the distribution scheme of the multi-branch network to increase the proportion of samples output in the deep branches to all samples.

[0141] 3-1-4-2) If the current target requirement is accuracy, but the accuracy of the current distribution scheme is higher than the target requirement, then update the distribution scheme of the multi-branch network to increase the proportion of samples output from shallow branches. However, it is necessary to ensure that the accuracy requirement is met in order to provide a faster inference scheme.

[0142] 3-1-4-3) If the current target requirement is throughput requirement, but the average inference time of the current distribution scheme is greater than the target requirement, then update the distribution scheme of the multi-branch network to increase the proportion of samples output in shallow branches to all samples.

[0143] 3-1-4-4) If the current target requirement is throughput, but the average inference time of the current distribution scheme is less than the target requirement, then update the distribution scheme of the multi-branch network to increase the proportion of samples output from deeper branches. However, it is necessary to ensure that the throughput requirement is met in order to provide a faster inference scheme.

[0144] To implement the above embodiments, a second aspect of this disclosure proposes a multi-branch network collaborative reasoning system for the Internet of Things, comprising:

[0145] The initial prediction module, deployed on the IoT device, is used to input the sample to be predicted into the first branch of a preset multi-branch network to obtain the corresponding initial prediction result and uncertainty.

[0146] The output branch determination module is used to obtain the output branch corresponding to the sample from the preset distribution scheme of the multi-branch network based on the uncertainty.

[0147] The collaborative reasoning module is used to obtain the final prediction result of the sample by utilizing the output branch according to the preset model partitioning scheme of the multi-branch network; the model partitioning scheme includes the hierarchical calculation allocation results of each branch of the multi-branch network on the IoT device and the corresponding server.

[0148] To implement the above embodiments, a third aspect of this disclosure provides an electronic device, comprising:

[0149] At least one processor; and a memory communicatively connected to said at least one processor;

[0150] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-described multi-branch network collaborative reasoning method for the Internet of Things.

[0151] To implement the above embodiments, a fourth aspect of this disclosure provides a computer-readable storage medium storing computer instructions for causing the computer to execute the above-described multi-branch network collaborative reasoning method for the Internet of Things.

[0152] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0153] The aforementioned computer-readable medium may be included in the aforementioned electronic device; alternatively, it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform a multi-branch network collaborative reasoning method for the Internet of Things according to the above embodiments.

[0154] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0156] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0157] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0158] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0159] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0160] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0161] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0162] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for collaborative reasoning in multi-branch networks for the Internet of Things, characterized in that, include: On an IoT device, the sample to be predicted is input into the first branch of a preset multi-branch network to obtain the corresponding initial prediction result and uncertainty. Based on the uncertainty, the output branch corresponding to the sample is obtained from the preset distribution scheme of the multi-branch network; According to the preset model partitioning scheme of the multi-branch network, the final prediction result of the sample is obtained by using the output branch; the model partitioning scheme includes the hierarchical calculation allocation result of each branch of the multi-branch network on the IoT device and the corresponding server; The step of obtaining the final prediction result of the sample using the output branch according to the preset model partitioning scheme of the multi-branch network includes: 1) If the output branch corresponding to the sample is the first branch, then the initial prediction result is taken as the final prediction result of the sample; 2) If the output branch corresponding to the sample is not the first branch, the final prediction result is obtained as follows: 2-1) If the levels of the output branches corresponding to the samples are all assigned to the IoT device, then the final prediction result is calculated on the IoT device using the output branches; 2-2) If all levels in the output branch corresponding to the sample are assigned to the server, the final prediction result is calculated on the server using the output branch and returned to the IoT device; 2-3) If the level of the output branch corresponding to the sample is divided by the IoT device and the server respectively, then the intermediate result is first obtained through the level divided by the branch in the IoT device and sent to the server. Then the intermediate result is obtained through the level divided by the branch in the server to obtain the final prediction result and returned to the IoT device.

2. The method according to claim 1, characterized in that, The method further includes: The initial prediction result includes the probability of each prediction category output by the sample via the first branch. The uncertainty of the sample is obtained by subtracting the second maximum value of the probability from the maximum value of the probability.

3. The method according to claim 1, characterized in that, The model partitioning scheme consists of model partitioning points for each branch of the multi-branch network, and the model partitioning points minimize the inference time of the branch.

4. The method according to claim 1, characterized in that, The method further includes: If the output branch corresponding to the sample is not the first branch, then the output result of the backbone of the multi-branch network contained in the first branch is used to continue the calculation on the output branch to obtain the final prediction result.

5. The method according to claim 2, characterized in that, The method for determining the distribution scheme of the multi-branch network is as follows: 1) Using the multi-branch network, calculate the uncertainty of each sample in the preset evaluation set, and determine the uncertainty distribution of the evaluation set; the evaluation set includes multiple samples and corresponding classification results; 2) Based on the uncertainty distribution of the evaluation set, divide all samples in the evaluation set into M groups on an average basis to obtain the uncertainty level classification result, where M is the preset total number of uncertainty levels; 3) Determine the initial distribution scheme, wherein the current output branch corresponding to each sample of uncertainty level in the evaluation set is the first branch of the multi-branch network; 4) Let the current candidate branch corresponding to each uncertainty level be the next branch of the current output branch; 5) Using the evaluation set, for each level of uncertainty, calculate the speedup ratio corresponding to the current candidate branch, whereby the speedup ratio is the ratio of the increase in prediction accuracy brought about by using the current candidate branch compared to the current output branch to the increase in inference time brought about by using the current candidate branch compared to the current output branch; 6) Select the uncertainty level corresponding to the maximum speedup ratio among all current candidate branches, and take the current candidate branch of the uncertainty level as the new current output branch of the uncertainty level to obtain the updated current distribution scheme; Update the current candidate branch at the uncertainty level to obtain the updated candidate branch set; 7) Repeat steps 5) to 6) until all current candidate branches in the candidate branch set meet the set target requirements, then the current distribution scheme is taken as the final distribution scheme of the multi-branch network.

6. The method according to claim 3, characterized in that, The method for determining the model partitioning scheme is as follows: 1) Update network bandwidth using the exponential moving average method, as shown in the following expression: Band=(1-α)*Band+α*B_runtime Where Band is the network bandwidth, B_runtime is the real-time network bandwidth; a is a hyperparameter, 0≤a≤1; 2) Determine the optimization objective for the multi-branch network model partitioning: Where T represents the average inference time of the multi-branch network. p represents the reasoning time for the m-th branch. m This represents the probability that the m-th branch is selected. 3) Determine the model partitioning points for each branch to obtain the model partitioning scheme for the multi-branch network; The method for determining the model split point for any branch is as follows: 3-1) Construct the directed acyclic graph corresponding to this branch; Treat any branch as a DNN model and construct the corresponding directed acyclic graph G = (V, E); where V represents the set of nodes in graph G, each node is a layer in the DNN model corresponding to graph G; and E represents the set of edges in graph G, each edge is a link in the DNN model corresponding to graph G. Let link l ij =(a i ,a j ) represents node a i The output is node a j The input, d i Representative node a i If the output data size is such that the link l ij =(a i ,a j Network transmission time Divide set V into two disjoint subsets Vi device and V edge V device V represents a subset of nodes that execute on IoT devices. edge Let L represent the subset of nodes executed on the server; let L represent the set of links between the two subsets, i.e., the model splitting points. Then the total latency of collaborative inference is the latency of executing subset V on the device. device Total execution time And executing subset V on the server edge Total execution time The sum of, among which, For node a i The execution time of the corresponding layer on the IoT device. For node a i The execution time of the corresponding layer on the server; the sum of the total data transmitted to the model partitioning point L. but: 3-2) Add two virtual nodes d and e to graph G; where d represents the IoT device and is the source node; and e represents the edge server node and is the destination node. Add new edges to graph G such that each edge corresponds to a delay, which includes network transmission time, execution time on the IoT device, and execution time on the edge server. After construction, a new directed acyclic graph is obtained, denoted as . 3-3) Obtain the graph The minimum cut between the source node d and the destination node e is used as the model split point for this branch; with the cut as the boundary, in the graph... Nodes on the same side as the source node are assigned to perform computations on the IoT device, while nodes on the same side as the destination node are assigned to perform computations on the server.

7. A multi-branch network collaborative reasoning system for the Internet of Things based on the method described in claim 1, characterized in that, include: The initial prediction module, deployed on the IoT device, is used to input the sample to be predicted into the first branch of a preset multi-branch network to obtain the corresponding initial prediction result and uncertainty. The output branch determination module is used to obtain the output branch corresponding to the sample from the preset distribution scheme of the multi-branch network based on the uncertainty. The collaborative reasoning module is used to obtain the final prediction result of the sample by utilizing the output branch according to the preset model partitioning scheme of the multi-branch network; the model partitioning scheme includes the hierarchical calculation allocation results of each branch of the multi-branch network on the IoT device and the corresponding server.

8. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-6.

Images