A model partitioning method based on cloud-edge collaborative inference

Abstract

This invention relates to a model partitioning method based on cloud-edge collaborative inference. By setting a split point, the simplified model is divided into two parts. The mobile terminal executes the first part, transmitting intermediate computational data to the edge cloud, which then completes the remaining part and returns the final result. Specifically: a simplified model library is constructed based on different compression and simplification algorithms for the original task model. Dual backups are implemented on the edge cloud and the mobile terminal. When a task arrives, the task objective is obtained, including latency, accuracy, or model quality. The communication environment is assessed, including the computing power of the edge cloud and the local computing power that can provide computational services. The parameters of each simplified model in the simplified model library are traversed: the number of network layers and the size of each layer. The performance of the simplified model is estimated according to the defined formulas: inference latency, inference accuracy, and partitioning quality. A suitable simplified model and partitioning layer are selected based on the specific task objective.

Description

Technical Field

[0001] This invention belongs to the field of neural network technology and relates to a model partitioning method based on cloud-edge collaborative reasoning. Background Technology

[0002] Deep learning (DL) has achieved significant breakthroughs in a wide range of fields, including computer vision, speech recognition, stock prediction, and natural language processing. These breakthroughs have led to the rapid rise and widespread adoption of a range of intelligent applications, such as smart personal assistants, personalized shopping recommendations, video surveillance, and smart home appliances. However, DL models are typically very large. For example, the VGG16 model trained on the ImageNet dataset exceeds 500MB in size, and real-time reliable inference using deep neural networks (DNNs) requires substantial computing resources. Compared to the edge and cloud, mobile devices generally have weaker computing power, limited storage capacity, and energy consumption. This places more stringent demands on deploying deep learning models on resource-constrained edge devices, making DNN inference challenging under reasonable latency and energy consumption. To address this issue, Mobile Edge Computing (MEC) technology has been proposed to assist in performing some computations on the edge and cloud. This involves model partitioning techniques, which improve end-to-end latency performance through device-edge collaborative inference. Dividing the DNN model inference task between mobile devices and the edge cloud, with the mobile device computing the intermediate layers of the DNN model and transmitting the output features of the intermediate layers to the edge cloud to complete the remaining computations of the DNN, not only balances the transmission and computational workload between mobile devices and servers but also optimizes end-to-end latency of DNN inference and the energy consumption of mobile devices. The selection of the optimal partition point depends not only on various system factors, such as wireless channel conditions, wireless communication technology, and the computing power of mobile devices and the edge cloud, but also on the DNN model itself. Due to the increased data volume after convolutional layers, the output features of intermediate layers can be much larger than the original data, resulting in higher transmission latency and energy consumption when uploading intermediate data compared to uploading the original data. Therefore, in many cases, pre-trained, fixed DNN models cannot directly benefit from partitioning. To address this issue, some studies have proposed early exit mechanisms and feature encoding techniques to reduce the amount of intermediate data transmitted. The main drawback of feature encoding is that it introduces additional computational latency, while early exit mechanisms have low accuracy under severe latency constraints.

[0003] Currently, DNNs for handling complex tasks typically have a large number of layers to achieve high recognition accuracy, but each device is limited by energy and computing power. Therefore, each edge device can partially execute a DNN, distributing the remaining DNN computations to other computing nodes in the network, such as... Figure 1 As shown. Model partitioning can be divided into two types: partitioning between cloud servers and devices, and partitioning between devices.

[0004] Neurosurgeon is a pioneering work in model partitioning technology. When partitioning DNN models between devices and servers, the key challenge is to find a suitable partitioning point to obtain the best model inference performance. The authors consider both latency and energy efficiency and propose a regression-based method to estimate the latency of each layer in the DNN model and return an optimal partitioning point so that the model inference meets the latency or energy requirements.

[0005] Subsequently, Ko et al. combined model partitioning with lossy feature encoding, compressing the intermediate data after model partitioning using lossy feature encoding before transmission. JALAD also utilizes model partitioning and lossy feature encoding in conjunction, defining model partitioning as an integer linear programming problem to minimize model inference latency while ensuring accuracy. For DNNs with directed acyclic graphs (DAGs) rather than linked features in their structure, model partitioning to optimize and minimize latency is often proven to be NP-hard. To address this problem, Hu et al. proposed an approximate algorithm based on graph minimum partitioning, providing performance guarantees in the worst case. All of the above frameworks assume that the server has the DNN model for the edge intelligent application. IONN proposes an incremental offloading technique for edge intelligent applications. IONN partitions the DNN layers and uploads them incrementally, allowing mobile devices and the edge cloud to perform collaborative DNN model inference. Compared to uploading the entire model, IONN significantly improves query performance and energy consumption during the DNN model upload process.

[0006] Another type of model partitioning is inter-device partitioning. As a pioneer in inter-device model partitioning, MoDNN introduces WiFi Direct technology, utilizing multiple licensed WiFi-enabled mobile devices to build a micro-scale computing cluster in a WLAN for partitioning DNN model inference. The mobile device responsible for DNN inference acts as the control node of the cluster, while other devices serve as worker nodes. To accelerate the execution of DNN layers, the paper proposes two partitioning schemes, proving that a latency-proven optimal number of worker nodes between 2 and 4 results in a 2.17–4.28x speedup for DNN model inference. In the subsequent work, MeDNN, a two-dimensional partitioning method based on a greedy algorithm adaptively partitions the DNN model across multiple mobile devices and uses structured sparse pruning to compress the DNN model, improving DNN model inference by 1.86–2.44x and saving 26.5% of additional computation time and 14.2% of additional communication time. In MoDNN and MeDNN, the DNN layers are horizontally partitioned, while DeepThings employs a convolutional layer fusion block partitioning technique to vertically partition the DNN layers, reducing memory usage.

[0007] DeepX divides a DNN model into several sub-models and deploys them on different local processors. Through runtime layer compression and deep architecture decomposition techniques, the compressed layers are executed by specific local processors. LEO is a novel sensor algorithm scheduler that maximizes performance for multiple consecutive mobile sensor applications by dividing the execution of sensor algorithms and distributing tasks across CPUs, coprocessors, GPUs, and the cloud.

[0008] However, the model partitioning methods in related technologies are poorly adaptable to new architectures. Currently, running computationally intensive deep neural networks (DNNs) on resource-constrained mobile devices allows mobile clients to query DNNs from a central cloud server, where the corresponding DNN models are pre-installed. Unfortunately, this centralized, cloud-based DNN offloading is not suitable for emerging decentralized cloud infrastructures (e.g., cloudlets, edge / fog servers), where clients may send computation requests to any nearby server located at the network edge.

[0009] Furthermore, the model partitioning methods used in related technologies are inefficient. Using a generic edge cloud to execute DNNs requires the client to first upload its fixed DNN model to the server it wants to connect to. However, due to varying channel conditions and different MEC server configurations, uploading a generic model inevitably reduces the effectiveness of cloud-edge collaboration. Summary of the Invention

[0010] To address the aforementioned problems in existing technologies, this invention provides a model partitioning method based on cloud-edge collaborative reasoning. The technical problem to be solved by this invention is achieved through the following technical solution:

[0011] This invention provides a model partitioning method based on cloud-edge collaborative reasoning, comprising the following steps:

[0012] S0. Construct a simplified model library;

[0013] S1. Obtain the task objective, which includes: latency, accuracy, or model quality;

[0014] S2. Sensing the communication environment and edge cloud computing power, traversing each simplified model, and estimating the inference latency, inference accuracy, and partitioning quality of each simplified model.

[0015] The expression for the inference latency of the simplified model is:

[0016]

[0017] Where latency is the inference latency of the simplified model. The runtime latency from the first layer to the j-th layer of the simplified model in the mobile terminal; This refers to the runtime latency after the j-th layer of the simplified model; This represents the transmission time required for the j-th layer output data of the simplified model to travel on the wireless network; the The data is obtained based on the output data volume of the j-th layer of the simplified model and the wireless channel transmission rate. Based on the factors related to runtime of each layer of the simplified model, the amount of output data of each layer, and the CPU frequency of the mobile terminal; and the... The data is obtained based on the factors related to runtime of each layer in the simplified model, the amount of output data of each layer, and the CPU frequency of the edge cloud.

[0018] S3. When the task objective is latency, select the simplified model i with the smallest inference latency and the partitioning layer j from the simplified models whose inference accuracy meets the latency requirement; or, when the task objective is accuracy, obtain the simplified model whose inference latency meets the minimum tolerance latency requirement; among the simplified models that meet the minimum tolerance latency, obtain the simplified model i with the highest inference accuracy and the partitioning layer j with the lowest inference latency corresponding to the simplified model with the highest inference accuracy; or, when the task objective is the partitioning quality of the model, obtain the simplified model i and the partitioning layer j with the highest partitioning quality of the simplified model, where i and j are positive integers.

[0019] In one embodiment of the present invention, the runtime delay from the first layer to the j-th layer of the simplified model in the mobile terminal The expression is:

[0020]

[0021] Where, φ l D represents the set of factors related to runtime in the l-th layer of the simplified model. l This represents the output data volume of the l-th layer of the simplified model, where C... mobile This refers to the CPU frequency of the mobile terminal.

[0022] In one embodiment of the present invention, the runtime latency after the j-th layer of the simplified model The expression is:

[0023]

[0024] Where, φ l D represents the set of factors related to runtime in the l-th layer of the simplified model. l This represents the output data volume of the l-th layer of the simplified model, where C... server This refers to the CPU frequency at the edge cloud.

[0025] In one embodiment of the present invention, the transmission time required for the j-th layer output data of the simplified model in the wireless network. The expression is:

[0026]

[0027] Wherein, the D j R represents the output data volume of the j-th layer of the simplified model; R is the upload rate.

[0028] In one embodiment of the present invention, the partitioning quality (QoMP) of the simplified model is... i j The expression is:

[0029]

[0030] Wherein, the QoMP i j The partitioning quality of the j-th layer of the simplified model i; the latency is the inference latency of the simplified model; the QoM i The quality of simplified model i; where η is QoM i The weight of latency; μ is the weight of latency.

[0031] In one embodiment of the present invention, the expression for the quality of the simplified model i is:

[0032]

[0033] Wherein, Acc represents the accuracy of the original model corresponding to the simplified model i, and acc i The ratio is the accuracy of the simplified model obtained by pruning the i-th simplified model. i The compression ratio of the simplified model obtained by pruning the i-th simplified model, where α is... The weight, where β is The weight, the ratio; min It represents the lowest compression ratio among all simplified models.

[0034] In one embodiment of the present invention, the compression ratio of the simplified model obtained by pruning the i-th simplified model is... i The expression is:

[0035]

[0036] Wherein, sum is the total number of convolutional kernels in the original model corresponding to the simplified model i, and rem i The number of convolution kernels in the simplified model i.

[0037] In one embodiment of the present invention, the method further includes:

[0038] For the QoMP i Perform normalization processing to make the QoMP i The values ​​of are all within the range of (0, 1]. The beneficial effects of this invention are:

[0039] The cloud-edge collaborative inference-based model partitioning method provided by this invention has strong adaptability. Unlike fixed models, this invention proposes to prepare a set of simplified models in advance (existing model simplification algorithms can achieve compression rates of tens of times, so multiple simplified models can be prepared without increasing storage consumption), and adaptively select the best model based on channel conditions, MEC server configuration, etc., and cascade the model partitioning function. Furthermore, the cloud-edge collaborative inference-based model partitioning method provided by this invention has a wide service coverage. Because there are sufficient alternative models, a trade-off can be made between accuracy, latency, and energy according to service requirements, and the selection of models and partitioning points can meet different target needs.

[0040] Other aspects and features of the invention will become apparent from the following detailed description with reference to the accompanying drawings. However, it should be understood that the drawings are for illustrative purposes only and not as a limitation of the scope of the invention, as reference should be made to the appended claims. It should also be understood that, unless otherwise indicated, the drawings are not necessarily drawn to scale; they are merely intended to conceptually illustrate the structures and processes described herein. Attached Figure Description

[0041] Figure 1 A schematic diagram of the model segmentation architecture provided in an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the adaptive model segmentation collaborative reasoning architecture provided in an embodiment of the present invention;

[0043] Figure 3 This is a flowchart illustrating a model partitioning method based on cloud-edge collaborative reasoning, provided in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0045] For the technologies involved, distributed cloud infrastructure is needed to complete collaborative reasoning and allocate computational tasks. While terminal devices are given some computing power, they cannot support local processing of all tasks, especially complex tasks requiring deep learning. Uploading raw data to the cloud not only increases communication overhead but may also compromise user privacy. Concepts such as cloudlets and edge / fog servers have emerged, therefore, we aim to achieve this by having terminals and edge / cloud systems collaborate on inference tasks, transmitting only intermediate computation results, thus protecting user privacy and reducing communication pressure. A key issue in this process is how to partition the inference tasks to achieve optimal performance.

[0046] Furthermore, the algorithm's adaptability is improved to address complex communication environments and cloud devices with varying performance levels. Changes in the availability of edge cloud devices and communication links due to the movement of terminal devices can impact task partitioning performance, and user business needs are not static; therefore, a more flexible collaboration solution is required.

[0047] Please see Figure 2 , Figure 2 This is a schematic diagram of a collaborative inference architecture for adaptive model segmentation provided in an embodiment of the present invention. The present invention provides a schematic diagram of a collaborative inference architecture for adaptive model segmentation, which includes three parts:

[0048] Different simplified models of the same original model; online partitioning and deployment inference.

[0049] Specifically, in this invention, a set of simplified models corresponding to the same model is pre-selected. Then, based on the channel state, the mobile terminal's computing resources (including the computing resources of the mobile terminal and the computing resources of the MEC server in the edge cloud of the coverage area), storage resources (including the storage resources of the mobile terminal and the storage resources of the edge cloud of the coverage area), and the latency and accuracy requirements of the service, the optimal model and its corresponding optimal segmentation point are selected. For collaborative inference, the mobile terminal and the MEC server collaborate to complete the inference service. The mobile terminal only needs to load some scaled-down intermediate model data into the edge cloud, reducing the latency of data transmission over the wide area network and thus shortening the response time.

[0050] The model partitioning process is as follows:

[0051] The first step is to extract layer configuration information. The main task is to identify the composition structure of the neural network, such as the network type of each layer and the corresponding number of input and output neurons.

[0052] The second step uses a time prediction model to estimate the latency and output data volume of each layer of the simplified model.

[0053] The third step is to adaptively calculate a reasonable simplified model i and a layer j based on performance indicators such as target delay and target accuracy, combined with the available bandwidth of the current wireless channel and the hardware and software characteristics of the local device and the edge cloud device.

[0054] Finally, in the deployment phase, based on the partitioning layer j selected by the online partitioning algorithm, the simplified model i is divided into two parts. The part requiring less computation is left to be executed on the mobile terminal close to the data source, while the rest is offloaded to the edge cloud for execution, and the results are sent back to the mobile terminal, thus completing the inference task.

[0055] It is evident that the primary choice of split points in a simplified model depends on the following factors:

[0056] 1. The latency and output data volume of each layer of the network model;

[0057] 2. Available bandwidth between mobile devices and the edge cloud;

[0058] 3. Predefined performance indicators such as delay requirements and accuracy requirements.

[0059] The method of the present invention is applied to the above-mentioned online partitioning stage, and the method of the present invention is described in detail below.

[0060] Figure 3This is a flowchart illustrating a model partitioning method based on cloud-edge collaborative reasoning, provided as an embodiment of the present invention. The present invention provides a model partitioning method based on cloud-edge collaborative reasoning, which includes the following steps:

[0061] S0. Construct a simplified model library;

[0062] A simplified model library is constructed based on solutions in relevant technologies.

[0063] S1. Obtain the task objective, which may include: latency, accuracy, or model quality;

[0064] S2, perceive the communication environment and edge cloud computing power, traverse each simplified model, and estimate the inference latency, inference accuracy, and partitioning quality of each simplified model.

[0065] The expression for the inference latency of the simplified model is:

[0066]

[0067] Here, latency represents the inference latency of the simplified model. It's important to note that latency here is a general term; this formula applies to all simplified models. The runtime latency from the first layer to the j-th layer of the simplified model in the mobile terminal; This represents the runtime latency after the j-th layer of the simplified model; This represents the transmission time required for the j-th layer output data of the simplified model to be transmitted over a wireless network. The data is obtained based on the output data volume of the j-th layer of the simplified model and the wireless channel transmission rate. Based on the factors related to runtime at each layer of the simplified model, the amount of output data at each layer, and the CPU frequency of the mobile terminal; and The data is obtained based on factors related to runtime in each layer of the simplified model, the amount of output data in each layer, and the CPU frequency in the edge cloud.

[0068] The runtime of a neural network layer is affected by multiple factors, such as the computing power of the device (here, "device" refers to all devices involved in the computation, such as mobile terminals and MEC servers), real-time memory usage, and model parameters. Table 1 shows the latency-related factors in the simplified model.

[0069] Table 1 Factors affecting delay

[0070]

[0071]

[0072] The network's runtime latency t can be expressed as follows, where φ represents the set of multiple factors related to runtime in the network model, D represents the amount of data, and C represents the device's CPU frequency.

[0073]

[0074] Assuming the original model has a total of M layers, the set of indices of the original model layers is represented as {L}. j |j=1,2,…,M} (It is worth noting that: here we assume that the simplified model has no simplified layers, so the number of layers M here applies to all simplified models, as can be seen in the following description, the number of layers of all simplified models is also M), the set of simplified models is represented as {N i |i=1,2,…,M N} indicates that there are a total of M N A simplified model (which can also be understood as M in the simplified model library) N (A simplified model). Assume a simplified model N. i The accuracy of the reasoning is {A} i |i=1,2,…,M N}, the Nth i The Lth simplified model j The output data volume of the layer is Let T represent the minimum tolerable latency of the target task, and A represent the inference accuracy requirement of the target task.

[0075] This invention focuses on computational and transmission latency during task inference.

[0076] The computational latency can be calculated using a layer latency prediction model. Assuming both the mobile terminal and the edge cloud have CNN models (referring to simplified models), for ease of representation, we use... Let $\mathbf{j}$ represent the runtime latency of the first to j-th layers of the CNN model executed on the mobile terminal and the runtime latency of the j-th layer and beyond executed on the edge cloud, respectively, which can be calculated using Equation 1:

[0077] Runtime delay from the first layer of the simplified model to the j-th layer of the simplified model The expression is:

[0078]

[0079] Where, φ l D represents the set of factors related to runtime in the l-th layer of the simplified model. l C represents the output data volume of the l-th layer of the simplified model. mobile This refers to the CPU frequency of the mobile terminal.

[0080] Runtime latency after the j-th layer of the simplified model The expression is:

[0081]

[0082] Where, φ l D represents the set of factors related to runtime in the l-th layer of the simplified model. l C represents the output data volume of the l-th layer of the simplified model. server This refers to the CPU frequency at the edge cloud.

[0083] In one feasible approach, using This represents the transmission time required for the output data of the j-th layer of the simplified model to travel on a wireless network. The transmission time on the wireless network is related to the amount of output data D of the j-th layer of the neural network. j It is related to the upload rate, and the upload rate R is related to the wireless communication technology used in the system. Here, R refers to the average upload rate in this invention.

[0084] Transmission time required for the j-th layer output data of the simplified model on the wireless network The expression is:

[0085]

[0086] Among them, D j R represents the output data volume of the j-th layer of the simplified model; R is the upload rate.

[0087] The inference latency of a simplified model can then be expressed as the sum of computation latency and transmission latency:

[0088]

[0089] Inference accuracy refers to the ratio of the number of input samples correctly predicted by a CNN to the total number of input samples, reflecting the performance of the CNN model. For mobile intelligent applications requiring high reliability, such as autonomous vehicles and facial recognition, ultra-high accuracy is often necessary. Besides the inference ability of the CNN model itself, the accuracy is also affected by the speed at which input data is fed into the CNN model.

[0090] A higher compression ratio will lead to a decrease in model accuracy. On the one hand, a high compression ratio can reduce the number of parameters and thus reduce the amount of computation, making the model easier to deploy. On the other hand, high accuracy means that the model has better generalization ability.

[0091] With QoM i The quality of the simplified model i is measured, and the expression for the quality of the simplified model i is:

[0092]

[0093] Where Acc represents the accuracy of the original model corresponding to the simplified model, acc i The ratio is the accuracy of the simplified model obtained by pruning the i-th simplified model. i Let α be the compression ratio of the simplified model obtained by pruning the i-th simplified model. The weights, β is Weights; -ratio min It represents the lowest compression ratio among all simplified models.

[0094] Here, α and β represent the task's requirements. For applications requiring high reliability, α can take a higher value, while for resource-constrained terminal devices, β often corresponds to a higher value.

[0095] The compression ratio of the simplified model obtained by pruning the i-th simplified model i The expression is:

[0096]

[0097] Where sum is the total number of convolutional kernels in the original model corresponding to the simplified model i, and rem i This represents the number of convolutional kernels in simplified model i.

[0098] Based on this, the latency caused by edge collaborative inference is combined with QOM to measure the compression and latency benefits of model partitioning, and the selection of model partition layer indexes is guided by the Quality of Model Partition (QoMP).

[0099] The partitioning quality QoMP of the j-th layer of the simplified model i i j The expression is:

[0100]

[0101] Equation 8 defines the overall benefit of the simplified model i splitting at level j. Here, η and μ represent QoM, respectively. i The weights and the weights of the inference delay calculated using Equation 5.

[0102] It can also be used for QoMP i j Normalization is performed so that all values ​​are within the range of (0,1) to make the results more intuitive to be displayed.

[0103] S3. When the task objective is time delay, select the simplified model i with the smallest inference delay and the partitioning layer j from the simplified models that meet the inference accuracy requirements.

[0104] or,

[0105] When the task objective is accuracy, obtain the simplified model whose inference latency meets the minimum tolerance latency requirement; among the simplified models that meet the minimum tolerance latency, obtain the simplified model i with the highest inference accuracy, and the partitioning layer j with the lowest inference latency corresponding to the simplified model with the highest inference accuracy.

[0106] or,

[0107] When the task objective is the partitioning quality of the model, obtain the simplified model i and the partitioning layer j with the highest partitioning quality of the simplified model.

[0108] Specifically, this invention also provides the steps of the on-demand model partitioning algorithm:

[0109] The algorithm is the pseudocode of the online on-demand model partitioning point selection algorithm. The algorithm takes into account factors such as the difference in computing resources between mobile terminals and edge cloud, and wireless networks, and achieves dynamic partitioning with OptTarget as the target of the shortest latency, the highest model accuracy, or the model quality.

[0110]

[0111]

[0112] Algorithm execution flow

[0113] The inputs required for the on-demand model partitioning algorithm include:

[0114] ① The amount of hierarchical output data and computational latency of multiple simplified models;

[0115] ② The accuracy of different simplified models, the tolerable accuracy loss and the minimum tolerable latency, because different tasks have different requirements and the algorithm needs basic target information;

[0116] ③ System factors, such as wireless channel conditions, mobile devices, and the computing power of the edge cloud. The algorithm outputs a suitable simplified model and its corresponding partition layer index.

[0117] As shown in steps 1-7, the algorithm first calculates the latency of different pruned CNNs. Steps 8-16 serve three objectives: when the optimization objective is latency, it outputs the index of the partition point that minimizes latency and the corresponding simplified model, while meeting the minimum accuracy requirement; when the optimization objective is accuracy, it finds the output corresponding to the model with the highest accuracy while meeting the minimum latency requirement; when storage is limited and the optimization objective is model quality, it sorts the models according to their QoMP scores. It is evident that a trade-off is made between accuracy, latency, and energy based on business needs to find the partition points that meet the objectives.

[0118] After obtaining the simplified model i and the partitioned layer j, a deployment phase is required. The simplified model i is divided into two parts. The part with less computation is executed on the terminal close to the data source, while the rest is offloaded to the edge cloud for execution. The results are then sent back to the terminal, thus completing the inference task.

[0119] Specifically, a split point j is defined to divide the simplified model i into two parts. The mobile terminal acquires the perception task and input data, executes the first j layers, and uploads the calculation results of the j-th layer to the edge cloud. The edge cloud then executes the j-th layer to the last layer. After the calculation is completed, the decision results are sent back to the mobile terminal.

[0120] This invention abandons the practice of deploying a general model to the cloud, and instead prepares multiple sets of simplified models to meet the different service needs of mobile terminals. This invention does not only consider the computing power of mobile terminals and edge clouds to select fixed model segmentation points, but also considers the flexible selection of segmentation points in response to changes in the wireless communication environment.

[0121] In summary, the cloud-edge collaborative inference-based model partitioning method provided by this invention exhibits strong adaptability. Unlike fixed models, this invention proposes pre-preparing a set of simplified models (existing model simplification algorithms can achieve compression rates of tens of times, thus allowing for the preparation of multiple simplified models without incurring storage costs), and adaptively selecting the optimal model based on channel conditions, MEC server configuration, etc., and cascading the model partitioning function. Furthermore, the cloud-edge collaborative inference-based model partitioning method provided by this invention has a wide service coverage. Due to the sufficient number of candidate models, a trade-off can be made between accuracy, latency, and energy according to service requirements, and the selection of models and partitioning points can meet different target needs.

[0122] Furthermore, specifically, this invention relates to a model partitioning method based on cloud-edge collaborative inference. By setting a split point, the inference model is divided into two parts. The mobile terminal executes the first part, transmitting intermediate computational data to the edge cloud, which then completes the remaining part and returns the final result. Specifically: a simplified model library is constructed based on different compression and simplification algorithms for the original task model. Dual backups are implemented on the edge cloud and the mobile terminal. When a task arrives, the task objective is obtained, including latency, accuracy, or model quality; the communication environment is perceived, including the computing power of the edge cloud and the local computing power that can provide computational services. Each simplified model parameter in the simplified model library is traversed: the number of network layers and the size of each layer. The performance of the simplified model is estimated based on the following formulas: inference latency, inference accuracy, and partitioning quality. When the task objective is latency, the simplified model i with the minimum inference latency and the partitioning layer j are selected from the simplified models whose inference latency meets the accuracy requirement. Alternatively, when the task objective is accuracy, the simplified model whose inference latency meets the minimum tolerance latency requirement is obtained. Among the simplified models that meet the minimum tolerance latency requirement, the simplified model i with the highest inference accuracy and the partitioning layer j with the lowest inference latency corresponding to the simplified model with the highest inference accuracy are obtained. Alternatively, when the task objective is the partitioning quality of the model, the simplified model i with the highest partitioning quality and the partitioning layer j are obtained.

[0123] Another aspect of the present invention provides a storage medium storing a computer program for executing the steps of the model partitioning method based on cloud-edge collaborative inference in the above embodiments. Yet another aspect of the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the model partitioning method based on cloud-edge collaborative inference in the above embodiments. Specifically, the integrated modules implemented as software functional modules can be stored in a computer-readable storage medium. The software functional modules stored in a storage medium include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0124] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0125] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or data point described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or data points described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0126] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A model partitioning method based on cloud-edge collaborative reasoning, characterized in that, Includes the following steps: S0. Construct a simplified model library; S1. Obtain the task objective, which includes: latency, accuracy, or model quality; S2. Sensing the communication environment and edge cloud computing power, traversing each simplified model, and estimating the inference latency, inference accuracy, and partitioning quality of each simplified model. The expression for the inference latency of the simplified model is: Where latency is the inference latency of the simplified model. The runtime latency from the first layer to the j-th layer of the simplified model in the mobile terminal; This refers to the runtime latency after the j-th layer of the simplified model; This represents the transmission time required for the j-th layer output data of the simplified model to travel on the wireless network; the The data is obtained based on the output data volume of the j-th layer of the simplified model and the wireless channel transmission rate. Based on the factors related to runtime of each layer of the simplified model, the amount of output data of each layer, and the CPU frequency of the mobile terminal; and the... The data is obtained based on the factors related to runtime of each layer in the simplified model, the amount of output data of each layer, and the CPU frequency of the edge cloud. S3. When the task objective is latency, select the simplified model i with the smallest inference latency and the partitioning layer j from the simplified models whose inference accuracy meets the latency requirement; or, when the task objective is accuracy, obtain the simplified model whose inference latency meets the minimum tolerance latency requirement; among the simplified models that meet the minimum tolerance latency, obtain the simplified model i with the highest inference accuracy and the partitioning layer j with the lowest inference latency corresponding to the simplified model with the highest inference accuracy; or, when the task objective is the partitioning quality of the model, obtain the simplified model i and the partitioning layer j with the highest partitioning quality of the simplified model, where i and j are positive integers.

2. The model partitioning method based on cloud-edge collaborative reasoning according to claim 1, characterized in that, The runtime delay of the simplified model from the first layer to the j-th layer in the mobile terminal The expression is: Where, φ l D represents the set of factors related to runtime in the l-th layer of the simplified model. l This represents the output data volume of the l-th layer of the simplified model, where C... mobile This refers to the CPU frequency of the mobile terminal.

3. The model partitioning method based on cloud-edge collaborative reasoning according to claim 1, characterized in that, Runtime latency after the j-th layer of the simplified model The expression is: Where, φ l D represents the set of factors related to runtime in the l-th layer of the simplified model. l This represents the output data volume of the l-th layer of the simplified model, where C... server This refers to the CPU frequency at the edge cloud.

4. The model partitioning method based on cloud-edge collaborative reasoning according to claim 1, characterized in that, The transmission time required for the j-th layer output data of the simplified model in the wireless network The expression is: Wherein, the D j R represents the output data volume of the j-th layer of the simplified model; R is the upload rate.

5. The model partitioning method based on cloud-edge collaborative reasoning according to claim 1, characterized in that, The partitioning quality (QoMP) of the simplified model i j The expression is: Wherein, the QoMP i j The partitioning quality of the j-th layer of the simplified model i; the latency is the inference latency of the simplified model; the QoM i The quality of simplified model i; where η is QoM i The weight of latency; μ is the weight of latency.

6. The model partitioning method based on cloud-edge collaborative reasoning according to claim 5, characterized in that, The expression for the quality of the simplified model i is: Wherein, Acc represents the accuracy of the original model corresponding to the simplified model i, and acc i The ratio is the accuracy of the simplified model obtained by pruning the i-th simplified model. i The compression ratio of the simplified model obtained by pruning the i-th simplified model, where α is... The weight, where β is The weight, the ratio; min It represents the lowest compression ratio among all simplified models.

7. The model partitioning method based on cloud-edge collaborative reasoning according to claim 6, characterized in that, The compression ratio of the simplified model obtained by pruning the i-th simplified model i The expression is: Wherein, sum is the total number of convolutional kernels in the original model corresponding to the simplified model i, and rem i The number of convolution kernels in the simplified model i.

8. The model partitioning method based on cloud-edge collaborative reasoning according to claim 5, characterized in that, The method further includes: For the QoMP i Perform normalization processing to make the QoMP i The values ​​of are all within the range of (0,1].

Images