An end-side CNN distributed collaborative computing method and system

Abstract

The application provides a kind of end side CNN distributed collaborative computing method and system, belong to edge computing and computer vision technical field.The method includes: the target CNN model is converted into standardization operator sequence and the calculation database is established;According to network level, the operator sequence is divided into multiple sub-sequences;Real-time monitoring node processing delay, judge whether there is load imbalance;The input channel of bottleneck convolution layer is split and distributed to adjacent nodes according to the proportion of computing power;The segmented sub-model is deployed to each end side device;Each node executes inference task in a pipeline manner and transmits intermediate feature map through local area network.The application also provides a corresponding system.The application adopts coarse-grained hierarchical segmentation + fine-grained operator deconstruction two-level dynamic segmentation architecture, solves the load imbalance problem in multi-end side collaboration, breaks through the single device computing power bottleneck, reduces the dependence on cloud computing power and network bandwidth, and improves the system robustness and data privacy protection ability.

Description

Technical Field

[0001] This invention relates to the fields of edge computing and computer vision technology, and in particular to a distributed collaborative computing method and system for edge-side CNNs. Background Technology

[0002] To address the challenges of computational power dependence, network latency, data privacy risks, and centralized bottlenecks in cloud-based CNN inference, as well as the insufficient computational power and memory constraints faced by single-device inference on edge devices, those skilled in the art have attempted to overcome single-point performance limitations by employing multi-device collaborative computing. However, existing edge collaborative solutions typically partition the model using network layers such as convolutional layers and fully connected layers as the smallest unit. This coarse-grained, static partitioning method fails to perceive the differences in computational density within each layer and cannot dynamically adjust task allocation based on the real-time computing power of the devices. This results in a severe imbalance in workload among nodes, with some devices becoming overloaded and performance bottlenecks while other devices remain idle, severely restricting the throughput and scalability of distributed inference.

[0003] The root cause of the above problems lies in the following: On the one hand, the centralized cloud architecture inherently suffers from network dependence and centralization risks, making it difficult to meet the needs of edge computing scenarios with high real-time requirements and sensitive data privacy; on the other hand, multi-device collaborative computing has not yet solved the core problem of load balancing. The coarse-grained segmentation method based on network layers cannot achieve fine allocation of computing power, which limits the overall performance of the cluster to the slowest node and fails to fully realize the computing potential of multi-device collaboration.

[0004] Therefore, there is an urgent need for an edge-side CNN distributed collaborative computing method that can achieve load-balanced collaborative inference based on multiple embedded devices with limited computing power while protecting data privacy. This would break through the single-point computing power bottleneck, reduce dependence on the cloud, and meet the real-time and reliability requirements of scenarios such as autonomous driving and industrial quality inspection. Summary of the Invention

[0005] The purpose of this invention is to provide a distributed collaborative computing method and system for edge-side CNNs. It addresses the network dependence, privacy risks, computing power bottlenecks of single-edge inference, and load imbalance of multi-edge collaborative inference in existing cloud-based CNNs. It provides a two-level dynamic segmentation architecture based on "coarse-grained hierarchical segmentation + fine-grained operator deconstruction" to achieve load balancing and high-throughput inference of edge-side device clusters, while reducing dependence on cloud computing power and network bandwidth.

[0006] To achieve the above objectives, this invention provides a distributed collaborative computing method for edge-side CNNs, comprising the following steps: Step S1: Convert the target CNN model into a normalized operator sequence. And establish a database of computational costs for each network layer; Step S2: Number of nodes in the end-side device cluster , operator sequence Classified by network hierarchy A sequence of segments, each segment corresponding to a computing task on an edge device; Step S3: Monitor the processing latency of each node in real time to determine if there is a load imbalance. If the difference in processing latency between nodes exceeds a preset threshold, fine-grained segmentation is triggered. Step S4: Split the input channels of the bottleneck convolutional layer in the overloaded node and distribute them to adjacent nodes for collaborative computing according to the computing power ratio; Step S5: Deploy the sub-models after coarse-grained segmentation or a combination of coarse and fine-grained segmentation to each end-side device. Step S6: Each end-side device executes the assigned inference task in a pipeline manner, transmits intermediate feature maps through the local area network, and the final inference result is output by the end aggregation node.

[0007] Preferably, establishing the computational complexity database in step S1 includes: traversing each network layer of the CNN model and calculating the multiplication and addition operations of each layer, with the specific calculation formula as follows: Convolutional layers: ; in, For multiplication and addition operations, The kernel size is [size]. , These represent the number of input and output channels for the convolutional layer, respectively. , These represent the height and width of the output feature map of the convolutional layer, respectively. Fully connected layer: ; in, The computational cost of the fully connected layer, This represents the number of input neurons in the fully connected layer. This represents the number of output neurons; BN layer: ; in, The computational cost of the BN layer, The number of feature map channels. For feature map height, The width of the feature map; Activation function: ; in, The computational cost of the activation function; Pooling layer: ; in, The computational cost of the pooling layer, This represents the pooling kernel size.

[0008] Preferably, the coarse-grained hierarchical segmentation in step S2 further includes: Based on the computational database and real-time performance monitoring data of each node, a dynamic programming algorithm is used to search for the optimal split point, with the objective function being to minimize the maximum single-node time. ; The segmentation boundary is determined by the node computation cost model: ; in, For nodes The computational delay, For nodes Transmission delay, To calculate the cost, To be assigned to a node The set of network layers hierarchical The computational load, For nodes Real-time available computing power.

[0009] Preferably, the preset threshold for bottleneck detection in step S3 is set according to the system's real-time requirements, and the processing latency of each node is obtained through a cross-node time synchronization mechanism. The time synchronization mechanism includes: Record the timestamps of data packet transmission and reception between nodes, and calculate the time offset. : ; in, , These are the sending and receiving timestamps of device A, respectively. , These are the sending and receiving timestamps for device B, respectively.

[0010] Preferably, the fine-grained operator deconstruction in step S4 further includes: The bottleneck convolutional layer of overloaded nodes is identified by its computational density, and the communication computation ratio (CCR) is introduced to evaluate the migration cost. ; in, For the time consumed by cross-node transmission, Calculate the time taken locally; The input channels of the convolutional layer are split proportionally, and the channel splitting ratio is dynamically adjusted based on the real-time clock frequency and load rate of each node to ensure load balance among nodes: ; in, For nodes Real-time available computing power The sum of computing power of the nodes participating in collaborative computing; After each node completes its local convolution calculation, it outputs the local result. Then, it aggregates all the local results through an addition operation to restore the original output tensor.

[0011] Preferably, the sub-model deployment in step S5 includes: loading each segmented sub-model and its corresponding network parameters into the memory of the corresponding end device, establishing communication connections between nodes, and configuring data transmission protocols.

[0012] Preferably, collaborative reasoning in step S6 further includes: The first-end acquisition node acquires video stream data in real time and performs image scaling and normalization preprocessing operations. The computational complexity of image scaling is quantified by the following formula: ; in, For multiplication and addition operations, , These represent the height and width of the input image, respectively, and Channel represents the number of channels in the image. ; in, These are the original eigenvalues. The mean of the feature map, For the variance of the feature map, To prevent values ​​with a denominator of 0; After the first acquisition node completes the first inference task, FP16 half-precision quantization technology is used to compress the intermediate feature maps, with a compression ratio of 2:1. The compression ratio formula is as follows: ; in, This represents the storage size of the intermediate feature map at FP32 precision. This represents the storage size of the intermediate feature map at FP16 precision. Each node executes its assigned inference task in sequence and continuously transmits intermediate feature maps to downstream nodes; The end aggregation node receives the intermediate feature map transmitted by the last node, performs the final inference task and post-processing operation, and outputs the inference result. Post-processing operations include IoU-based target box filtering.

[0013] The present invention also provides an edge-side CNN distributed collaborative computing system for performing the method described above, including: Multiple end-side devices, including at least one head-end acquisition node, several relay computing nodes and one end-end aggregation node, the end-side devices being Raspberry Pi 4B; A gigabit LAN switch is used to connect devices on each end side to enable data communication. Edge devices are used to implement distributed collaborative inference for CNN models; The overall end-to-end delay model of the system is as follows: ; in, The total end-to-end latency of the system, The processing latency of the first-end data acquisition node. For the first Each node transmits data to Data transmission latency of each node For the first The processing latency of each node, The processing latency of the end aggregation node, This represents the total number of endpoint devices in the cluster.

[0014] Preferably, the edge device supports FP16 half-precision quantization and asynchronous thread pool processing to achieve parallel execution of computation and transmission, thereby increasing system throughput. The calculation formula is: ; in, For nodes The computational delay, For nodes Transmission delay.

[0015] Preferably, for CNN models with branching structures, an atomic block encapsulation strategy is adopted, dividing the branching structure as a whole unit to ensure the integrity of the model structure and the correctness of inference. The softmax function of the output layer is: ; in, For the input sample to belong to the first Predicted probability of class This represents the total number of neurons in the output layer of the CNN model. For the output layer The raw computation results received by each neuron is the base of the natural logarithm. For the output layer The original input values ​​of each neuron.

[0016] Therefore, the present invention employs the above-described edge-side CNN distributed collaborative computing method and system, and the beneficial technical effects are as follows: (1) To address the problem of load imbalance among nodes caused by the existing coarse-grained segmentation method with network layers as the smallest segmentation unit, this invention proposes a two-level dynamic segmentation architecture of "coarse-grained hierarchical segmentation + fine-grained operator deconstruction". By monitoring the processing latency of each node in real time, the input channels of the bottleneck convolutional layers with excessive load are split, and the task allocation is dynamically adjusted according to the real-time computing power of the nodes, so that the computing load of each node in the cluster tends to be balanced, effectively eliminating performance bottlenecks, fully tapping the potential of multi-device parallel computing, and the system throughput increases significantly with the expansion of the node scale.

[0017] (2) To address the issues of high computational latency and high memory consumption when running complex CNN models on a single-device platform, this invention utilizes a multi-device pipeline parallel approach to divide the complete model into several sub-models and deploy them to various nodes in the cluster. Each node only needs to load local model parameters, significantly reducing the peak memory consumption of a single node and avoiding process crashes caused by insufficient memory. Simultaneously, by using an asynchronous thread pool to process computation and transmission tasks in parallel, inference latency is greatly shortened, meeting the real-time requirements of scenarios such as autonomous driving and industrial quality inspection.

[0018] (3) To address the issues of strong network dependence, high data privacy risks, and single points of failure inherent in centralized cloud-based inference, this invention completely offloads computing tasks to the edge device cluster. Raw data does not need to be uploaded to the cloud; only intermediate feature maps are transmitted between local devices, eliminating the risk of data leakage at the source. Simultaneously, FP16 half-precision quantization is used to compress the intermediate feature maps, combined with an optimized transmission protocol, effectively reducing communication overhead. When network conditions fluctuate or some nodes fail, the system dynamically adjusts the split points based on real-time performance monitoring data, achieving adaptive load migration, avoiding centralized bottlenecks, and improving the system's robustness and availability. Attached Figure Description

[0019] Figure 1 This is a hardware framework diagram; Figure 2 This is a flowchart of the edge-side distributed computing process; Figure 3 This is a schematic diagram of task segmentation using the MobileNet model as an example; Figure 4 A schematic diagram of the three endpoints for coarse-grained segmentation; Figure 5 A schematic diagram of the three endpoints for fine-grained segmentation; Figure 6 This is a schematic diagram illustrating the division between fine-grained and coarse-grained particles; Figure 7 A schematic diagram of distributed collaborative inference in the YOLOv5s model; Figure 8 Histograms comparing inference latency for different device sizes in MobileNet; Figure 9 A stacked diagram of load distribution at each stage of MobileNet four-device collaborative inference. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0022] Example 1 This embodiment describes in detail a distributed collaborative computing method and system for edge-side CNNs. Based on a two-level dynamic segmentation architecture of "coarse-grained hierarchical segmentation + fine-grained operator deconstruction", it achieves efficient distributed inference of CNN models by accurately allocating computing tasks within a Raspberry Pi cluster.

[0023] I. System Hardware Architecture.

[0024] like Figure 1 As shown, the system in this embodiment consists of multiple end-side devices and a gigabit LAN switch. The end-side devices are Raspberry Pi 4B, including a first-end data acquisition node (Node 1). One relay computing node (node) ) and an end-gathering node (node) All nodes are connected via a gigabit LAN switch with a bandwidth of 1000Mbps. The total end-to-end system latency model is described by formula (1): (1); in, The total end-to-end latency of the system, For the processing latency of node 1, For the first Each node transmits data to Data transmission latency of each node For the first The processing latency of each node, For nodes Processing latency, This represents the total number of endpoint devices (nodes) in the cluster. For the first Each device (node) is represented. This model provides a foundational metric for subsequent task segmentation and performance optimization.

[0025] II. Overall Workflow.

[0026] Figure 2 The flowchart of the edge-side CNN distributed collaborative computing method of the present invention includes the following steps: Step S1, System Modeling: Transform the target CNN model into a normalized sequence of operators. A database of computational costs for each network layer was established. This database serves as the basis for subsequent coarse-grained segmentation.

[0027] Step S2, coarse-grained hierarchical segmentation: Based on the computational database from step S1, the operator sequences are segmented according to the model network hierarchy. Divided into The segment sequence corresponds to a task of an end-side node. The segmentation results are directly used for bottleneck detection.

[0028] Step S3, Bottleneck Detection: Perform load simulation and real-time monitoring on the tasks of each node after coarse-grained segmentation in Step S2 to determine whether there is a load bottleneck, i.e., the processing latency difference between nodes exceeds a threshold (set to 11ms in this embodiment). If a bottleneck exists, fine-grained segmentation is triggered; otherwise, deployment proceeds directly.

[0029] Step S4, Fine-grained operator deconstruction: If a bottleneck is detected in step S3, operator-level deconstruction is performed on the bottleneck convolutional layer of the overloaded node. Based on the input channel splitting strategy, some computational tasks are migrated to adjacent nodes. The output of this step is the optimized task partitioning scheme.

[0030] Step S5, Sub-model Deployment: Deploy the sub-models that have been divided in Step S2 (or Step S2 + Step S4) to their respective Raspberry Pi nodes to complete the physical allocation.

[0031] Step S6, Collaborative Reasoning: Each node executes the reasoning task according to the sub-model deployed in Step S5. After completing the first segment of reasoning, the first node compresses the intermediate feature map and transmits it. The relay nodes calculate and transmit sequentially, and the final aggregation node outputs the final result. During the reasoning process, real-time monitoring data is fed back to Step S3 to achieve dynamic adjustments.

[0032] III. Coarse-grained segmentation algorithm.

[0033] Figure 3 The MobileNet model is used as an example to illustrate task segmentation. In the diagram, rectangles represent data flow Actors (processing units), each actor containing specific CNN layer components (such as Conv2D, BatchNorm, ReLU6), with values ​​representing the input / output data dimensions. PP represents the task segmentation point. Figure 3PP1, PP2, etc., are task splitting points: indicating that the data flow graph can be split into multiple subtasks at this position and assigned to different computing nodes. DWCL1-DWCL11 represent the depthwise convolution layers in MobileNet, which are the core components of depthwise separable convolution.

[0034] Coarse-grained segmentation relies on the hierarchical computational database established in step S1, specifically including: 1. Layer-by-layer computational complexity assessment: Traverse each network layer of the CNN model (convolutional layer, fully connected layer, batch normalization layer, activation layer, pooling layer), and calculate the multiply-accumulate operations (MACs) of each layer using formulas (2) to (6): Convolutional layers: (2); in, For multiplication and addition operations, The kernel size is [size]. , These represent the number of input and output channels for the convolutional layer, respectively. , These represent the height and width of the output feature map of the convolutional layer, respectively.

[0035] Fully connected layer: (3); in, The computational cost of the fully connected layer, This represents the number of input neurons in the fully connected layer. This represents the number of output neurons.

[0036] BN layer: (4); in, The computational cost of the BN layer, The number of feature map channels. For feature map height, This represents the width of the feature map.

[0037] Activation function (ReLU): (5); in, This represents the computational cost of the activation function.

[0038] Pooling layer: (6); in, The computational cost of the pooling layer, This represents the pooling kernel size.

[0039] 2. Optimal split point search: Using dynamic programming, with the goal of "minimizing the maximum single node time" (Formula (7)), combined with the real-time available computing power of each node (calculated by Formula (8) to determine the node's computational cost). Determine the optimal partition boundary: (7); (8); in, For the first The computational latency of each node, For the first Transmission latency of each node To be assigned to the A set of network hierarchy nodes hierarchical The computational load, For nodes The real-time available computing power. The segmentation result is used as input for step S3.

[0040] Coarse-grained partitioning only divides tasks based on network layers, without considering the differences in computing power requirements within a single layer, which may lead to load imbalance. For example... Figure 4 As shown, the processing delays of devices A, B, and C are... , , There are significant differences, and the overall inference efficiency is limited by the node with the heaviest load. This phenomenon is the bottleneck that needs to be detected in step S3.

[0041] IV. Fine-grained segmentation algorithm.

[0042] When step S3 detects a load bottleneck (processing latency difference between nodes > threshold 11ms), fine-grained operator deconstruction is triggered. This algorithm relies on the bottleneck node information provided in step S3 and achieves load balancing through the following sub-steps: 1. Operator Channel Decomposition: Identify the bottleneck convolutional layer of overloaded nodes by judging its computational density and introducing the communication computation ratio. (Formula (9)) Evaluate migration costs: (9); in, For the time consumed by cross-node transmission, The time taken for local calculation.

[0043] when At this time, operator migration has a performance gain. The input channels of the convolutional layer are split proportionally, and the channel splitting ratio is dynamically adjusted according to the real-time main frequency and load rate of each node through formula (10). : (10); in, For nodes Real-time available computing power This refers to the total computing power of the nodes participating in collaborative computing.

[0044] 2. Result Fusion: After each node completes the local convolution calculation, it outputs the local result (partial sum). All local results are aggregated through addition operations to restore the original output tensor and ensure the correctness of model inference.

[0045] like Figure 5 As shown, after fine-grained segmentation optimization, the processing delays of devices A, B, and C tend to be consistent, and the load imbalance problem is significantly improved. Figure 6 Further comparison was made between coarse-grained segmentation (dashed line) and fine-grained segmentation (solid line): coarse-grained segmentation divides tasks according to CNN layers (device A processes layers 1-3, device B processes layers 4-6, and device C processes layers 7-8); fine-grained segmentation splits channels within a layer (such as devices B and C co-processing different channels of layer 6) to achieve load balancing within the layer.

[0046] The task partitioning scheme, after fine-grained adjustments, will be used for the deployment of sub-models in step S5.

[0047] V. Collaborative Reasoning Process.

[0048] This section describes in detail the complete inference process from the first-end data collection node to the last-end data aggregation node. This process relies on the sub-model deployed in step S5 and is continuously dynamically adjusted during operation using real-time monitoring data (step S3).

[0049] 5.1 Data Preprocessing (First Node).

[0050] The first-end acquisition node acquires video stream data in real time and performs preprocessing operations such as image scaling and normalization. The computational cost of image scaling is quantified by formula (11), and its result affects the overall latency: (11); in, These are multiply-accumulate operations (MACs). , These represent the height and width of the input image, respectively, and Channel represents the number of channels in the image.

[0051] The feature map normalization operation is described by formula (12): (12); in, These are the new eigenvalues ​​after normalization. These are the original eigenvalues. The mean of the feature map, For the variance of the feature map, To prevent extremely small values ​​where the denominator is 0 (usually 1e-5 is taken).

[0052] 5.2 Intermediate feature map compression and transmission (between nodes).

[0053] After the first node completes the first inference task, the intermediate feature map is compressed using FP16 half-precision quantization technology with a compression ratio of 2:1, as shown in formula (13): (13); in, This represents the storage size of the intermediate feature map at FP32 precision. This represents the storage size of the intermediate feature map at FP16 precision. This represents the compression ratio. The compressed data is sent to the next node via an optimized transmission protocol.

[0054] To reduce communication overhead, the transmission protocol is optimized using the TCP_NODELAY option, effectively improving bandwidth utilization. Evaluated by formula (14): (14); in, To provide an effective amount of feature map data, This represents the total amount of data transmitted (including protocol headers, checksums, etc.). This indicator is used to guide adjustments to transmission parameters.

[0055] Each relay computing node executes its assigned inference task sequentially and continuously transmits intermediate feature maps to downstream nodes. The output tensor data volume of each layer can be estimated in advance using formula (15) to predict transmission overhead. : (15); in, , , The first The height, width, and number of channels of the layer output feature map. The data storage bit width (e.g., 16 bits for FP16).

[0056] 5.3 Post-processing and output (end aggregation node).

[0057] The final aggregation node receives the intermediate feature map transmitted by the last relay node and performs final inference tasks and post-processing operations. Post-processing includes... Target box filtering, The calculation formula is shown in (16): (16); in, The region for predicting the target box for the model This represents the region of the true target bounding box. This step outputs the final inference result (such as the coordinates of the detected target bounding box).

[0058] System throughput (FPS) is calculated using formula (17) and used to evaluate overall performance: (17); in, For nodes The computational delay, For nodes Transmission delay.

[0059] 5.4 Real-time performance monitoring and time synchronization (throughout the entire process).

[0060] To ensure accurate load balancing, the system obtains the processing latency of each node through a cross-node time synchronization mechanism. (Time offset) Calculated using formula (18): (18); in, , These are the sending and receiving timestamps of device A, respectively. , These are the sending and receiving timestamps for device B, used to achieve cross-node time synchronization and ensure the accuracy of performance monitoring.

[0061] Based on the synchronized timestamps, the system can monitor the processing latency of each node online. , , The data is then fed back to step S3. If load imbalance is detected again (latency difference > threshold), step S4 is triggered again for dynamic adjustment.

[0062] 5.5 Branch structure processing (special considerations when splitting the model).

[0063] For CNN models with branching structures (such as ResNet residual blocks and YOLOv5 slice layers), an atomic block encapsulation strategy is adopted in steps S2 and S4 to ensure structural integrity and inference correctness during model segmentation. The output layer Softmax function is shown in formula (19) to ensure the rationality of the inference results: (19); in, For the input sample to belong to the first The predicted probability of a class takes a value in the range (0,1), and the sum of the predicted probabilities of all classes is 1, forming a valid probability distribution. This represents the total number of neurons in the output layer of the CNN model (i.e., the total number of task categories). The sum of the exponents of the inputs to all neurons in the output layer; For the input vector The Softmax normalization operation is performed. For the output layer The raw computation results received by each neuron; The base of the natural logarithm (a fixed value of approximately 2.71828) is used to transform the input... Convert to non-negative values ​​to ensure the non-negativity of the final output probability; For the output layer The original input values ​​of each neuron.

[0064] Example 2 Distributed collaborative reasoning in the YOLOv5s model.

[0065] Figure 7 This diagram illustrates distributed collaborative inference of the YOLOv5s model across three Raspberry Pi 4B devices (nodes A, B, and C). The implementation steps are as follows, each relying on the aforementioned algorithm: Hierarchical partitioning: Based on the coarse-grained segmentation in step S2, node A runs sequence N1 after segmentation by the YOLOv5s model, node B runs sequence N2, and node C runs sequence N3. The output tensor data volume of each layer is calculated by formula (15) (BitDepth=16bit, using FP16) and used to predict transmission overhead.

[0066] Bottleneck optimization: Through real-time monitoring in step S3, it was found that the computational density of one of the modules of node B was too high, causing pipeline blockage and triggering fine-grained partitioning.

[0067] Fine-grained segmentation: According to step S4, the convolutional kernel channels of this module are split in a ratio of 0.4:0.6 and allocated to nodes B and C for collaborative computation. The splitting ratio is dynamically determined according to formula (10) and ensured by formula (9). To obtain performance gains.

[0068] Pipeline parallelism: An asynchronous thread pool is enabled to process Socket data transmission and inference computation tasks in parallel, eliminating IO blocking. System throughput (FPS) is calculated using formula (17) to verify the optimization effect.

[0069] Example 3 Distributed collaborative reasoning for the MobileNet model.

[0070] The inference acceleration effect of the MobileNet model under different node scales was tested on 1-4 Raspberry Pi 4B devices (gigabit LAN environment). The experimental modes included single-device independent inference, dual-device parallel pipeline, three-device parallel pipeline, and four-device parallel pipeline. All partitions were performed according to the algorithm proposed in this embodiment.

[0071] Average inference latency per device =134.547ms, used as the speedup benchmark. Overall pipeline delay. Defined by formula (20): (20); in, The longest processing latency among all nodes The experimental results are as follows: Parallel pipeline with two devices: Edge + transmission latency 71.199ms, cloud inference latency 73.189ms, overall pipeline latency 73.189ms, load difference only 1.990ms. Speedup. Calculated using formula (21): (twenty one); in, For single-node inference latency, For distributed collaborative reasoning latency; Acceleration ratio: .

[0072] Three-device parallel pipeline: The delays at each stage are 46.244ms, 48.083ms, and 35.069ms, respectively, with an overall pipeline delay of 48.083ms and a speedup ratio of [missing information]. .

[0073] Four-device parallel pipeline: The delays at each stage are 42.02ms, 31.78ms, 32.35ms, and 34.61ms, respectively, with an overall pipeline delay of 42.02ms and a speedup ratio of [missing information]. .

[0074] Figure 8 The histogram comparing inference latency for different device sizes in MobileNet visually demonstrates the trend of latency continuously decreasing as the number of nodes increases. Figure 9 The load distribution stacking diagram for each stage of the four-device collaborative inference is shown. The time consumption of each stage is very close (42.02ms, 31.78ms, 32.35ms, 34.61ms), verifying that the dynamic segmentation algorithm proposed in this invention can effectively achieve load balancing. Load balancing is available. Value (Formula (22)) quantification: (twenty two); in, For nodes Processing latency, This represents the average processing latency across all nodes.

[0075] Experimental results show that the algorithm proposed in this invention exhibits strong performance scalability on lightweight models such as MobileNet. By increasing computing nodes and combining them with a coarse-grained dynamic segmentation strategy, it effectively breaks through the bottleneck of single-point computing power.

[0076] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0077] Therefore, this invention adopts the above-mentioned edge-side CNN distributed collaborative computing method and system. Through a two-level dynamic segmentation architecture of coarse-grained hierarchical segmentation + fine-grained operator deconstruction, it realizes load balancing and high-throughput inference of edge device clusters. It effectively solves the problems of network dependence, privacy risks, single-end computing power bottleneck and multi-end collaborative load imbalance of cloud CNN inference. It is suitable for edge computing scenarios with high real-time requirements and sensitive data privacy, such as autonomous driving and industrial quality inspection.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A distributed collaborative computing method for edge-side CNNs, characterized in that, Includes the following steps: Step S1: Convert the target CNN model into a normalized operator sequence. And establish a database of computational costs for each network layer; Step S2: Number of nodes in the end-side device cluster , operator sequence Classified by network hierarchy A sequence of segments, each segment corresponding to a computing task on an edge device; Step S3: Monitor the processing latency of each node in real time to determine if there is a load imbalance. If the difference in processing latency between nodes exceeds a preset threshold, fine-grained segmentation is triggered. Step S4: Split the input channels of the bottleneck convolutional layer in the overloaded node and distribute them to adjacent nodes for collaborative computing according to the computing power ratio; Step S5: Deploy the sub-models after coarse-grained segmentation or a combination of coarse and fine-grained segmentation to each end-side device. Step S6: Each end-side device executes the assigned inference task in a pipeline manner, transmits intermediate feature maps through the local area network, and the final inference result is output by the end aggregation node. Step S4, fine-grained operator deconstruction, further includes: The bottleneck convolutional layer of overloaded nodes is identified by its computational density, and the communication computation ratio (CCR) is introduced to evaluate the migration cost. ； in, For the time consumed by cross-node transmission, For local calculation time, ; The input channels of the convolutional layer are split proportionally, and the channel splitting ratio is dynamically adjusted based on the real-time clock frequency and load rate of each node. Ensure load balancing among nodes: ； in, For nodes Real-time available computing power The sum of computing power of the nodes participating in collaborative computing; After each node completes its local convolution calculation, it outputs the local result. Then, it aggregates all the local results through an addition operation to restore the original output tensor.

2. The edge-side CNN distributed collaborative computing method according to claim 1, characterized in that, Step S1, establishing the computational complexity database, includes: traversing each network layer of the CNN model and calculating the multiplication and addition operations for each layer. The specific calculation formula is as follows: Convolutional layers: ； in, For multiplication and addition operations, The kernel size is [size]. , These represent the number of input and output channels for the convolutional layer, respectively. , These represent the height and width of the output feature map of the convolutional layer, respectively. Fully connected layer: ； in, The computational cost of the fully connected layer, This represents the number of input neurons in the fully connected layer. This represents the number of output neurons; BN layer: ； in, The computational cost of the BN layer, The number of feature map channels. For feature map height, The width of the feature map; Activation function: ； in, The computational cost of the activation function; Pooling layer: ； in, The computational cost of the pooling layer, This represents the pooling kernel size.

3. The edge-side CNN distributed collaborative computing method according to claim 2, characterized in that, Step S2, coarse-grained hierarchical segmentation, further includes: Based on the computational database and real-time performance monitoring data of each node, a dynamic programming algorithm is used to search for the optimal split point, with the objective function being to minimize the maximum single-node time. ； The segmentation boundary is determined by the node computation cost model: ； in, For nodes The computational delay, For nodes Transmission delay, To calculate the cost, To be assigned to a node The set of network layers hierarchical The computational load, For nodes Real-time available computing power.

4. The edge-side CNN distributed collaborative computing method according to claim 3, characterized in that, In step S3, the preset threshold for bottleneck detection is set according to the system's real-time requirements, and the processing latency of each node is obtained through a cross-node time synchronization mechanism. The time synchronization mechanism includes: Record the timestamps of data packet transmission and reception between nodes, and calculate the time offset. : ； in, , These are the sending and receiving timestamps of device A, respectively. , These are the sending and receiving timestamps for device B, respectively.

5. The edge-side CNN distributed collaborative computing method according to claim 4, characterized in that, The sub-model deployment in step S5 includes: loading each segmented sub-model and its corresponding network parameters into the memory of the corresponding end device, establishing communication connections between nodes, and configuring data transmission protocols.

6. The edge-side CNN distributed collaborative computing method according to claim 5, characterized in that, Step S6, collaborative reasoning, further includes: The first-end acquisition node acquires video stream data in real time and performs image scaling and normalization preprocessing operations. The computational complexity of image scaling is quantified by the following formula: ； in, For multiplication and addition operations, , These represent the height and width of the input image, respectively, and Channel represents the number of channels in the image. ； in, These are the new eigenvalues ​​after normalization. These are the original eigenvalues. The mean of the feature map, For the variance of the feature map, To prevent values ​​with a denominator of 0; After the first acquisition node completes the first inference task, FP16 half-precision quantization technology is used to compress the intermediate feature maps, with a compression ratio of 2:

1. The compression ratio formula is as follows: ； in, This represents the storage size of the intermediate feature map at FP32 precision. This represents the storage size of the intermediate feature map at FP16 precision. Each node executes its assigned inference task in sequence and continuously transmits intermediate feature maps to downstream nodes; The end aggregation node receives the intermediate feature map transmitted by the last node, performs the final inference task and post-processing operation, and outputs the inference result. Post-processing operations include IoU-based target box filtering.

7. A distributed collaborative computing system for edge-side CNNs, characterized in that, For performing the method as described in any one of claims 1-6, comprising: Multiple end-side devices, including at least one head-end acquisition node, several relay computing nodes and one end-end aggregation node, the end-side devices being Raspberry Pi 4B; A gigabit LAN switch is used to connect devices on each end side to enable data communication. Edge devices are used to implement distributed collaborative inference for CNN models; The overall end-to-end delay model of the system is as follows: ； in, The total end-to-end latency of the system, The processing latency of the first-end data acquisition node. For the first Each node transmits data to Data transmission latency of each node For the first The processing latency of each node, The processing latency of the end aggregation node, This represents the total number of endpoint devices in the cluster.

8. A distributed collaborative computing system for edge-side CNNs according to claim 7, characterized in that, The edge device supports FP16 half-precision quantization and asynchronous thread pool processing, enabling parallel execution of computation and transmission, and improving system throughput. The calculation formula is: ； in, For nodes The computational delay, For nodes Transmission delay.

9. A distributed collaborative computing system for edge-side CNNs according to claim 8, characterized in that, For CNN models with branching structures, an atomic block encapsulation strategy is adopted, dividing the branching structure as a whole unit to ensure the integrity of the model structure and the correctness of inference. The softmax function of the output layer is: ； in, For the input sample to belong to the first Predicted probability of class This represents the total number of neurons in the output layer of the CNN model. For the output layer The raw computation results received by each neuron is the base of the natural logarithm. For the output layer The original input values ​​of each neuron.

Images