A pipeline inference method and device based on a heterogeneous edge scene

Abstract

This invention discloses a pipelined inference method and apparatus based on heterogeneous edge scenarios. The method, executed by a master node, includes: predicting the computational and communication latency of the pipeline using a task-device co-analyzer based on given task and device group information, and abstracting the cross-device group adaptability of the deep neural network into an undirected connected graph; generating a type set containing all potentially optimal single-pipeline solutions using a multi-pipeline parallel planner based on the undirected connected graph, and generating a multi-pipeline parallel strategy using a genetic algorithm; and allocating tasks to devices in different pipelines for inference according to the multi-pipeline parallel strategy. By introducing a task-device co-analyzer to accurately estimate pipeline runtime latency and a multi-pipeline parallel planner to generate multi-pipeline parallel strategies, this method supports high-quality offline parallel planning in advance, thereby achieving seamless task switching and efficient pipeline parallelism in the DNN task flow.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of deep neural network technology, and in particular to a pipelined inference method and apparatus based on heterogeneous edge scenarios. Background Technology

[0002] Due to the inherent advantages of edge computing environments in terms of instant communication and privacy, deep neural network (DNN) tasks are gradually shifting to edge devices. However, edge devices are typically limited in storage and computing resources, making it difficult to complete intelligent model inference tasks with rapidly increasing parameter numbers and computational complexity. To address this, the industry has proposed model compression techniques such as pruning, quantization, and knowledge distillation to reduce the number of model parameters and computational complexity without significantly compromising model accuracy. Unlike these model-level optimizations, model parallelism divides large DNNs into multiple slices for parallel processing across multiple devices. This approach integrates the dispersed memory and computing resources on edge devices, supporting the deployment of larger models while accelerating inference speed.

[0003] Model parallelism techniques, such as tensor parallelism and pipelined parallelism, have been well-established in cloud computing environments, but migrating them to edge environments is not easy. Tensor parallelism requires distributing the computational workload of a single operator (e.g., matrix operations) across multiple devices. Since edge devices typically lack high-speed dedicated interconnect networks, such as NVLink or InfiniBand, the frequent communication required for intra-operator synchronization introduces significant synchronization overhead, severely impacting parallel efficiency. In contrast, pipelined parallelism divides the model into multiple sequential stages, executed sequentially by different devices, with each device simply forwarding its output to the next. This avoids collective communication and synchronization among all workers, leveraging the overlap of computation and communication to reduce latency. However, in cloud computing environments, pipelined parallelism research primarily focuses on homogeneous devices or considers only limited heterogeneity, such as heterogeneous communication topologies with homogeneous graphics processing units (GPUs) or heterogeneous clusters with homogeneous networks. In contrast, edge computing scenarios involve the coexistence of multiple heterogeneous devices interconnected with different configurations. This makes existing pipelined parallelism methods from cloud computing scenarios unsuitable for edge computing.

[0004] Currently, some research has designed pipelined parallel methods for addressing computational and communication heterogeneity in edge computing scenarios. In these methods, accurately estimating computational and communication latency in highly heterogeneous environments is crucial, as these latency directly impacts the results of parallel planning. To this end, most studies employ a profiling strategy, decomposing the DNN into smaller units and executing them on various devices to measure the corresponding latency. The profiling process is very time-consuming, leading to significant waste of resources and inefficiency in the planning process. Some studies have introduced cost models, but these models typically only consider the statistics of the DNN and the configuration information of the devices, such as the number of DNN parameters and device communication bandwidth. Experiments show that the predictions of these cost models deviate significantly from actual results, leading to uneven load distribution of the DNN across devices and consequently impairing pipeline efficiency. Summary of the Invention

[0005] To address the challenge of low efficiency in existing methods during planning or pipeline operation, this invention provides a pipelined inference method and apparatus based on heterogeneous edge scenarios to support high-quality offline parallel planning in advance, thereby achieving seamless task switching and efficient pipeline parallelism in DNN task flow.

[0006] In a first aspect, embodiments of the present invention provide a pipelined inference method based on heterogeneous edge scenarios, the method being executed by a master node, including:

[0007] Based on information about given tasks and device groups, the computational and communication latency of the pipeline is predicted by a task-device co-analysis analyzer, and the adaptability of deep neural networks across device groups is abstracted as an undirected connected graph.

[0008] Based on the aforementioned unconnected graph, a type set containing all potential optimal single-pipeline solutions is generated by a multi-pipeline parallel planner, and a multi-pipeline parallel strategy is generated using a genetic algorithm.

[0009] The task is assigned to devices in different pipelines for inference according to the multi-pipeline parallel strategy.

[0010] Optionally, each node of the undirected connected graph represents the inference latency of any segment of the deep neural network across various devices.

[0011] An edge in an undirected connected graph corresponds to a communication matrix, which represents the communication latency between devices.

[0012] Optionally, pipeline computational latency can be predicted using a task-device co-analyzer, including:

[0013] A prediction model based on a self-attention mechanism is adopted, in which fine-grained task features are input into the encoder, data-driven device features are embedded into the decoder, and then the outputs of the decoder and encoder are cross-attention mechanism and connected to a multilayer perceptron to generate computational latency.

[0014] Optionally, the fine-grained task features include multiple embedding vectors, each representing a separate layer, and each separate layer consists of three components: layer type information, layer features, and position encoding.

[0015] Optionally, the estimation function for the communication delay is:

[0016]

[0017] Among them, T ende For data encoding and decoding latency, T ende =w1×x plan +ε1,x plan For the efficiency of encoding and decoding operations on the device's CPU;

[0018] T n To handle latency for nodes, T p For the delay of propagation, T n +T p =ε2;

[0019] T t For transmission delay, b represents bandwidth;

[0020] T q Due to queuing delays, Δx represents the queuing delay between current devices, Δx represents the difference in the amount of data transmitted between devices, and Δx represents the difference in the average amount of data transmitted among all workers.

[0021] Optionally, the multi-pipeline parallel planner uses a dynamic programming algorithm to generate a type set containing all potential optimal single-pipeline solutions;

[0022] The dynamic programming algorithm uses the maximum number of pipelines to limit the solution space of the multi-pipeline parallel strategy.

[0023] The multi-pipeline parallel planner is also designed with a single-pipeline partitioning generator, which is used to generate the optimal partitioning PTS for a single pipeline within a specified device subgroup.

[0024] Optionally, a multi-pipeline parallel strategy can be generated using a genetic algorithm, including:

[0025] In genetic algorithms, each multi-pipeline partitioning strategy is considered as a chromosome, and multiple chromosomes form a population. Each chromosome is composed of S... max Gene composition;

[0026] Each gene selects a pipeline from the PTS, formalized as g k =i(k∈[1,S) max ], i∈[0,M]);

[0027] The initial population consists of one chromosome. In each generation, the genetic algorithm uses a roulette wheel strategy to select two chromosomes from the population for crossover and mutation to form two new chromosomes. This process is repeated until a new population is generated. After multiple generations of iteration, the chromosome with the highest fitness throughout the entire genetic algorithm process is selected as the optimal partitioning strategy.

[0028] Optionally, the pipelined inference method based on heterogeneous edge scenarios uses PyTorch's Gloo as the communication backend.

[0029] Secondly, ten embodiments of the present invention provide a pipelined inference device based on heterogeneous edge scenarios, comprising:

[0030] The task-device co-processing analyzer predicts pipeline computation and communication latency based on information from a given task and device group, and abstracts the adaptability of deep neural networks across device groups as an undirected connected graph.

[0031] A multi-pipeline parallel planner is used to generate a set of types containing all potential optimal single-pipeline schemes based on the unconnected graph, and to generate multi-pipeline parallel strategies using a genetic algorithm.

[0032] The inference task allocation module is used to allocate tasks to devices in different pipelines for inference according to the multi-pipeline parallel strategy.

[0033] This invention provides an innovative pipelined inference framework (EPipe) that supports high-quality offline parallel planning for heterogeneous edge devices. Specifically, 1) it abstracts the cross-device compatibility of DNNs into a UCG and designs a Task-Device Cooperative Analyzer (TDC) to obtain accurate latency estimates. 2) it designs a dynamic programming (DP)-based genetic algorithm (MPP) to expand the solution space of traditional single-pipeline parallelism, achieving multi-pipeline parallelism. EPipe provides an end-to-end automatic parallel implementation framework for a given task and device, automatically completing the entire parallel process. Furthermore, comprehensive experimental evaluations conducted on an edge test platform show that EPipe can significantly improve the inference task throughput of individual tasks and task flows. Attached Figure Description

[0034] Figure 1 This embodiment provides a framework diagram for a pipelined inference method based on heterogeneous edge scenarios;

[0035] Figure 2 A schematic diagram of computational delay modeling provided in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of communication delay modeling provided in an embodiment of the present invention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0038] This embodiment provides a pipelined inference method based on heterogeneous edge scenarios. See details below. Figure 1 The upper part of the figure shows that the method includes a Task-Device Cooperative Analyzer (TDC) and a Multi-Pipeline Parallel Planner (MPP), where TDC is used to ensure accurate latency estimation and MPP is used for offline planning of multi-pipeline parallel optimization schemes. The lower part of the figure highlights the automated end-to-end implementation process of EPipe.

[0039] Within the EPipe framework, the master node uses information about a given task and device group to coordinate multi-pipeline parallelism to optimize the inference throughput of the device group, and schedules each shard of the DNN to the corresponding edge computing device worker according to the coordinated scheme.

[0040] Specifically, the pipelined inference method based on heterogeneous edge scenarios provided in this embodiment of the invention includes the following steps:

[0041] S1. Based on the information of the given task and device group, predict the computational and communication latency of the pipeline through the task-device co-analysis analyzer, and abstract the adaptability of the deep neural network across device groups as an undirected connected graph.

[0042] S2. Based on the aforementioned unconnected graph, a type set containing all potential optimal single-pipeline solutions is generated using a multi-pipeline parallel planner, and a multi-pipeline parallel strategy is generated using a genetic algorithm.

[0043] S3. Based on the multi-pipeline parallel strategy, the task is assigned to devices in different pipelines for inference.

[0044] In step S1, due to different hardware architectures and characteristics, the performance of the same DNN slice can vary significantly across heterogeneous devices. Therefore, TDC uses an undirected connected graph (UCG) to represent the fitness of the DNN on the device group. Specifically, each node of the UCG corresponds to a single worker, whose information includes the computational latency of each layer, while the edges of the UCG correspond to a communication matrix containing the communication latency of the output during transmission between all workers. This embodiment completes the information in the UCG by injecting features of the task and the device, which is the basis for efficient pipelined parallel planning.

[0045] Furthermore, the computational latency and communication latency in this embodiment are predicted through computational latency modeling and communication latency modeling, specifically including the following:

[0046] 1) Computational delay modeling

[0047] The purpose of computational latency modeling is to complete the information about UCG nodes. For example... Figure 2 As shown, TDC employs a prediction model based on a self-attention mechanism (Transformer), taking fine-grained task features and data-driven device feature embeddings as inputs and the computational latency of each layer on the device as output.

[0048] Fine-grained task features: Compared to methods that rely solely on statistical metrics, TDC employs fine-grained features to more fully characterize the DNN task, specifically including multiple embedding vectors. Each embedding vector represents a separate layer and consists of three components: layer type information, layer features, and position encoding.

[0049] First, because different types of layers exhibit varying computational latency even with the same computational cost, layer type information needs to be introduced for representation. Subsequently, with this layer type information, the computational latency of a single layer becomes affected by features such as computational cost; these features can be derived from third-party libraries like TorchInfo. Furthermore, the data dependencies between layers in DNNs become increasingly complex, and multi-branch network structures and intricate data dependencies significantly impact inference performance. Therefore, TDC incorporates positional encoding into each layer to effectively capture layer-specific dependency attributes, such as the number of branches (Depth) and the relative position (Index) of that layer within its branches. Additionally, information about the parent node is included to incorporate its data dependencies.

[0050] Data-Driven Device Feature Embedding: In this embodiment, TDC designs a data-driven approach to embed device features, rather than relying solely on hardware configuration. Initially, TDC constructs a dataset by testing the inference latency of various real-world DNNs on a given device, with each sample being a uniform key-value pair (key: DNN statistics, value: latency). Subsequently, a linear regression model is employed to capture the predictive relationships between these key-value pairs, aiming to capture the predictive relationships through the parameters (w) of the linear model. k This method condenses the performance differences between different DNNs on a device into a unified representation. Each device category has a different set of datasets based on its own dataset, which are then used as feature embeddings for the device. This embedding method normalizes the performance differences of heterogeneous devices onto a unified, continuous hyperspace. It solves the problem of unified representation difficulties caused by the inconsistency, incompleteness, and discontinuity of device configuration information, thereby achieving comprehensive end-to-end performance representation of heterogeneous devices.

[0051] Transformer-based prediction model: Based on the two inputs mentioned above, the Transformer is used as the backbone network for computing the latency prediction model in TDC. These fine-grained task features are input as tokens into the encoder, and device features are embedded into the decoder. The decoder and encoder outputs undergo a cross-attention mechanism and are connected to a multilayer perceptron (MLP) to generate the final predicted latency.

[0052] 2) Communication delay modeling

[0053] The purpose of communication delay modeling is to complete the edge information of UCG. For example... Figure 3 As shown, to facilitate pipeline overlap between computation and communication, each node is equipped with at least two asynchronous threads: a computation thread and a communication thread. The communication thread uses separate input and output queues to effectively isolate its tasks from those managed by the computation thread.

[0054] Figure 3 Communication latency is defined as the duration from the start of data transmission in the output queue to the completion of reception in the input queue. It includes data encoding and decoding latency (T0). ende ), node processing latency (T) n Queuing delay (T) q ), transmission delay (T) t ) and propagation delay (T) p This can be formalized as:

[0055] T all =T ende +T n +T q +T t +T p

[0056] To ensure the accuracy of communication delay estimation, TDC analyzed the factors affecting each stage of communication between workers.

[0057] T ende To facilitate the transmission of tensors between different hosts, it is necessary to serialize the tensor into a byte stream and then serialize it again upon receipt. Assume the efficiency of the encoding and decoding operations on the device's CPU (x...) pian This delay can be fitted by introducing a linear function, i.e.: T ende =w1×x plan +ε1.

[0058] T n ,T p In edge LAN environments, due to the extremely low bit error rate and the close geographical proximity of workers, node processing and propagation delays are stable and significantly lower than the total communication delay. A constant bias can be introduced to estimate them, i.e., T n +T p =ε2.

[0059] T t Transmission delay is determined by the ratio of data volume to bandwidth. It can be expressed as:

[0060] T q Network latency is affected by actual network congestion. When network traffic is high and the processing capacity of devices is relatively insufficient, network congestion will increase significantly. We will discuss this in detail below.

[0061] The queuing latency of a pipeline being planned offline can be approximated by measuring the current queuing latency. However, the architecture of the DNN used in the current task may differ significantly from that of the DNN being planned, so the network traffic in the current pipeline may differ greatly from the network traffic expected in the pipeline being planned.

[0062] Therefore, TDC considers task variations and designs models to accurately estimate queuing delays. Specifically, this is achieved by capturing the characteristics of varying pipeline task traffic. These characteristics include the differences in the amount of data transferred between workers (Δx = x). now -x plan ) and the difference in the average amount of data transferred among all workers (Δxa=xa) now -xa plan Once the DNN structure and pipeline planning scheme are given, the above features can be calculated directly, and if no pipeline is currently running in the equipment group, then x now and xa nowIt should be zero. This takes into account the characteristics of these pipeline flow differences and the queuing delays between current workers. T q It can be defined as:

[0063]

[0064] Here, w* and ε* are the parameters to be fitted and are related to the communication efficiency between workers.

[0065] Based on the above delays, the estimation of communication delay (latency) in planning pipeline tasks is as follows:

[0066]

[0067] TDC requires a warm-up phase for calibration during device group initialization or when the communication topology changes. The calibration requires collecting relevant network traffic data and end-to-end communication traffic between Workers to construct a sample set, i.e.: {key: (x plan ,b,Δx,Δxa), value: delay}. These data pairs make It can obtain relatively accurate parameters. In addition, the model can perform continuous data collection and further parameter calibration during pipeline operation.

[0068] 3) Runtime evaluation

[0069] For a given task and device group, the complete UCG can be obtained based on the above computation and communication model, including the computational latency of each layer and the communication matrix between devices, which is composed of the above communication computation functions. Therefore, the runtime computation and communication latency of executing arbitrary slices on any device subgroup can be estimated through table lookup and function computation, thereby supporting decisions in parallel planning strategies. Furthermore, considering memory constraints, not all slices are feasible on some devices. Therefore, this embodiment also estimates the memory requirements for executing slices based on fine-grained information, considering weight tensors, input / output tensors, temporary tensors, and resident buffers to prevent memory shortages.

[0070] Furthermore, in step S2, EPipe introduces a multi-pipeline parallel planner to generate multi-pipeline parallel strategies. Existing strategies typically provide a single-pipeline parallel scheme for each task, which may not fully utilize all available devices due to resource constraints. Multi-pipeline parallel schemes can construct multiple different independent pipelines within a device group to improve inference efficiency and device utilization. Therefore, MPP broadens the pipeline solution space and designs a dynamic programming (DP)-based genetic algorithm to search for the optimal solution.

[0071] For a multi-pipeline parallel strategy, its solution space [pipe1, pipe2, ...] contains all the numbers and types of pipelines (including various pipeline partitions on arbitrary device subgroups). Due to the complexity of its solution, the maximum number of available pipelines is... Restrict the solution space, where ∑ i∈D m i The total memory of all devices is represented by , and P represents the size of the DNN. Furthermore, MPP employs a single pipeline partitioning generator to generate the optimal partitioning of a single pipeline within a specified device subgroup (i.e., all valuable pipeline types), thereby further simplifying the solution space.

[0072] Based on this, the multi-pipeline partitioning solver in MPP can generate better multi-pipeline partitioning schemes.

[0073] 1) Single-flow partition generator.

[0074] This generator uses a dynamic programming algorithm to generate the optimal single pipeline partitioning strategy for any group of devices.

[0075] Specifically, in single-pipeline inference, the slowest stage directly limits the pipeline's performance. Therefore, the subproblem of dynamic programming can be defined as finding the time of the first i slowest pipeline segments running the model on a selected subgroup of devices S (S∈D), denoted as H(i,S,d). k ), where d k The next device to consider. For a DNN with L layers, the optimal partitioning on any given device subgroup S is determined by the time H(i,S,d) of the slowest pipeline segment. k )Sure.

[0076] Due to the asynchronous execution of device computation and communication, the runtime of one stage of the pipeline can be formalized as:

[0077] T st (i,j,d k ,d l ,P j )=max{T cp (i,j,d k ),T cm (d k ,d l ,X j )}

[0078] Among them, T cp (i,j,d k ) represents the i-th to j-th layers of the model in device d. k The computation time, T cm (d k ,d l ,X j) indicates intermediate output X j From device d k To device d l The communication time. Therefore, the state transition equation of dynamic programming can be formally expressed as:

[0079] H(j,S∪d k ,d l )=minmax{(H(i,S,d k ),T st (i,j,d k ,d l ,X j )}.

[0080] Due to constraint T st For any given device subgroup S, by H(i,S,d) k The optimal partitioning strategy determined by the set of devices used. It is not necessarily U=S. This means that the number of candidate pipeline types generated is M≤2. |D| In dynamic programming, this embodiment introduces a pipeline type set (PTS) to store all valuable pipeline types, denoted as PTS = {(U i H i ,R i ) | i∈[1,M]}, where i is the index of each pipeline, U i pipe i The set of devices actually used, H i Indicates the corresponding R i This specifies the particular partitioning strategy, which maps DNN slices to corresponding devices. The PTS is the simplified solution space. The specific algorithm process is shown in the following steps.

[0081]

[0082] When homogeneous devices with identical computing and communication capabilities exist, the computational complexity can be expressed as: Where N represents the total number of device categories, and each category i has n i One device.

[0083] 2) Multi-pipeline partitioning solver.

[0084] After obtaining the simplified solution space, the solver uses a genetic algorithm (GA) to generate a multi-pipeline partitioning strategy.

[0085] In this embodiment of the genetic algorithm, each multi-pipeline partitioning strategy is treated as a chromosome, and multiple such chromosomes form a population. Each chromosome consists of S... max Each genome composition indicates that this multi-pipeline partitioning strategy contains at most S max Streamline. Each gene selects one pipeline from the PTS, formalized as g. k =i(k∈[1,S) max ], i∈[0,M]), where, where, g k Let represent the k-th pipeline sample, i represent the selection of the ith valuable pipeline type, M = 0 indicates that no pipeline type corresponding to this gene exists, and i = 0 indicates that no pipeline is selected for this gene. This ensures that the search space in this embodiment contains less than S pipeline segments. max The fitness of each chromosome C is defined as follows:

[0086] To improve the performance lower bound of the multi-pipeline partitioning strategy, the initial population includes one chromosome. The chromosome that represents the best performance for a single pipeline partitioning of the representative equipment group is used, while other chromosomes are randomly initialized. In each generation, the algorithm uses a roulette wheel strategy to select two chromosomes, then performs crossover and mutation to form two new chromosomes, repeating this process until a new population is generated. After multiple generations of iteration, the chromosome with the highest fitness in the entire genetic algorithm process is selected as the optimal partitioning strategy, as shown in the steps of Algorithm 2 below:

[0087]

[0088] 3. Deployment and Implementation Methods

[0089] In this embodiment, EPipe uses PyTorch's Gloo as the communication backend. The Master is responsible for planning and distributing the complete scheduling strategy to each worker involved in each pipeline, with each pipeline managed independently by a different thread. All worker processes receive the parallel scheme for their respective pipelines. To achieve offline planning in EPipe, the Master uses `isend()` to asynchronously dispatch instructions, while the worker maintains a task queue to asynchronously receive instructions from `irecv()`. Task sequence numbers are encoded as a global logical clock to ensure the consistency of task order.

[0090] Each worker is a separate process, containing one computation thread for task inference and two communication threads for managing input and output queues, respectively. Based on this, EPipe uses the send() and recv() functions to implement synchronous communication between different stages of a single pipeline, ensuring data consistency and ordered processing within the pipeline stages.

[0091] This embodiment provides a pipelined inference method based on heterogeneous edge scenarios, which supports high-quality offline parallel planning for heterogeneous edge devices. Specifically, 1) this method abstracts the cross-device compatibility of DNNs into a UCG and designs a Task-Device Cooperative Analyzer (TDC) to obtain accurate latency estimates. 2) this method designs a genetic algorithm (MPP) based on Dynamic Programming (DP) to expand the solution space of traditional single-pipeline parallelism, realizing multi-pipeline parallelism. Furthermore, comprehensive experimental evaluations conducted on an edge test platform show that EPipe can significantly improve the inference task throughput of individual tasks and task flows.

[0092] Example 1: Real-time video analytics in intelligent transportation systems

[0093] In intelligent transportation systems, real-time video analytics tasks place extremely high demands on real-time data processing. The EPipe framework, through its multi-pipeline parallel inference capabilities, can effectively improve data processing speed and accuracy. The specific implementation steps are as follows:

[0094] 1) System Architecture Design: First, define an intelligent traffic monitoring system containing multiple cameras. The video data stream from each camera is sent to the EPipe framework for processing. These data streams include different tasks such as vehicle recognition and traffic flow monitoring.

[0095] 2) Task and device allocation: The master node in the EPipe framework will use TDC to predict the execution latency of each task on different devices based on the complexity of the task, the data traffic, and the processing capabilities of each edge device, and formulate a multi-pipeline parallel strategy through MPP (Multi-pipeline Parallel Planner).

[0096] 3) Parallel Inference Implementation: Based on the MPP planning results, tasks are assigned to different pipelines, each potentially consisting of different devices, such as NVIDIA's Jetson series edge computing devices. The system optimizes the execution order of tasks within the pipelines, reducing data transmission and processing latency.

[0097] 4) Dynamic Adjustment and Optimization: EPipe can dynamically adjust its parallel strategy based on real-time feedback. For example, during peak traffic hours, the pipeline for processing traffic flow analysis can be increased, while during periods of lower traffic flow, the resource allocation for these tasks can be reduced, optimizing the overall system performance and energy consumption.

[0098] 5) Optimization of switching overhead: When it is necessary to switch tasks or adjust the parallel strategy, EPipe uses offline planning and prediction models to pre-calculate the optimal task switching strategy, which significantly reduces the latency caused by real-time calculation.

[0099] Example 2: Real-time quality inspection in intelligent manufacturing

[0100] In the field of smart manufacturing, the EPipe framework can be applied to real-time quality inspection on production lines to ensure product quality meets standards. Implementation steps include:

[0101] 1) Production line integration: Integrates multiple testing stations, each equipped with a high-speed camera and sensor to collect product images and data on the production line in real time.

[0102] 2) Data flow analysis and task allocation: The master node uses TDC to evaluate the execution time of each inspection task (such as size measurement, defect detection, etc.) on different devices, and uses MPP to generate a multi-pipeline parallel strategy to reasonably allocate tasks to each edge computing device.

[0103] 3) High-efficiency parallel processing: According to EPipe's plan, different detection tasks are processed in parallel in different pipelines. Each pipeline can run independently on a specific device, such as NVIDIA Jetson AGX Xavier or Jetson Nano.

[0104] 4) Real-time adjustment and feedback mechanism: The system can adjust its parallel processing strategy in real time according to changes in the product line to optimize resource utilization. By collecting processing results and performance data, the model and processing flow can be further optimized.

[0105] 5) Reduce switching overhead: When switching between different types of quality inspection tasks, EPipe minimizes task switching time and reduces downtime and overhead caused by switching through pre-planning and scheduling.

[0106] Example 3: Inference of Fast Propagation Fading Parameters in Smart Channel Models

[0107] In the field of wireless communication simulation, the EPipe framework can be applied to real-time inference of intelligent channel model parameters to improve the speed of inference and thus the speed of wireless communication channel simulation. Implementation steps include:

[0108] 1) System Architecture Design: First, a communication scenario is defined, comprising multiple wireless communication devices. Each device corresponds to specific geographical location information and operational status information (moving speed, antenna orientation, etc.), which is represented in the form of a state matrix. As the wireless communication devices move, the state matrices of multiple wireless devices at each moment are sent to the EPipe framework for processing, forming multiple propagation fading parameter inference tasks. The output of each task includes large-scale propagation fading values, etc.

[0109] 2) Data flow analysis and task allocation: The master node uses TDC to evaluate the execution time of each propagation fading parameter inference task on different devices, and uses MPP to generate a multi-pipeline parallel strategy to reasonably allocate propagation fading parameter inference tasks to each edge computing device.

[0110] 3) High-efficiency parallel processing: According to EPipe's plan, different propagation and fading parameter inference tasks are processed in parallel in different pipelines. Each pipeline can run independently on a specific device, such as NVIDIA Jetson ORIN NX or NVIDIA Jetson Nano.

[0111] 4) Adjustment and Feedback Mechanism: EPipe adjusts its parallel processing strategy in real time based on changes in device and propagation fading parameters during inference tasks, optimizing resource utilization. By collecting processing results and performance data, it further optimizes the model and processing flow.

[0112] This invention also provides a pipelined inference device based on heterogeneous edge scenarios, comprising:

[0113] The task-device co-processing analyzer predicts pipeline computation and communication latency based on information from a given task and device group, and abstracts the adaptability of deep neural networks across device groups as an undirected connected graph.

[0114] A multi-pipeline parallel planner is used to generate a set of types containing all potential optimal single-pipeline schemes based on the unconnected graph, and to generate multi-pipeline parallel strategies using a genetic algorithm.

[0115] The inference task allocation module is used to allocate tasks to devices in different pipelines for inference according to the multi-pipeline parallel strategy.

[0116] Wherein, each node of the undirected connected graph represents the inference latency of any segment of the deep neural network across various devices;

[0117] An edge in an undirected connected graph corresponds to a communication matrix, which represents the communication latency between devices.

[0118] The Task-Device Collaboration Analyzer is specifically used for:

[0119] A prediction model based on a self-attention mechanism is adopted, in which fine-grained task features are input into the encoder, data-driven device features are embedded into the decoder, and then the outputs of the decoder and encoder are cross-attention mechanism and connected to a multilayer perceptron to generate computational latency.

[0120] The fine-grained task features include multiple embedding vectors, each representing a separate layer. Each separate layer consists of three components: layer type information, layer features, and position encoding.

[0121] The estimation function for the communication delay is:

[0122]

[0123] Among them, T ende For data encoding and decoding latency, T ende =w1×x plan +ε1,x plan For the efficiency of encoding and decoding operations on the device's CPU;

[0124] T n To handle latency for nodes, T p For the delay of propagation, T n +T p =ε2;

[0125] T t For transmission delay,

[0126] T q Due to queuing delays,

[0127] Optionally, the dynamic programming algorithm uses the maximum number of pipelines to limit the solution space of the multi-pipeline parallel strategy;

[0128] The multi-pipeline parallel planner is also designed with a single-pipeline partitioning generator, which is used to generate the optimal partitioning PTS for a single pipeline within a specified device subgroup.

[0129] Specifically, genetic algorithms are used to generate multi-pipeline parallel strategies, including:

[0130] In genetic algorithms, each multi-pipeline partitioning strategy is considered as a chromosome, and multiple chromosomes form a population. Each chromosome is composed of S... max Gene composition;

[0131] Each gene selects a pipeline from the PTS, formalized as g k =i(k∈[1,S) max ], i∈[0,M]);

[0132] The initial population consists of one chromosome. In each generation, the genetic algorithm uses a roulette wheel strategy to select two chromosomes from the population for crossover and mutation to form two new chromosomes. This process is repeated until a new population is generated. After multiple generations of iteration, the chromosome with the highest fitness throughout the entire genetic algorithm process is selected as the optimal partitioning strategy.

[0133] The pipelined inference device based on heterogeneous edge scenarios provided in this embodiment of the invention can execute the pipelined inference method based on heterogeneous edge scenarios provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method, which will not be described in detail here.

[0134] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A pipelined inference method based on heterogeneous edge scenarios, characterized in that, Executed by the master node, including: Based on information about a given task and device group, the computational and communication latency of the pipeline is predicted by a task-device co-analysis analyzer, and the adaptability of the deep neural network across device groups is abstracted as an undirected connected graph; the communication latency includes: data encoding and decoding latency, node processing latency, propagation latency, transmission latency, and queuing latency; Based on the undirected connected graph, a type set containing all potential optimal single pipeline schemes is generated by a multi-pipeline parallel planner, and a multi-pipeline parallel strategy is generated using a genetic algorithm. The task is assigned to devices in different pipelines for inference according to the multi-pipeline parallel strategy. The multi-pipeline parallel planner uses a dynamic programming algorithm to generate a type set containing all potential optimal single-pipeline solutions. The dynamic programming algorithm uses the maximum number of pipelines to limit the solution space of the multi-pipeline parallel strategy. The multi-pipeline parallel planner is also designed with a single-pipeline partitioning generator, which is used to generate the optimal partitioning of a single pipeline within a specified device subgroup. Genetic algorithms are used to generate multi-pipeline parallel strategies, including: In genetic algorithms, each multi-pipeline partitioning strategy is considered a chromosome, and multiple chromosomes form a population. Each chromosome consists of... Gene composition; Each gene selects a pipeline from 𝑃𝑇𝑆, formalized as ,in, Let represent the k-th pipeline sample defined, and i represent the selection of the ith valuable pipeline type. This indicates that the pipeline type corresponding to this gene does not exist; The initial population consists of one chromosome. In each generation, the genetic algorithm uses a roulette wheel strategy to select two chromosomes from the population for crossover and mutation to form two new chromosomes. This process is repeated until a new population is generated. After multiple generations of iteration, the chromosome with the highest fitness throughout the entire genetic algorithm process is selected as the optimal partitioning strategy.

2. The method according to claim 1, characterized in that, Each node of the undirected connected graph represents the inference latency of any segment of the deep neural network across various devices. An edge in an undirected connected graph corresponds to a communication matrix, which represents the communication delay between devices.

3. The method according to claim 1, characterized in that, Predict pipeline computation latency using a task-device co-analyzer, including: A prediction model based on a self-attention mechanism is adopted, in which fine-grained task features are input into the encoder, data-driven device features are embedded into the decoder, and then the outputs of the decoder and encoder are cross-attention mechanism and connected to a multilayer perceptron to generate computational latency.

4. The method according to claim 3, characterized in that, The fine-grained task features include multiple embedding vectors, each representing a separate layer. Each separate layer consists of three components: layer type information, layer features, and position encoding.

5. The method according to claim 1, characterized in that, The estimation function for the communication delay is: in, For data encoding and decoding latency, , For the efficiency of encoding and decoding operations on the device's CPU; To handle latency for nodes, To delay the spread, ; For transmission delay, b is the bandwidth; Due to queuing delays, , Queuing delay between current devices, Due to differences in the amount of data transmitted between devices, The difference in the average amount of data transmitted between all devices.

6. The method according to any one of claims 1-5, characterized in that, The pipelined inference method based on heterogeneous edge scenarios uses PyTorch's Gloo as the communication backend.

7. A pipelined inference device based on heterogeneous edge scenarios, characterized in that, include: A task-device collaboration analyzer is used to predict the computational and communication latency of a pipeline based on information about a given task and device group, and to abstract the adaptability of a deep neural network across device groups as an undirected connected graph; the communication latency includes: data encoding and decoding latency, node processing latency, propagation latency, transmission latency, and queuing latency. A multi-pipeline parallel planner is used to generate a set of types containing all potential optimal single-pipeline schemes based on the undirected connected graph, and to generate multi-pipeline parallel strategies using a genetic algorithm. The inference task allocation module is used to allocate tasks to devices in different pipelines for inference according to the multi-pipeline parallel strategy. The multi-pipeline parallel planner uses a dynamic programming algorithm to generate a type set containing all potential optimal single-pipeline solutions. The dynamic programming algorithm uses the maximum number of pipelines to limit the solution space of the multi-pipeline parallel strategy. The multi-pipeline parallel planner is also designed with a single-pipeline partitioning generator, which is used to generate the optimal partitioning of a single pipeline within a specified device subgroup. Genetic algorithms are used to generate multi-pipeline parallel strategies, including: In genetic algorithms, each multi-pipeline partitioning strategy is considered a chromosome, and multiple chromosomes form a population. Each chromosome consists of... Gene composition; Each gene selects a pipeline from 𝑃𝑇𝑆, formalized as ,in, Let represent the k-th pipeline sample defined, and i represent the selection of the ith valuable pipeline type. This indicates that the pipeline type corresponding to this gene does not exist; The initial population consists of one chromosome. In each generation, the genetic algorithm uses a roulette wheel strategy to select two chromosomes from the population for crossover and mutation to form two new chromosomes. This process is repeated until a new population is generated. After multiple generations of iteration, the chromosome with the highest fitness throughout the entire genetic algorithm process is selected as the optimal partitioning strategy.

Images