A real-time video task end network edge cooperative scheduling method and device

Abstract

A real-time video task end network edge collaborative scheduling method first collects end side / edge side platform framework information, end side device energy consumption information, various deep neural network model operator information and data transmission information between operators; secondly, according to the information collection result, the topological network relationship and the weight of the edge between the operators are constructed, and the directed acyclic graph of the deep neural network model is established accordingly; thirdly, based on the data collection and the construction result of the directed acyclic graph, the decision variables, objective function and constraints of the real-time video task end network edge collaboration are defined and mathematically expressed, and then the end network edge collaborative scheduling model is constructed; finally, the mathematical representation of the decision variables, objective function and constraints is compiled and solved in the solver environment, and the optimal scheduling result is obtained and deployed in the actual physical environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a real-time video task edge-network collaborative scheduling method and apparatus. Background Technology

[0002] In recent years, with the large-scale deployment and application of intelligent robots, mobile phones, and surveillance cameras, global video data has experienced explosive growth. According to research and analysis of the video streaming market by Bages et al., the global video streaming market size reached RMB 751.754 billion in 2021, accounting for approximately 80% of total internet traffic, and is projected to reach RMB 2.323144 trillion by 2027, with a compound annual growth rate of 20.69% during the forecast period. Meanwhile, thanks to the explosive development of artificial intelligence technology, video analysis methods based on deep neural networks are constantly innovating and achieving breakthroughs in areas such as target recognition, target tracking, and anomaly detection, demonstrating significant application value in fields such as intelligent robots, intelligent security, and intelligent transportation. However, due to the massive computational demands of deep neural networks, dedicated mobile GPU hardware (such as NVIDIA Jetson AGX Orin, Orin NX, etc.) is required to achieve efficient inference on edge devices such as robot bodies, significantly increasing deployment costs and battery life pressures, thus limiting the widespread application of deep neural networks on edge devices. Therefore, effectively resolving the contradictions between edge computing power, hardware costs, and battery life is of paramount importance for reducing edge production costs, improving intelligence, and enhancing market competitiveness.

[0003] The edge-cloud collaborative video task analysis paradigm based on centralized cloud computing transmits video tasks generated on the edge to a cloud server in the network center. The powerful computing capabilities of the cloud server are then used for inference and analysis of the video tasks, and the inference results are sent back to the edge for subsequent decision-making. Related research, including the IEEE CCBD paper "Cloud Resource Management for Image and Video Analysis of Big Data from Network Cameras," has explored collaborative task scheduling and cloud resource management methods for edge-cloud video task analysis systems. However, edge-cloud collaboration based on cloud computing architecture suffers from problems such as limited network bandwidth and high transmission latency. Therefore, the research results are suitable for offline video task analysis and cannot meet the high real-time requirements of robots and other applications, thus limiting the application of edge-cloud collaboration in related fields.

[0004] Edge computing-based collaborative video task analysis paradigms can, to some extent, address the network bandwidth issues between edge devices and additional computing resources. Edge servers and edge devices such as robots are physically close, have simpler network conditions, and edge sites generally do not involve multi-tenant services, allowing for relatively stable high-bandwidth and low-latency communication between the edge and edge. Edge collaborative video task analysis primarily involves research on task scheduling, resource allocation, and algorithm acceleration. Task scheduling and resource allocation are techniques applied to collaborative task processing between the edge and edge sides. They aim to optimize end-to-end system performance by rationally offloading video tasks and effectively allocating computing resources between the edge and edge sides. For example, the ACM SenSys paper "Better Never Than Late: Timely Edge Video Analytics Over the Air" proposes a deep reinforcement learning-based edge video analytics feedback controller that adjusts video analysis process parameters such as video frame size based on the actual state. Distributed algorithm acceleration, from the perspective of the edge device layer, considers the impact of edge configuration and deployment at the model level (operators, layers) on evaluation metrics such as system latency. For example, the ACM ASPLOS paper "Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge" proposes the Neurosurgeon segmentation algorithm based on the characteristics of transmission latency, computation latency, and energy consumption at different segmentation points. Task scheduling, resource allocation, and algorithm acceleration represent different granularities and depths of consideration for the same type of problem. However, the above methods still have the following problems in real-time video analysis:

[0005] (1) Impact of network bandwidth: Due to network fluctuations caused by robot movement and the large transmission volume generated by high frame rate video, a large load will be placed on the network between the end side and the edge side. However, existing methods usually ignore the impact of the network on end-edge coordination, or simply abstract network latency as transmission delay, ignoring the changes in data transmission volume caused by different offload point selections. This will cause the end-edge coordination scheduling model to fail to truly reflect the characteristics of actual video task processing, resulting in poor end-edge coordination scheduling results;

[0006] (2) Last-to-First-Served Rule: Since robots need to complete relevant decision-making and control tasks based on the results of visual tasks such as target detection, recognition, and tracking, robot video analysis tasks have high real-time requirements. The robot end-side continuously acquires video frames and builds a video frame queue through the onboard camera to wait for analysis and processing by the end-side vision module. When there is congestion in the video frame queue, due to the real-time requirements, the vision module pays more attention to the information of the latest video frame (i.e., the latest arriving video frame) rather than the information of the earliest arriving video frame in the queue. Therefore, this queuing system follows the last-to-first-served rule, which is fundamentally different from the first-to-first-served rule in existing offline video analysis methods. This makes it difficult to apply the end-to-edge collaborative scheduling technology based on the first-to-first-served rule to the actual scenario of real-time video analysis of robots.

[0007] (3) Changes in evaluation metrics: In existing edge-to-edge collaborative scheduling methods, most use the end-to-end latency of task processing as the evaluation metric for the scheduling scheme in order to minimize the end-to-end latency as much as possible. However, due to the last-to-first-served real-time video task processing rules, edge-to-edge video analysis systems may encounter "invalid frames" when processing video tasks. As a result, the end-to-end latency of video task processing can only be calculated based on the "valid frames" processed by the vision module. Secondly, for deep neural networks with temporal characteristics such as target tracking, the visual task processing rate (i.e., the number of video frames processed per unit time) will also affect the accuracy of the visual algorithm. At the same time, edge-side energy consumption, such as computational energy consumption and transmission energy consumption, will also change due to changes in the scheduling scheme. Therefore, when evaluating edge-to-edge collaborative scheduling schemes, it is necessary to further integrate the influence of factors such as visual task processing rate and edge-side energy consumption. Summary of the Invention

[0008] To address the shortcomings of existing technologies and enable efficient processing of end-side video tasks in scenarios with limited computing power on end-side devices, this invention provides a real-time video task end-network-edge collaborative scheduling method and apparatus. This method utilizes the additional computing resources on the edge side, taking into account actual network bandwidth, to reduce end-side inference latency, increase task processing speed, and reduce end-side energy consumption.

[0009] The present invention provides a real-time video task end-to-end collaborative scheduling method, comprising the following steps:

[0010] Step S1, Data Acquisition. Acquire platform framework information, including hardware and software platforms on the edge and device sides; acquire power consumption information of edge devices, including computing power consumption, transmission power consumption, and standby power consumption; acquire operator information of various DNN models, including operator type and computation latency information; acquire data transmission information between operators, including data transmission direction and data transmission volume.

[0011] Step S2: Construction of the Directed Acyclic Graph (DAG) of the DNN model. The DAG of the DNN model consists of two types of elements: nodes V and edges E. The topological network relationship between operators is established according to the data flow direction between each operator; the weights of the edges of the DAG are assigned according to the amount of data transmitted between each operator.

[0012] Step S3: Construction of the end-to-end collaborative scheduling model. Based on the data acquisition results of Step S1 and the DAG graph construction results of Step S2, an end-to-end collaborative scheduling model is established using the integer programming modeling method. First, the decision variables of the model are defined and mathematically expressed, including end-side allocation decisions, communication decisions, and end-side sampling frame rates. Second, the optimization objectives of collaborative scheduling are defined and mathematically expressed, including video task processing latency, video task processing rate, and end-side energy consumption. Finally, various constraints are defined and mathematically expressed, including configuration constraints, non-overlapping constraints, end-side computation latency constraints, edge-side computation latency constraints, start / end time constraints, network transmission latency constraints, and end-side energy consumption constraints.

[0013] Step S4: Solve the scheduling model based on the solver. Based on the end-network-edge collaborative scheduling model construction results from Step S3, using programming languages ​​such as CLPEX and Gurobi, the mathematically represented decision variables, objective functions, and constraints are compiled in the solver environment. The DAG graph network structure of the specific DNN model is then input into the solver to obtain the optimal scheduling result (if the optimal value cannot be obtained within the set time threshold, a suboptimal solution is output).

[0014] Step S5: Actual deployment of the scheduling scheme. Based on the final solution results of the decision variables in step S4, the DNN models of the end devices and edge servers, camera sampling rates, etc., are deployed in the actual physical environment.

[0015] Furthermore, the data acquisition in step S1 includes the following steps:

[0016] Step S11: Collect platform framework information, including hardware and software platform information for both the edge and device sides. Hardware platform information includes hardware name, type (e.g., CPU, GPU, TPU, VPU), hardware processing power, memory bandwidth, etc. Software platform information includes software name (e.g., TensorRT, TFLite, Openvino, MNN, NCNN), software platform design features, and related parameters, etc.

[0017] Step S12: Collect power consumption information of the edge device, including computing power consumption, standby power consumption, and transmission power consumption. Among them, computing power consumption represents the power consumption of the edge device under DNN model inference conditions, standby power consumption represents the power consumption of the edge device when not performing DNN model inference, and transmission power consumption represents the power consumption generated when the edge device transmits data to the edge server.

[0018] Step S13: Collect operator information for various deep neural network (DNN) models, including operator type and computation latency. In DNN models, operators can be considered the most basic computational units, and larger computational units—layers—are composed of operators of the same or different types. DNN models such as ResNet and YOLO v3 typically include convolutional layers, pooling layers, and activation function layers. Convolutional layers include convolution operators, depthwise separable convolution operators, and dilated convolution operators; pooling layers include max pooling operators, min pooling operators, and average pooling operators; and activation function layers include ReLU operators, ReLU6 operators, and sigmoid operators.

[0019] Step S14: Collect data transmission information between operators, including data transmission direction and data transmission volume. In the DNN model, data transmission occurs between operators. For example... Figure 1 As shown, in this network, the computation result of the first add operator is input to the ReLU operator and the conv1 operator, in the direction of add operator → ReLU operator and add operator → conv1 operator. Meanwhile, the amount of data output from the same operator is fixed, and the input and output data sizes of each operator are recorded.

[0020] Furthermore, the construction of the DAG graph of the DNN model in step S2 includes the following steps:

[0021] Step S21: Establish the topological network relationship between operators. The DAG graph of the DNN model is represented as G = (V, E), where node V represents an operator, and edge E represents the data transmission relationship between operators. As shown in Figure 2, there are three types of connection relationships between nodes V: direct connection, multiple-output connection, and multiple-input connection. Based on the logical analysis results between nodes V and edges E, the topological network relationship between operators is constructed.

[0022] Step S22: Assign weights to the edges of the DAG graph. Based on the data transmission volume determined in step S14, assign weights to the edges of the DAG graph topology network in the DNN model. The weight value is equal to the data transmission volume between the two operators.

[0023] Furthermore, the construction of the end-to-end collaborative scheduling model in step S3 includes the following steps:

[0024] Step S31: Define and describe the decision variables of the end-network-edge collaborative scheduling model. In end-network-edge collaborative scheduling, it is necessary to decide the placement location of the operator, the data transmission path of the operator, the sampling frame rate of the end device, and the start and end times of the operator, thereby generating the allocation decision x. in Communication Decision i End-side sampling frame rate decision f end Start Time Decision S i and the decision of the end time C i Among them, the end-side allocation decision x in The variable is 0 / 1, defined as whether operator i is placed at device n (1 for yes, 0 for no); communication decision f i The variable is 0 / 1, defined as whether operator i has cross-device data transmission (1 for yes, 0 for no); the end-side sampling frame rate decision f end The variable is an integer, defined as the frame rate at which the video frame is sampled at the end; the start time decision S i and the decision of the end time C i This is an integer variable, defined as the start and end times of the operator.

[0025] Step S32: Define and describe the optimization objectives of the end-network-edge collaborative scheduling model. The optimization objectives include video task processing latency, video task processing rate, and end-side energy consumption. Specifically, the video task processing latency T... total The processing latency of only "valid frames" can be further expressed as the end-side computation latency T. end Transmission delay T net And edge side calculation delay T edge The sum is shown in equation (1); the video task processing rate is determined by the processing rate bottlenecks of the end, network, and edge, and the processing rates of the end, network, and edge are μ. end μ net μ edge The relationship between the computation delay and the terminal energy consumption E is expressed by equation (2); end It includes computing power consumption, standby power consumption and communication power consumption; the optimization objective of scheduling is ultimately expressed as the weighted sum of end-side computing latency, video task processing rate and end-side power consumption, as shown in equation (3).

[0026] T total =Te nd+ T net +T edge (1)

[0027] μ end =1 / T end μ net =1 / T net μ edge =1 / T edge (2)

[0028]

[0029] Step S33: Define and describe the constraints of the end-network-edge collaborative scheduling model. The scheduling constraints include configuration constraints, non-overlapping constraints, end-side computation delay constraints, edge-side computation delay constraints, start / end time constraints, network transmission delay constraints, and end-side energy consumption constraints. The specific descriptions and mathematical expressions of each constraint are as follows, and the meanings of the parameters and variables involved are shown in the table below:

[0030]

[0031] Configuration constraints: Includes constraints (4-5). Constraint 4 states that any operator i can only be deployed on one device; since data transmission between endpoints and edges during the inference process is unidirectional, constraint 5 states that any subsequent operator of operator i... When placed on the end side, operator i can only be placed on the end side.

[0032]

[0033]

[0034] Non-overlapping constraint: When two operators are deployed on the same device, the device performs calculations serially, therefore their calculation times cannot overlap. Specific constraints are shown in constraint (6-7). Where M... s and M l These are two large numbers, namely M. s >>0, M l >>0.

[0035]

[0036]

[0037] End-side computation delay constraint: The computation delay of the end-side device is the sum of the computation delays of all operators deployed on the end-side, as expressed by equation (8).

[0038]

[0039] Transmission delay constraint: Transmission delay is expressed as the ratio of bandwidth to transmitted data volume, as denoted by constraint (9); constraint (10) represents the relationship between decision variables, which is only true when operator i is deployed on the end side and there exists a direct successor operator j deployed on the edge side (i.e., x i0 =1,x j0 =0, (i,j)∈E), only then is it necessary to transmit the output of operator i from the end side to the edge side (i.e., f i =1), other cases fi =0.

[0040]

[0041]

[0042] Edge-side computation delay constraint: The edge-side computation delay is expressed as the effective frame end-to-end delay minus the end-side computation delay and the end / edge-side transmission delay, as shown in constraint (11).

[0043]

[0044] Start / End Time Constraints: The start / end time constraints are shown in constraints (12-14). Wherein, constraint (12) represents any subsequent operator of operator i. The start time of operator i cannot be earlier than the end time of operator i; constraint (13) means that if there is data transmission between operator i and operator j, then any operator The start time cannot be earlier than the transmission completion time; constraint (14) represents the relationship between the start time, end time and computation time of the operator.

[0045]

[0046]

[0047]

[0048] End-side power consumption constraints: End-side power consumption includes three parts: processing power consumption, standby power consumption, and transmission power consumption. Processing power consumption represents the power consumption of the end-side device when processing video frames, which can be further divided into effective frame processing power consumption E. C，1 Energy consumption E for invalid frame processing C，2 The standby power consumption E is represented by constraint (15-16). I This represents the power consumption required for normal standby when the end-side device is not processing video frames, as shown in constraint (17); transmission power consumption E T The energy consumption of the end device when transmitting data to the edge server is represented by constraint (18); the total end-side energy consumption is represented as the sum of the three types of energy consumption, as shown in constraint (19).

[0049]

[0050] E C，2 =e com ·max{0,max{T end T net T edge}-1 / f end} (16)

[0051]

[0052] E T =e net ·T net (18)

[0053] E end =E P，1 +E P，2 +E I +E T (19) Furthermore, the solver-based scheduling model solution in step S4 includes the following steps:

[0054] Step S41: Based on solvers such as CLPEX and Gurobi, compile the decision variables, objective function, and constraints of the edge-network collaborative scheduling model.

[0055] Step S42: Input the relevant parameters of the DNN model into the solver, including the DAG graph network, operator end-side / edge-side processing latency, operator output data volume, network bandwidth, etc.

[0056] Step S43: Run the solver and output the final results. Solver parameters need to be set, including TimeLimit and MIPFocus. Finally, output the solution results.

[0057] Furthermore, the actual deployment of the scheduling scheme in step S5 includes the following steps:

[0058] Step S51, transformation of decision variables into implementation schemes. Based on the allocation decision x on the remote side in step S4... in Communication Decision i and end-side sampling frame rate decision f end The final solution result will determine the end-side allocation decision x. in Chinese x i0 The operators corresponding to 1 need to be deployed on the edge device, while the remaining operators need to be deployed on the corresponding edge servers; in addition, the edge sampling frame rate decision f end The value indicates the video sampling frame rate that the end-side device needs to set.

[0059] Step S52: Implement the actual deployment of the solution. Based on the implementation plan in S51, deploy the DNN model, camera sampling rate, etc., of the end-side devices and edge-side servers in the actual physical environment.

[0060] A real-time video task end-to-end collaborative scheduling device includes: a data acquisition module, a DAG graph construction module, an end-to-end collaborative scheduling model construction module, and a model solving module connected in sequence.

[0061] The data acquisition module collects hardware / software platform framework information from the end side and edge side; collects end-side device energy consumption information such as computing energy consumption, transmission energy consumption, and standby energy consumption; collects operator information of the DNN model such as operator type and computing latency; and collects data transmission information between operators such as data transmission direction and data transmission volume.

[0062] The DAG graph construction module constructs the topological network relationship between operators based on the logical arrangement results between nodes V and edges E. It assigns edge weights to the DAG graph topological network of the DNN model according to the amount of data transmitted between operators, with the weight value being the amount of data transmitted between the two operators.

[0063] The endpoint-edge collaborative scheduling model construction module, based on the results of the data acquisition module and the DAG graph construction module, first performs endpoint allocation decisions x. in Communication Decision i and end-side sampling frame rate f end The decision variables of the model are defined and mathematically expressed; secondly, the weighted sum of video task processing latency, video task processing rate, and edge power consumption is used as the objective function of the scheduling model, i.e. Finally, various constraints are defined and mathematically expressed, including configuration constraints, non-overlapping constraints, end-side computation delay constraints, edge-side computation delay constraints, start / end time constraints, network transmission delay constraints, and end-side energy consumption constraints.

[0064] The model solving module, based on solvers such as CLPEX and Gurobi, compiles the decision variables, objective function, and constraints of the edge-network collaborative scheduling model; inputs relevant parameters of the DNN model into the solver, including the DAG graph network, operator edge / end processing latency, operator output data volume, network bandwidth, etc.; sets solver-related parameters such as TimeLimit and MIPFocus, runs the solver, and outputs the final running results.

[0065] The advantages and beneficial effects of this invention are as follows:

[0066] To address the shortcomings of existing video task analysis and processing technologies, which often adhere to the traditional first-come-first-served (FFS) rule and fail to meet the last-come-first-served (LIFO) characteristics of real-time video tasks, neglecting data transmission variations caused by different segmentation point choices in problem assumptions and ignoring the impact of visual task processing rate on temporal characteristics in model construction, leading to suboptimal scheduling schemes, this invention integrates network bandwidth into the real-time video task processing framework. It expands the traditional end-edge scheduling framework to an end-network-edge scheduling framework, distinguishes between "invalid frames" and "valid frames" based on the LIFO rule, and proposes an end-network-edge collaborative scheduling model construction and solution method with end-side computation latency, video task processing rate, and end-side energy consumption as optimization indicators. This significantly improves the applicability of the scheduling scheme in real-time video task processing scenarios with high requirements, such as target detection and tracking. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the data transmission direction of the DNN model in this invention.

[0068] Figure 2a This is a schematic diagram of the direct connection topology between operators in this invention.

[0069] Figure 2b This is a schematic diagram of the multi-output connection topology between operators in this invention.

[0070] Figure 2c This is a schematic diagram of the multi-input connection topology between operators in this invention.

[0071] Figure 3 This is a flowchart of the method of the present invention.

[0072] Figure 4 This is a flowchart illustrating the data acquisition and DAG graph construction process in an embodiment of the present invention.

[0073] Figure 5 This is a schematic diagram of the YOLO v3 residual block network structure in an embodiment of the present invention.

[0074] Figure 6 This is a flowchart illustrating the operation of the end-to-end collaborative scheduling model in this embodiment of the invention.

[0075] Figure 7 This is a flowchart illustrating the operation of the scheduling model solution and actual deployment in this embodiment of the invention.

[0076] Figure 8 This is a schematic diagram of the device structure of the present invention. Detailed Implementation

[0077] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0078] An edge-network collaborative scheduling method for real-time video tasks. When edge devices such as robots perform real-time video tasks such as target detection and tracking, how can they efficiently run DNN models under limited edge computing resources to achieve predetermined video processing latency and speed targets? Existing technologies typically employ edge computing-based edge-network collaborative scheduling frameworks. By deploying additional computing resources at the edge, tasks are offloaded to the edge for processing, reducing edge computing pressure and improving overall end-to-end performance. However, current edge-network collaborative scheduling methods follow a first-come, first-served rule for task processing, which does not align with the last-come, first-served characteristic of real-time video tasks. Furthermore, they fail to consider the changes in data transmission volume caused by different offloading points and ignore the impact of visual task processing speed on temporal characteristics. This leads to problems such as scheduling models that do not conform to actual scene characteristics and poor deployment results. Therefore, this invention integrates the network into the real-time video task processing framework, expanding the traditional end-edge scheduling framework to an end-network-edge scheduling framework. At the same time, it fully integrates the characteristics of the last-come-first-served rule and the comprehensive optimization requirements of indicators such as computation latency, processing rate, and end-side energy consumption during the modeling process, thereby greatly improving the rationality of the scheduling model and enhancing the applicability of the scheduling scheme in real-time video task processing scenarios.

[0079] Specifically, such as Figure 3 As shown, the method includes the following steps:

[0080] Step S1, Data Acquisition. Acquire platform framework information, including hardware and software platforms on the edge and device sides; acquire power consumption information for edge devices, including computing power consumption, transmission power consumption, and standby power consumption; acquire hierarchical operator information for various deep neural network (DNN) models, determine multiple hierarchical types based on the characteristics of the operator information, and classify the operator information into each hierarchical type; acquire data transmission information between operators, including data transmission direction and data transmission volume. This information can be printed out using a computer.

[0081] In this embodiment, the processes of steps S1 and S2 are as follows: Figure 4 As shown. Step S1, data acquisition, includes the following steps:

[0082] Step S11: Collect platform framework information, including hardware and software platforms on both the edge and client sides. Hardware platform information includes hardware name, type (e.g., CPU, GPU, TPU, VPU), hardware processing power, memory bandwidth, etc. Software platform information includes software name (e.g., TensorRT, TFLite, Openvino, MNN, NCNN), software platform design features, and related parameters. See the table below for details.

[0083]

[0084]

[0085] Step S12: Collect power consumption information of the edge device, including computing power consumption, standby power consumption, and transmission power consumption. Among them, computing power consumption represents the power consumption of the edge device under DNN model inference conditions, standby power consumption represents the power consumption of the edge device when not performing DNN model inference, and transmission power consumption represents the power consumption generated when the edge device transmits data to the edge server.

[0086] Step S13: Collect operator information for various deep neural network (DNN) models, including operator type and computation latency. YOLO v3 is a classic object detection algorithm, so this example uses YOLO v3 as an example, which includes convolution operators, average pooling operators, SILU operators, fully connected operators, batch normalization operators, etc.

[0087] Step S14: Collect data transmission information between operators, including analyzing the data transmission direction and recording the input and output data volume of each operator.

[0088] Step S2: Constructing the DAG graph of the DNN model. Establish the topological network relationship between operators based on the data flow direction; assign weights to the edges of the DAG graph based on the amount of data transmitted between operators.

[0089] In this embodiment, the DAG graph construction of the S2DNN model includes the following steps:

[0090] Step S21: Establish the topological network relationship between operators. The DAG graph contains duotuples of nodes V and edges E, representing the operators and the data transmission relationships between operators, respectively. Based on the logical arrangement results between nodes V and edges E in YOLO v3, construct the DAG graph of the topological network relationship between operators. Due to space limitations, only the network structure of the residual block in YOLO v3 is shown, such as... Figure 5 As can be seen, the convolution operator and batch normalization operator in the residual block are directly connected, which is represented as convolution operator → batch normalization operator.

[0091] Step S22: Assign weights to the edges of the DAG graph. Based on the data transmission volume determined in step S14, assign weights to the edges of the DAG graph topology network in the DNN model. The weight value is equal to the data transmission volume between the two operators.

[0092] Step S3: Construction of the end-to-end collaborative scheduling model. Based on the data acquisition results of Step S1 and the DAG graph construction results of Step S2, the decision variables of the model are first defined, then the optimization objective of collaborative scheduling is defined and mathematically expressed, and finally various constraints such as configuration constraints, non-overlapping constraints, end-side computation delay constraints, edge-side computation delay constraints, and start / end time constraints are defined and mathematically expressed.

[0093] In this embodiment, as Figure 6 As shown, step S3, the construction of the edge-end collaborative scheduling model, includes the following steps:

[0094] S31 defines and describes the decision variables of the end-to-end collaborative scheduling model. End-to-end collaborative scheduling includes the end-side allocation decision x. in Communication Decision i End-side sampling frame rate decision f end Start Time Decision S i and the decision of the end time C i The variables are defined as shown in the table below.

[0095] variable meaning <![CDATA[x in ]]> End-side allocation decision, 0 / 1 variables. Operator i is 1 if placed on device n, otherwise 0; <![CDATA[f i ]]> Communication decisions are handled by 0 / 1 variables. Operator i is 1 if data is being transmitted across the endpoint device, and 0 otherwise. <![CDATA[f end ]]> End-sampling frame rate decision, an integer variable. The frame rate at which video frames are sampled at the end; <![CDATA[S i ]]> The start time is determined by an integer variable. It represents the start time of operator i. <![CDATA[C i ]]> The end time decision is an integer variable. Let be the end time of operator i.

[0096] Step S32: Define and describe the optimization objective of the end-network-edge collaborative scheduling model. Based on the following formula, construct the optimization objective for the end-network-edge collaborative scheduling problem, which is a weighted sum of video task processing latency, video task processing rate, and end-side energy consumption. Note that since processing latency, processing rate, and end-side energy consumption are measured in different ways, the weighting coefficients need to be dynamically adjusted by the decision-maker based on historical experience and the deployment effects of the scheduling scheme.

[0097]

[0098] Step S33: Define and describe the constraints of the end-network-edge collaborative scheduling model. Based on the following constraints, establish the model's configuration constraints, non-overlapping constraints, end-side computation delay constraints, edge-side computation delay constraints, start / end time constraints, network transmission delay constraints, and end-side energy consumption constraints. Specifically, by traversing the immediate successors of each operator and utilizing the connection relationships between operators, the set of indirect successors Succ(i) for any operator i can be constructed. The constraint definition results are shown in the table below:

[0099]

[0100]

[0101] Step S4: Solve the scheduling model based on the solver. The decision variables, objective function, and constraints of the above scheduling problem model are compiled using the programming languages ​​of solvers such as CLPEX and Gurobi. The DAG network structure of YOLO v3, operator delays, and other parameters are input into the solver. The solver is then run to obtain the final scheduling result.

[0102] In this embodiment, the processes of steps S4 and S5 are as follows: Figure 7 As shown. Step S4, which solves the problem based on the solver's scheduling model, includes the following steps:

[0103] Step S41: Based on the compilation languages ​​of solvers such as CLPEX and Gurobi in the Python 3.9 environment, compile the decision variables, objective function and constraints of the mid-end network edge collaborative scheduling model in S3.

[0104] Step S42: Input the relevant parameters of the YOLO v3 model into the solver, including the DAG graph network, operator end / edge processing delay, operator output data volume, etc.

[0105] Step S43: Run the solver and output the final results. Before running the solver, relevant parameters need to be set, including TimeLimit and MIPFocus. MIPFocus is determined by trying different solution speeds under parameters 0-3; this patent sets it to 1. TimeLimit is set to 7200 seconds. Finally, the end-side allocation decision x is... in Communication Decision i and end-side sampling frame rate decision f end The solution results are output.

[0106] Step S5: Actual deployment of the scheduling scheme. Based on the allocation decision x on the peer side in step S4. in Communication Decision i and end-side sampling frame rate f end The final solution results are used to deploy DNN models, camera sampling rates, etc., on end-side devices and edge-side servers in the actual physical environment.

[0107] In this embodiment, the actual deployment of the scheduling scheme in step S5 includes the following steps:

[0108] Step S51, transformation of decision variables into implementation schemes. Based on the allocation decision x on the remote side in step S4... in Communication Decision i and end-side sampling frame rate decision f end The final solution result will determine the end-side allocation decision x. inChinese x i0 The operators corresponding to =1 need to be deployed on the edge device; the remaining operators depend on x. in In the case where = 1 (n ≠ 0), operator i is deployed in the corresponding nth edge server; in addition, the end-side sampling frame rate decision f end The value indicates the video sampling frame rate that the end-side device needs to set.

[0109] Step S52: Implement the actual deployment of the solution. Based on the implementation plan in S51, deploy the DNN model, camera sampling rate, etc., of the end-side devices and edge-side servers in the actual physical environment.

[0110] like Figure 8 As shown, a real-time video task end-to-end collaborative scheduling device includes a data acquisition module, a DAG graph construction module, an end-to-end collaborative scheduling model construction module, and a model solving module connected in sequence.

[0111] The data acquisition module is responsible for collecting and recording information on the hardware / software platform framework, computing power consumption, transmission power consumption, standby power consumption, and other end-side device power consumption, as well as operator information of the DNN model, and data transmission information between operators such as data transmission direction and data transmission volume, which serve as input for subsequent modules.

[0112] The DAG graph construction module constructs the topological network relationships between operators based on the logical arrangement of nodes and edges in the DNN network. It assigns edge weights to the DAG graph topological network of the DNN model according to the amount of data transmitted between operators, with the weight value corresponding to the amount of data transmitted between the two operators.

[0113] The end-to-end collaborative scheduling model construction module, based on the results of the data acquisition module and the DAG graph construction module, defines and mathematically expresses the decision variables of the scheduling problem. Secondly, it constructs the objective function of the scheduling problem using the weighted sum of video task processing latency, video task processing rate, and end-side energy consumption as indicators. Thirdly, it mathematically expresses scheduling problem constraints such as configuration constraints, non-overlapping constraints, end-side computation latency constraints, edge-side computation latency constraints, start / end time constraints, network transmission latency constraints, and end-side energy consumption constraints. Finally, it integrates the decision variables, objective function, and constraints to complete the construction of the end-to-end collaborative scheduling problem model.

[0114] The model solving module compiles DNN model-related parameters such as DAG graph network, operator end-side / edge-side processing latency, and operator output data volume. It also compiles the decision variables, objective function, and constraints of the end-network-edge collaborative scheduling model, inputs the compilation results into the solver, sets solver-related parameters such as TimeLimit and MIPFocus, runs the solver, and outputs the final running results.

[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time video task end-edge collaborative scheduling, characterized in that Comprise the following steps: Step S1, data acquisition; Collect platform framework information, end-side device energy consumption information, video analysis deep neural network (DNN) model operator information, data transmission information between each operator; Step S2, directed acyclic graph (DAG) graph construction of the DNN model; the DAG graph of the DNN model is composed of two types of elements, nodes V and edges E , and is represented as G = ( V , E ) ; according to the data flow direction between the operators, the topological network relationship between the operators is established; and according to the size of the data transmission amount between the operators, the weight of the DAG graph edge is valued; Step S3, end-network-edge collaborative scheduling model construction; According to the data acquisition results and DAG graph construction results of the above steps, the end-network-edge collaborative scheduling model is established based on integer programming modeling method, and the decision variables, optimization objective and constraints of the model are defined and mathematically expressed; Specifically, it includes: Step S31, define and describe the decision variables of the end-network-edge collaborative scheduling model; Step S32, define and describe the optimization objective of the end-network-edge collaborative scheduling model; Step S33, define and describe the constraints of the end-network-edge collaborative scheduling model; Step S31 requires determining the placement location of the operator, the data transmission path of the operator, the sampling frame rate of the end device, and the start and end times of the operator, thereby generating end-side allocation decisions. Communication Decision End-side sampling frame rate decision Start time decision and end time decision Among them, end-side allocation decision Defined as an operator, which is a 0 / 1 variable. Whether to place on the device The value is 1 for yes and 0 for no; communication decision. Defined as an operator, which is a 0 / 1 variable. Does cross-device data transmission exist? 1 for yes, 0 for no; End-side sampling frame rate decision. This is an integer variable, defined as the frame rate at which video frames are sampled at the end; the start time decision. and end time decision These are integer variables, defined as the start and end times of the operator. The optimization target of the scheduling in S32 includes a video task processing delay, a video task processing rate, and an end-side energy consumption; wherein the video task processing delay The processing delay of only the effective frame can be further expressed as an end-side calculation delay , a transmission delay , and an edge-side calculation delay , which are shown in formula (1); the video task processing rate is determined by the processing rate bottleneck of the end, the network, and the edge; the relationship between the processing rate of the end, the network, and the edge , , , and the calculation delay is shown in formula (2); the end-side energy consumption includes a calculation energy consumption, a standby energy consumption, and a communication energy consumption; the optimization target of the scheduling is finally expressed as a weighted sum of the end-side calculation delay, the video task processing rate, and the end-side energy consumption, which is shown in formula (3); The end-network-edge collaborative scheduling in step S33 includes the following constraints, which are expressed by formula (4-19): Configuration constraints: contains constraint (4) and constraint (5), where constraint (4) represents any operator can only be deployed on one device; constraint (5) represents when an operator of any postorder is placed at the end side, the operator can also only be placed at the end side; Non-overlapping constraint: comprising constraint (6) and constraint (7), indicating that when two operators are arranged on the same device, the device carries out the calculation processing in a serial manner; wherein, and are two large numbers, respectively , End-side computing delay constraint: includes constraint (8), which represents that the computing delay of the end-side device is the sum of the computing delays of all operators deployed on the end-side; Transmission delay constraint: includes constraint (9) and constraint (10), wherein constraint (9) represents that the transmission delay is the ratio of the bandwidth and the transmission data volume; Constraint (10) represents the relationship between the decision variables; Edge-side computing delay constraint: includes constraint (11), which represents that the edge-side computing delay is the effective frame end-to-end delay minus the end-side computing delay and the end / edge-side transmission delay; Start / End Time Constraints: Includes constraint (12), constraint (13) and constraint (14), where constraint (12) represents the operator. Any postorder operator The start time cannot be earlier than the operator. The end time; constraint (13) indicates that if the operator and operators If data transmission exists, then any operator The start time cannot be earlier than the transmission completion time; constraint (14) represents the relationship between the start time, end time and computation time of the operator; End-side energy consumption constraint: includes constraint constraint (15), constraint (16), constraint (17), constraint (18) and constraint (19), wherein constraint (15) and constraint (16) represent processing energy consumption; Constraint (17) represents standby energy consumption; Constraint (18) represents transmission energy consumption; The overall end-side energy consumption is the sum of the three types of energy consumption, which is represented as constraint (19); Step S4, solving of the scheduling model based on the solver; Based on the programming language of the solver, the decision variables, objective function and constraints of the end-network-edge collaborative scheduling model are compiled in the solver environment, and the DAG graph network structure of the specific DNN model is input into the solver, and the optimal scheduling result is obtained by solving; Step S5, actual deployment of the scheduling scheme; According to the scheduling model solving result, the end-side device and the edge-side server are deployed in the actual physical environment. 2.The real-time video task terminal network edge collaborative scheduling method of claim 1, wherein The data acquisition of S1 comprises the following steps: Step S11, collect platform framework information, including hardware and software platform information of end-side and edge-side; Step S12, collect end-side device energy consumption information, including computing energy consumption, transmission energy consumption and standby energy consumption information; Step S13, collect operator information of DNN model, including operator type and computing delay information; Step S14, collect data transmission information between each operator, including data transmission direction and data transmission volume information.

3. The method of claim 1, wherein The DAG graph construction of the DNN model of S2 comprises the following steps: Step S21, establish the topological network relationship between operators; Step S22, assign weights to the edges of the DAG graph.

4. The method of claim 1, wherein The scheduling model solving based on the solver of S4 comprises the following steps: Step S41, based on CLPEX, Gurobi solver, the decision variables, objective function and constraints of the end network edge collaborative scheduling model are compiled; Step S42, the related parameters of the DNN model are input into the solver; Step S43, the related parameters of the solver are set, the solver is run, and the final running result is output.

5. The method of claim 1, wherein The scheduling scheme of S5 is actually deployed, including the following steps: Step S51, transformation of decision variables to implementation; according to end-side allocation decision , communication decision and end-side sampling frame rate decision , the final solution result of end-side allocation decision , the corresponding related operators in need to be deployed in the end-side device, and the remaining operators need to be deployed in the corresponding edge server; in addition, the value of end-side sampling frame rate decision indicates the video sampling frame rate that needs to be set by the end-side device; Step S52, the implementation scheme is actually deployed; according to the implementation scheme of S51, the DNN model of the end side device and the edge side server, and the camera sampling rate are deployed in the actual physical environment.

6. A real-time video task terminal edge collaborative scheduling apparatus, comprising: The sequentially connected data acquisition module, DAG graph construction module, end network edge collaborative scheduling model construction module and model solving module are characterized by: The data acquisition module is responsible for collecting hardware / software platform framework information of the end side and the edge side, computing energy consumption / transmission energy consumption / standby energy consumption end side device energy consumption information, operator information of the DNN model, data transmission information between operators, and recording as input for the subsequent modules; The DAG graph construction module constructs the topological network relationship between operators according to the logical analysis results of the DNN network nodes and edges; the edge weight of the DAG graph topological network of the DNN model is valued according to the data transmission amount between operators, and the weight value is the data transmission amount between two operators; The end network edge collaborative scheduling model construction module defines and mathematically expresses the decision variables of the scheduling problem based on the results of the data acquisition module and the DAG graph construction module; secondly, the scheduling problem objective function is constructed with the weighted sum of video task processing delay, video task processing rate and end side energy consumption as the index; thirdly, the scheduling problem constraints are mathematically expressed, including configuration constraints, non-overlapping constraints, end side computing delay constraints, edge side computing delay constraints, start / end time constraints, network transmission delay constraints and end side energy consumption constraints; finally, the decision variables, objective function and constraints are integrated to complete the end network edge collaborative scheduling problem model construction; specifically including: Step S31, defining and describing the decision variables of the end network edge collaborative scheduling model; Step S32, defining and describing the optimization objective of the end network edge collaborative scheduling model; Step S33, defining and describing the constraints of the end network edge collaborative scheduling model; Step S31 requires determining the placement location of the operator, the data transmission path of the operator, the sampling frame rate of the end device, and the start and end times of the operator, thereby generating end-side allocation decisions. Communication Decision End-side sampling frame rate decision Start time decision and end time decision Among them, end-side allocation decision Defined as an operator, which is a 0 / 1 variable. Whether to place on the device The value is 1 for yes and 0 for no; communication decision. Defined as an operator, which is a 0 / 1 variable. Does cross-device data transmission exist? 1 for yes, 0 for no; End-side sampling frame rate decision. This is an integer variable, defined as the frame rate at which video frames are sampled at the end; the start time decision. and end time decision These are integer variables, defined as the start and end times of the operator. The optimization target of the scheduling in S32 includes a video task processing delay, a video task processing rate, and an end-side energy consumption; wherein the video task processing delay The processing delay of only the effective frame can be further expressed as an end-side calculation delay , a transmission delay , and an edge-side calculation delay , which are shown in equation (1); the video task processing rate is determined by the processing rate bottleneck of the end, the network, and the edge; the relationship between the processing rate of the end, the network, and the edge , , , and the calculation delay is shown in equation (2); the end-side energy consumption includes a calculation energy consumption, a standby energy consumption, and a communication energy consumption; the optimization target of the scheduling is finally expressed as a weighted sum of the end-side calculation delay, the video task processing rate, and the end-side energy consumption, which is shown in equation (3); The end network edge collaborative scheduling in step S33 includes the following constraints, which are described by formula (4-19): Configuration constraints: contains constraint (4) and constraint (5), where constraint (4) represents any operator can only be deployed on one device; constraint (5) represents when an operator has any post-sequenced operator placed on the end side, the operator can also only be placed on the end side; Non-overlapping constraint: comprising constraint (6) and constraint (7), indicating that when two operators are arranged on the same device, the device carries out the calculation processing in a serial manner; wherein, and are two large numbers, respectively , End side computing delay constraint: includes constraint (8), which represents that the computing delay of the end side device is the sum of the computing delays of all operators deployed on the end side; Transmission delay constraint: includes constraint (9) and constraint (10), wherein constraint (9) represents that the transmission delay is the ratio of bandwidth to transmission data amount; constraint (10) represents the relationship between decision variables; Edge side computing delay constraint: includes constraint (11), which represents that the edge side computing delay is the effective frame end-to-end delay minus the end side computing delay and the end / edge side transmission delay; Start / End Time Constraints: Includes constraint (12), constraint (13) and constraint (14), where constraint (12) represents the operator. Any postorder operator The start time cannot be earlier than the operator. The end time; constraint (13) indicates that if the operator and operators If data transmission exists, then any operator The start time cannot be earlier than the transmission completion time; constraint (14) represents the relationship between the start time, end time and computation time of the operator; End-side energy consumption constraints: including constraints (15), constraints (16), constraints (17), constraints (18) and constraints (19), wherein constraints (15) and constraints (16) represent processing energy consumption; constraints (17) represent standby energy consumption; constraints (18) represent transmission energy consumption; the overall end-side energy consumption is the sum of the three types of energy consumption, represented as constraints (19); The model solving module compiles the DAG graph network, the operator end-side / edge-side processing delay, the operator output data volume DNN model related parameters, compiles the decision variables, the objective function and the constraints of the end network edge collaborative scheduling model, inputs the compilation result into the solver, sets the TimeLimit, MIPFocus solver related parameters, runs the solver, and outputs the final running result.

Images