A request scheduling method for multi-task edge video analysis

Abstract

The present invention belongs to the field of edge video analysis technology, and discloses a request scheduling method for multi-task edge video analysis, comprising the following steps: constructing a multi-task video analysis system based on edge computing; establishing a configuration selection model that considers the merging of multi-task requests; performing GPU resource modeling based on GPU virtualization technology; extracting video frame features based on multi-feature fusion based on the video analysis model; predicting video analysis accuracy based on the Transformer model; establishing a system total cost minimization optimization problem; and solving the system total cost minimization problem based on a heuristic algorithm. The present invention adopts the above-mentioned request scheduling method for multi-task edge video analysis, and minimizes the total system cost while meeting the user's accuracy and latency requirements, accurately predicts the relationship between configuration and analysis accuracy in real time, efficiently utilizes limited network and computing resources, realizes network transmission sharing, and reduces resource consumption.

Description

Technical Field

[0001] The present invention relates to the field of edge video analysis technology, and in particular to a request scheduling method for multi-task edge video analysis. Background Art

[0002] Currently, over one billion cameras deployed worldwide generate massive amounts of video data daily, widely used in areas such as public security, traffic monitoring, and autonomous driving. However, traditional manual video analysis methods often rely on post-analysis of video playback, which suffers from low accuracy, high false negative rates, and an inability to respond in real time. Advances in artificial intelligence and computer vision technologies have made highly accurate intelligent video analysis possible. These methods are typically based on computationally complex deep neural network inference, making them difficult to run directly on camera devices. Traditional intelligent video analysis primarily utilizes centralized cloud computing architectures, which are subject to network congestion, high response latency, and single points of failure. With the development of edge computing technology, intelligent video analysis methods are gradually evolving towards edge computing. By performing computing tasks at the edge of the network, close to where the data is generated, response times can be significantly improved, alleviating network pressure.

[0003] Existing edge computing-based intelligent video analytics systems typically focus on a single task, such as vehicle tracking, traffic accident detection, or pedestrian counting. However, in real-world applications, multiple video analysis requirements often exist for the same camera. Deploying a separate system for each task would waste costs and resources. Multi-task edge video analytics is an emerging architecture that supports simultaneous execution of multiple tasks on the same video stream, effectively reducing the overhead of duplicate deployments.

[0004] In edge video analytics, the key to improving system performance and achieving efficient, high-quality data analysis lies in adaptively adjusting configuration parameters and optimizing task scheduling based on dynamic computing and communication resources. Adjustable configuration parameters include video resolution, frame rate, deep learning model selection, and task execution location. Request scheduling for edge video analytics is typically modeled as a mathematical optimization problem. Prior to optimization, it is necessary to pre-compute the video analysis accuracy of each request under various configuration parameters. This is known as "configuration-accuracy analysis."

[0005] However, existing multi-task edge video analysis methods still face the following challenges:

[0006] (1) Configuration-accuracy analysis suffers from inaccuracy and high computational cost: Existing methods usually ignore dynamic changes in video content such as illumination changes and object speed, and most rely on offline or periodic methods, making it difficult to achieve an ideal balance between accuracy and efficiency.

[0007] (2) There is high computational redundancy when processing a large number of video analysis requests: Multiple analysis requests for the same video may involve repeated network transmission and deep neural network inference, and existing methods fail to fully share network and computing resources.

[0008] (3) Complexity of joint configuration adjustment and GPU resource allocation: Existing research often ignores the importance of GPU allocation in improving system performance. The complex correlation between multiple configuration parameters further increases the difficulty of decision-making in joint configuration adjustment and GPU resource allocation. Summary of the Invention

[0009] The purpose of the present invention is to provide a request scheduling method for multi-task edge video analysis, aiming to minimize the total system cost through dynamic configuration adjustment and GPU resource allocation under the premise of meeting user accuracy and latency requirements; capture the current video content and inter-frame temporal dependencies through a prediction module based on the Transformer model, and accurately predict the relationship between configuration and analysis accuracy in real time; adopt video analysis request merging technology to efficiently utilize limited network and computing resources, realize the sharing of network transmission and model calculation, and reduce resource consumption; based on the configuration-accuracy analysis results, use a heuristic algorithm to efficiently schedule video analysis requests, dynamically adjust video configuration, model selection, request merging, task deployment and GPU resource allocation to optimize system performance and reduce costs.

[0010] To achieve the above objectives, the present invention provides a request scheduling method for multi-task edge video analysis, comprising the following steps:

[0011] Step S1: Build a multi-task video analysis system based on edge computing: Multi-task edge video analysis adopts a cloud-edge-end collaborative architecture, which mainly consists of three layers: end device camera, edge server and cloud center;

[0012] Step S2: Establishing a configuration selection model considering multi-task request merging: Based on the multi-task video analysis system in step S1, a multi-task request merging model is constructed for communication and computing resources to optimize the configuration selection process;

[0013] Step S3: GPU resource modeling based on GPU virtualization technology: logically partitioning the physical GPU based on GPU virtualization technology to achieve dynamic allocation of GPU computing resources and video memory resources, and establishing a corresponding resource constraint model;

[0014] Step S4: Based on the video analysis model, perform multi-feature fusion video frame feature extraction: Based on the visual features of the video analysis model and the detection features based on the video analysis results, the video content and the detection results are comprehensively considered to fully reflect the current video characteristics; wherein, the intermediate feature map of the video analysis model is used as the visual feature, and the video analysis results are extracted as the detection feature;

[0015] Step S5: Predicting video analysis accuracy based on the Transformer model: Using the Transformer model to process the time series composed of the video frame features in step S4, capturing sequence features and temporal dependencies, and then processing the feature representation extracted by the Transformer through a multi-layer perceptron to predict the relationship between the camera configuration and analysis accuracy in the current scene;

[0016] Step S6: Establish an optimization problem for minimizing the total system cost: define the internal cost of the edge computing network and the external cost of task offloading based on serverless cloud computing, consider the latency and accuracy requirements, and construct a quadratically constrained quadratic programming problem for minimizing the total system cost;

[0017] Step S7: Solve the total system cost minimization problem in step S5 based on a heuristic algorithm: Based on the predicted configuration-accuracy relationship, adopt a multi-granularity heuristic solution improvement algorithm, improve the feasibility and quality of the solution simultaneously through the penalty function idea, and combine it with a greedy algorithm to achieve optimal allocation of GPU resources.

[0018] Preferably, a system framework for a request scheduling method for multi-task edge video analysis is composed of a configuration selection and GPU resource modeling module, a video analysis accuracy prediction module based on a Transformer model, an optimization problem module for minimizing the total system cost, and a request scheduling algorithm module based on heuristic thinking;

[0019] Among them, the configuration selection and GPU resource modeling modules establish a multi-task edge video analysis architecture with cloud-edge-end collaboration, and design a configuration selection model and GPU resource allocation model that considers the merging of multi-task requests;

[0020] The Transformer-based video analysis accuracy prediction module predicts the analysis accuracy corresponding to each camera configuration in the current scene based on the relationship between video content and video frame changes;

[0021] The optimization problem module for minimizing the total system cost is based on the system model and accuracy information, taking into account the latency and accuracy requirements, and constructing a quadratic constrained quadratic programming problem for minimizing the total system cost.

[0022] A multi-granularity heuristic solution improvement algorithm is designed based on the request scheduling module of the heuristic algorithm. The feasibility and quality of the solution are improved through the penalty function idea, and the greedy algorithm is combined to achieve the optimal allocation of GPU resources.

[0023] Preferably, the workflow of the multi-task video analysis system based on edge computing is:

[0024] A global scheduler schedules video analysis requests, optimizing configuration selection, request merging, and GPU resource allocation. It treats detection tasks for different categories of targets as separate video analysis requests.

[0025] Among them, the scheduler relies on four inputs: the accuracy of each video stream provided online by the video analysis accuracy prediction module; the data size of different frames obtained through offline analysis and the GPU memory usage of different video analysis models; user-defined accuracy and latency requirements; and real-time resource consumption of bandwidth, GPU and video memory.

[0026] Preferably, in step S2, a configuration selection model considering the merging of multiple task requests is established, specifically as follows:

[0027] The configuration options that can be adjusted in a video analysis request are represented as a 4-tuple δ(r δ ,f δ ,m δ ,p δ ), the set of all available configurations is represented by Δ = {δ}; where the variable r δ Indicates video resolution; f δ Indicates the number of frames per second for video analysis FAPS; m δ represents the video analysis model; p δ Indicates the requested execution location. When p δ = 0 means the request is processed locally on the edge server. δ =1 means the request is offloaded to the cloud for processing;

[0028] To implement configuration selection by adjusting the configuration tuple for each request, a binary variable is introduced. in Indicates that request q selects configuration δ. Since a video analysis request can only select one configuration, the following conditions must be met:

[0029]

[0030] in, represents the set of requests on all K cameras, Represents the camera collection of the end device;

[0031] In order to save communication and computing resources, a video analysis request merging scheme is adopted to introduce intermediate binary variables. The request merging process is controlled by the "Boolean OR" operation; where k is the camera number; when When at least one of is true, as shown below:

[0032]

[0033] in, represents the request set on camera k;

[0034] Therefore, the bandwidth resource constraints between the camera and the edge server and the bandwidth resource constraints between the edge server and the cloud center are as follows:

[0035]

[0036] Among them, b δ Indicates the data size of one frame when using configuration δ, Indicates the total bandwidth from the camera to the edge server. represents the total bandwidth from the edge server to the cloud center; the fixed function o(·) defines the mapping between the camera and the edge server, where the camera Connecting to the Edge Server {1,2,…,N} represents the set of edge servers.

[0037] Preferably, in step S3, based on the multi-task video analysis system architecture model and the multi-task video analysis request merging model based on edge computing, GPU computing resource and GPU memory resource models and constraints are established, and GPU resource modeling based on GPU virtualization is performed, specifically as follows:

[0038] (1) GPU computing resources: Let discrete variables Indicates the GPU computing resource allocation for each merge request; G is the total number of computing resource slices; the GPU computing capacity constraint of each edge server can be expressed as:

[0039]

[0040] in, represents the total GPU computing resources of edge server n;

[0041] (2) GPU memory resources: For GPU memory usage, the memory usage required by each video analysis model is obtained through offline analysis; the GPU memory resource constraint can be expressed as:

[0042]

[0043] Among them, u δ Indicates the video memory usage of the video analysis model when using configuration δ; Indicates the total GPU memory resources of edge server n.

[0044] Preferably, in step S4, based on the video analysis model, video frame feature extraction of multi-feature fusion is performed, wherein the feature vector of each frame is composed of three parts, namely, visual features, current video analysis model size and detection features, as follows:

[0045] (1) Visual features: Reuse the intermediate feature maps generated by the backbone of the video analysis model to obtain the visual feature map, and then use the global average pooling (GAP) operation to reduce the spatial dimension; use repeated padding operations to fill all vectors to the dimension that matches the vector generated by the largest video analysis model, and use f v Represents the visual features of a single video frame; uses a one-hot vector to encode the currently selected video analysis model, denoted as f m ; where f m As f v Indicator of filling mode;

[0046] (2) Detection features: The target detection results output by the video analysis model include category labels, bounding boxes, and confidence scores, which are used to characterize video features as follows:

[0047] A. Category label: Use vector f a Indicates the number of detected objects as follows:

[0048] f a =[a,a1,…,a T ] (7);

[0049] Among them, the total number of targets A and the number of targets in each category a1 to a T ;gather Represents all target categories that can be detected, all category labels in one frame

[0050] B. Bounding box size: Use B t The statistics of represent the target size information, as shown below:

[0051]

[0052] in, is the vector of the bounding box size; B t Represents a vector consisting of the bounding box sizes of all targets with category label t in a frame; min(B t)、max(B t )、mean(B t ) and var(B t ) represent B t the minimum, maximum, mean, and variance of ;

[0053] C. Confidence score: reflects the model's certainty about each detected target, using the statistical vector Describe the confidence scores as follows:

[0054]

[0055] Among them, C t is a vector of confidence scores of all targets with category labels t in a frame; sum(C t ) represents the sum of confidences of all objects with category label t; min(C t )、max(C t )、mean(C t ) and var(C t ) represent C t the minimum, maximum, mean, and variance of ;

[0056] By concatenating the above vectors, we can get the detection feature f of a video frame. d , as shown below:

[0057]

[0058] The visual features, current model size, and detection features are concatenated together to fully represent a video frame; before concatenation, these features are made to have similar dimensions through a linear projection layer, as shown below:

[0059] f=[f v ,f m ,f d ] (11);

[0060] Among them, f v Represents the visual features of a single video frame; f m Indicates that a single hot vector is used to encode the currently selected video analysis model, f d Represents the detection features of a video frame.

[0061] Preferably, in step S5, the video analysis accuracy prediction is performed based on the Transformer model, as follows:

[0062] To explore the potential implicit features in the video frame sequence, the Transformer model is used to directly capture the temporal dependency information in the time window F of the feature sequence, as shown below:

[0063] F=f (1) ,f (2) ,…,f (β) (12);

[0064] H = Encoder(F) (13);

[0065] z = Pooling(H) (14);

[0066] a t =MLP(z),t=1,…,T (15);

[0067] a=[a1,…,a T ] (16);

[0068] Where F represents the time window of the feature sequence directly captured by the Transformer model; the parameter β is the length of the window; by utilizing the self-attention mechanism, the Transformer model captures local and global dependencies and contextual relationships in the data, making it very effective in processing complex temporal pattern tasks;

[0069] The Transformer architecture consists of an encoder and a decoder. The feature representation is extracted using the Transformer encoder, which consists of multiple Transformer encoder layers. Positional encoding is added to provide the model with the position information of each element in the sequence.

[0070] After the Transformer encoder, an average pooling operation is applied to aggregate the features in the sequence and output a feature representation. The resulting feature vector is input to the output layer, which consists of multiple multi-layer perceptrons (MLPs). Each MLP is responsible for predicting the accuracy of a specific target detection category under different configurations. Each MLP consists of two fully connected layers that map the aggregated features to the final accuracy prediction.

[0071] Preferably, in step S6, an optimization problem of minimizing the total system cost is established based on system modeling, resource constraints, and prediction accuracy, as follows:

[0072] S61. Establish user delay constraints, and the delay requirement constraints for each request are as follows:

[0073]

[0074] in, Indicates the service delay per frame when using configuration δ; Indicates the maximum end-to-end delay requirement;

[0075] To prevent frame loss, the throughput of the selected model must be greater than the input frame rate, so the constraint expression is:

[0076]

[0077] S62. Establish user accuracy constraint: the prediction accuracy of the selected configuration for each request Its user accuracy requirements should be met as follows:

[0078]

[0079] in, Indicates the prediction accuracy of the selected configuration for each request; Indicates the minimum accuracy requirement; the prediction accuracy is provided by the video analysis accuracy prediction module based on the Transformer model in step S5;

[0080] S63. Problem Statement: The goal is to minimize the total cost for the edge service provider while satisfying the accuracy, latency, and resource constraints required for edge video analysis. Let C be the total cost of the edge service provider, which includes internal cost I and external cost E, and is defined as follows:

[0081] Internal cost I includes three parts: transmission cost within the edge network, GPU memory cost of all edge servers and GPU computing cost; usage coefficient ι t 、ι u and ι g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second; the internal cost per second is as follows:

[0082]

[0083] Among them, i trans is the internal transmission cost; i mem is the internal memory cost; i gpu is the internal GPU cost;

[0084] External costs E include: the edge video analysis system needs to pay fees to the cloud service provider for network bandwidth, GPU computing and memory resource usage; cloud computing coefficient ∈ t ,∈ u and ∈ g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second; the external cost per second is as follows:

[0085]

[0086] Among them, e trans is the external transmission cost; e mem is the external memory cost; e gpu Cost of external GPU;

[0087] Based on this, the system total cost minimization problem is expressed as:

[0088]

[0089] stC1,C2,C3,C4,C5,C6,C7,C8(27);

[0090]

[0091] Where G is the total number of computing resource slices;

[0092] The above problem is classified as a quadratically constrained quadratic programming (QCQP) problem. The complexity of solving the QCQP problem by semidefinite relaxation SDR is as follows:

[0093]

[0094] Where d is the variable dimension and ω is the accuracy of the solution.

[0095] Preferably, based on a heuristic algorithm, the total system cost minimization problem in step S5 is solved by coordinating configuration selection, request merging, and GPU allocation, as follows:

[0096] The key principle of the heuristic strategy is to start from an initial solution and gradually reduce the cost. In each iteration, the configuration selection x is first updated through a greedy approach, and then the corresponding GPU computing resources g are allocated. If the current objective has not been significantly improved within θ iterations and all constraints are satisfied, the loop is terminated and the solution is output.

[0097] In order to meet the user's accuracy requirements, a configuration set Δ that meets the accuracy requirements is established in advance for each request q q ,in In the subsequent phase, only each request Δ q Configuration within;

[0098] (1) Improvement of the solution based on the penalty function idea;

[0099] Inspired by the penalty function method for solving constrained optimization problems, the efficiency index φ is defined as the current target and constraint violation term σ(P L +P R ), as follows:

[0100]

[0101] Among them, σ is the penalty coefficient, P L represents the penalty term for delay, P R Penalty term representing resource constraints;

[0102] Penalty term P for delay constraints and resource constraints L and P R , as shown below:

[0103]

[0104] in, Indicates the currently used bandwidth from the camera to the edge server; Indicates the currently used bandwidth from edge server to cloud server; Indicates the GPU resources currently used; Represents the currently used memory resources; when the performance index φ is within the feasible solution set, the constraint violation item is zero, and becomes positive when it exceeds the feasible solution set;

[0105] Based on the multi-granularity solution update strategy, each iteration only updates one request or the configuration of all requests on a single camera; define φ q,δ To switch from the current configuration decision x to the new decision x (q,δ) The performance index of (q,δ) Indicates that only the configuration of request q is updated to configuration δ, while retaining the existing configurations of other requests; φ k,δ Indicates switching from x to x (k,δ) The performance index of (k,δ) Indicates that the configuration of all requests on camera k is updated to configuration δ, and the configuration of requests on other cameras remains unchanged; then, from {φ q,δ} and {φ k,δ}, and select the solution with the smallest performance index from the union of the two. This strategy reduces the search space to HQ+HK, where Q is the total number of video analysis requests, H is the total number of configurations, and K is the total number of cameras.

[0106] (2) Two-stage decision-making for configuration selection and GPU allocation: First, determine the current configuration selection decision x, and then use the greedy algorithm to determine the allocation of GPU computing resources g;

[0107] First, use x to update the intermediate request merge variable s, and then calculate the delay from the camera to the edge server and from the edge server to the cloud center;

[0108] Next, calculate the GPU computing resource allocation for each pair of k and δ If there is no request to select configuration δ on camera k, then Set to 0; otherwise for merge requests Requests in, determine the maximum computational delay that can be tolerated by merging requests;

[0109] If the latency requirement cannot be met even if all GPU computing resources are allocated, Set to 1 and mark the constraint satisfaction flag η as False; otherwise Set it to the minimum value that satisfies the latency requirement and throughput constraint, and set η to True.

[0110] Therefore, the present invention adopts the above-mentioned request scheduling method for multi-task edge video analysis, and the beneficial effects are as follows:

[0111] (1) The present invention provides an edge computing architecture for multi-task video analysis, which can effectively reduce the overhead caused by repeated deployment and improve resource utilization efficiency compared to traditional single-task video analysis systems;

[0112] (2) The configuration-accuracy relationship prediction model based on the Transformer model can comprehensively consider the relationship between video content and video frame changes to achieve efficient and high-precision video analysis accuracy prediction;

[0113] (3) The present invention realizes the merging of video analysis requests to fully share network and computing resources, effectively addressing the problem of high computational redundancy when processing a large number of video analysis requests;

[0114] (4) The heuristic algorithm for edge video analysis request scheduling in this invention effectively solves the complex joint optimization problem of configuration selection and GPU resource allocation, and can significantly reduce the total system cost while meeting user accuracy and latency requirements;

[0115] (5) This invention provides ideas for the allocation and scheduling of discrete resources, broadening the application scenarios of resource management in edge computing;

[0116] (6) The architecture and method proposed in this invention support a wide range of video analysis application scenarios, such as intelligent transportation, smart cities, industrial monitoring, etc., and have strong versatility.

[0117] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 This is a schematic diagram of the architecture and workflow of the multi-task edge video analysis system of the present invention;

[0119] Figure 2 Schematic diagram of a module for request scheduling method for multi-task edge video analysis of the present invention;

[0120] Figure 3Schematic diagram of the video analysis accuracy prediction model structure based on the Transformer model of the present invention; wherein (A) is the feature vector structure of each frame; (B) is the process of the Transformer model directly capturing the feature sequence;

[0121] Figure 4 is a heuristic algorithm diagram for video analysis request scheduling of the present invention;

[0122] Figure 5 It is a greedy algorithm diagram of GPU allocation of the present invention. DETAILED DESCRIPTION

[0123] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0124] like Figure 1 As shown, the present invention provides a request scheduling method for multi-task edge video analysis, comprising the following steps:

[0125] Step S1: Build a multi-task video analysis system based on edge computing: Multi-task edge video analysis adopts a cloud-edge-end collaborative architecture, which mainly consists of three layers: end device camera, edge server and cloud center;

[0126] Step S2: Establishing a configuration selection model considering multi-task request merging: Based on the multi-task video analysis system in step S1, a multi-task request merging model is constructed for communication and computing resources to optimize the configuration selection process;

[0127] Step S3: GPU resource modeling based on GPU virtualization technology: logically partitioning the physical GPU based on GPU virtualization technology to achieve dynamic allocation of GPU computing resources and memory resources, and establishing a corresponding resource constraint model;

[0128] Step S4: Based on the video analysis model, perform multi-feature fusion video frame feature extraction: Based on the visual features of the video analysis model and the detection features based on the video analysis results, the video content (such as lighting and scene) and the detection results (such as target category and size) are comprehensively considered to fully reflect the current video characteristics; wherein, the intermediate feature map of the video analysis model is used as the visual feature, and the video analysis results are extracted as the detection feature;

[0129] Step S5: Predicting video analysis accuracy based on the Transformer model: Using the Transformer model to process the time series composed of video frame features, capturing sequence features and temporal dependencies, and then processing the feature representation extracted by the Transformer through a multi-layer perceptron to predict the relationship between the camera configuration and analysis accuracy in the current scene;

[0130] Step S6: Establish an optimization problem for minimizing the total system cost: define the internal cost of the edge computing network and the external cost of task offloading based on serverless cloud computing, consider the latency and accuracy requirements, and construct a quadratically constrained quadratic programming problem for minimizing the total system cost;

[0131] Step S7, efficient request scheduling based on heuristic algorithm: Based on the predicted configuration-accuracy relationship, a multi-granularity heuristic solution improvement algorithm is adopted. The feasibility and quality of the solution are improved simultaneously through the penalty function idea, and the greedy algorithm is combined to achieve optimal allocation of GPU resources.

[0132] like Figure 2 As shown, the system framework of the request scheduling method for multi-task edge video analysis of the present invention is mainly composed of a configuration selection and GPU resource modeling module, a video analysis accuracy prediction module based on the Transformer model, an optimization problem module for minimizing the total system cost, and an efficient request scheduling algorithm module based on heuristic thinking.

[0133] Among them, the configuration selection and GPU resource modeling modules established a multi-task edge video analysis architecture with cloud-edge-end collaboration, designed a configuration selection model and GPU resource allocation model that considers the merging of multi-task requests, and provided system model support for subsequent optimization problems.

[0134] The video analysis accuracy prediction module based on the Transformer model predicts the analysis accuracy corresponding to each camera configuration in the current scene through the relationship between video content and video frame changes, providing accuracy information support for the formation of optimization problems.

[0135] The optimization problem module for minimizing the total system cost is based on the system model and accuracy information, and comprehensively considers the delay and accuracy requirements to construct a quadratic constrained quadratic programming problem for minimizing the total system cost.

[0136] A multi-granularity heuristic solution improvement algorithm is designed based on the efficient request scheduling module of the heuristic algorithm. The feasibility and quality of the solution are improved simultaneously through the penalty function idea, and the greedy algorithm is combined to achieve the optimal allocation of GPU resources.

[0137] Example

[0138] The following example specifically illustrates a request scheduling method for multi-task edge video analysis of the present invention. The specific processing flow calculation is described as follows:

[0139] Step S1: Build a multi-task video analysis system based on edge computing.

[0140] First, a cloud-edge-end collaborative architecture is adopted to build a multi-task video analysis system based on edge computing, which mainly consists of three layers: end device cameras, edge servers, and cloud centers. Figure 1 shown.

[0141] Terminal device camera collection Indicates that multiple video analysis requests can be executed simultaneously on a camera, and the requests on camera k can be represented as a set Any request Has its user-specified maximum end-to-end delay Requirements and minimum accuracy requirements To support its task. The request set on all K cameras is expressed as

[0142] Edge Server Collection They are located at the edge of the network and provide timely services to nearby cameras. Each edge server can provide services to multiple end-device cameras, while a camera is only connected to one edge server. The mapping between cameras and edge servers is defined by a fixed function o(·), where camera Connecting to the Edge Server

[0143] The cloud center is interconnected with the edge servers through the public Internet to process video analysis requests that exceed the computing capacity of the edge servers.

[0144] Secondly, the workflow of the edge computing-based multi-task video analysis system is as follows: a global scheduler schedules video analysis requests, optimizing configuration selection, request merging, and GPU resource allocation. This scheduler relies on four inputs: the accuracy of each video stream provided online by the video analysis accuracy prediction module; the data size of different frames and GPU memory usage of different video analysis models obtained through offline analysis; user-defined accuracy and latency requirements; and the real-time resource consumption of bandwidth, GPU, and video memory.

[0145] Considering that target detection is the basic module of most video analysis applications, the detection tasks of different categories of targets (such as vehicle detection, pedestrian detection, etc.) are regarded as different task video analysis requests.

[0146] Step S2: Establish a configuration selection model that considers the merging of multiple task requests.

[0147] Based on the multi-task video analysis system in step S1, a configuration selection model considering the merging of multi-task requests is established. The adjustable configuration options of the video analysis request can be expressed as a 4-tuple δ(r δ ,f δ ,mδ ,p δ ), the set of all available configurations is represented by Δ = {δ}; where the variable r δ Indicates video resolution; f δ Indicates the number of frames analyzed per second (FAPS); m δ represents the video analysis model; p δ Indicates the request execution location, p δ =0 means the request is processed locally on the edge server; δ =1 means the request is offloaded to the cloud for processing.

[0148] To save computing resources, we can select only a portion of frames for video analysis due to the temporal correlation between consecutive frames. FAPS is adjusted based on the accuracy requirements. If the accuracy requirement is not high, a lower FAPS can be used.

[0149] Configuration selection is achieved by adjusting the configuration tuple for each request. To this end, a binary variable in Indicates that the request q selects configuration δ. Since a video analysis request can only select one configuration, the following conditions must be met:

[0150]

[0151] To save communication and computing resources, a video analysis request merging scheme is adopted. That is, if multiple requests issued to the same camera select the same configuration (i.e., resolution, frame rate, model, and execution location), these requests can be merged into one. However, although request merging can save resource consumption, this merging process may reduce the performance of video analysis. Let's use an example to explain:

[0152] Suppose there are two requests for the same camera: "Request A" requires high accuracy and requires a high profile, while "Request B" requires high latency and requires a low profile. Merging "Request B" into "Request A" will increase the service latency of "Request B" and may even exceed its latency constraint. This is primarily because the video analysis process using the high profile takes longer in both transmission and processing time. Similarly, merging "Request A" into "Request B" sacrifices accuracy. Therefore, request merging must be carefully designed.

[0153] Introducing intermediate binary variables The request merging process is controlled by the "Boolean OR" operation. Where k is the camera number and δ is the configuration index. When at least one of is true, as shown below:

[0154]

[0155] Therefore, the bandwidth resource constraints between the camera and the edge server and the bandwidth resource constraints between the edge server and the cloud center are as follows:

[0156]

[0157] Among them, b δ Indicates the data size of one frame when using configuration δ, Indicates the total bandwidth from the camera to the edge server. Indicates the total bandwidth from the edge server to the cloud center.

[0158] Step S3: GPU resource modeling based on GPU virtualization technology.

[0159] Deep learning-based video analysis typically requires significant GPU resources, and its performance is often limited by GPU resource bottlenecks. Based on the edge computing-based multi-task video analysis system architecture model and the multi-task video analysis request merging model, we establish GPU computing resource and GPU memory resource models and constraints, and conduct GPU resource modeling based on GPU virtualization.

[0160] (1) GPU computing resources.

[0161] The Multi-Process Service (MPS) technology developed by NVIDIA can process multiple CUDA applications in parallel on a single GPU, thereby achieving efficient division and management of GPU resources. With MPS, the physical GPU computing power obtained by each application is specified, including the percentage of active threads (representing GPU computing power) and fixed device memory limits (representing GPU memory resources). MPS technology allows a physical GPU to be divided into multiple virtual GPUs, each of which is dedicated to running a video analysis model instance for a single video request. This design reduces frequent switching and preemption between GPU tasks, thereby improving the parallelism, isolation and stability of the service.

[0162] Let discrete variables Indicates the GPU computing resource allocation for each merge request. G is the total number of computing resource slices. The GPU computing capacity constraint of each edge server can be expressed as:

[0163] ;

[0164] in, Represents the total GPU computing resources of edge server n.

[0165] (2)GPU memory resources.

[0166] For GPU memory usage, we use offline analysis to obtain the memory usage required by each video analysis model. The GPU memory resource constraint can be expressed as:

[0167]

[0168] Among them, u δ Indicates the video memory usage of the video analysis model when using configuration δ; Indicates the total GPU memory resources of edge server n.

[0169] Step S4: Based on the video analysis model, perform video frame feature extraction by multi-feature fusion.

[0170] Traditional methods based on offline and online analysis require trying all configuration combinations on a small, representative video clip and calculating the accuracy to infer the accuracy over a period of time in the future. This type of method consumes a large amount of resources and a lot of time during analysis. The present invention adopts a video analysis accuracy prediction method based on the Transformer deep learning model, which consumes less resources and has high accuracy during online prediction. The video frame feature extraction based on multi-feature fusion can comprehensively characterize the current video characteristics, providing support for subsequent video analysis accuracy prediction based on the Transformer model.

[0171] The Transformer model has difficulty processing raw video streams directly, mainly due to the huge amount of video data and the sparsity of useful information in the raw data. The end-to-end Transformer model that learns directly from raw video data requires a complex architecture, high computing power requirements, and long inference latency, especially on edge servers with limited resources. To solve this problem, a video frame feature extraction method with multi-feature fusion is designed to make the Transformer model more lightweight and obtain better prediction results. The feature vector of each frame consists of three parts: visual features, current video analysis model size, and detection features, such as Figure 3 As shown in (A) in the figure, the details are as follows:

[0172] (1) Visual features.

[0173] Video content significantly impacts the accuracy of video analysis, such as changes in lighting conditions (daytime or nighttime), scene activity (busy or idle), and camera angle. Existing research typically uses standalone convolutional neural networks (CNNs), such as ResNet and DarkNet, to capture the relationship between video content and accuracy. In contrast, our new method reuses the intermediate feature maps generated by the backbone of the video analysis model, eliminating the need for additional CNN computations.

[0174] After obtaining these visual feature maps, a global average pooling (GAP) operation is used to reduce the spatial dimension. GAP calculates the average of all values ​​in each feature map channel and reduces the height and width to a single value per channel. Since the video analysis model is dynamically adjusted according to the scheduling results of the video request scheduler, the number of channels in the feature map and the dimension of the vector after GAP will also change. To this end, a repeated padding operation is used to pad all vectors to the dimension that matches the largest vector generated by the video analysis model, using f v Represents the visual features of a single video frame.

[0175] The dimension of the visual feature map will change according to the currently selected video analysis model, resulting in a dynamic change in the number of repeated fillings after the GAP. Therefore, a one-hot vector is used to encode the currently selected video analysis model, denoted as f m This vector can be used as f v Indicator of fill mode.

[0176] (2) Detection features.

[0177] The object detection results output by the video analysis model include category labels, bounding boxes, and confidence scores. This information can also be used to characterize video features, as follows:

[0178] A. Category Label: The number of targets has a significant impact on the accuracy of video analysis. Intuitively, a large number of targets indicates that the current scene may be very crowded and there may be serious occlusions between targets. Therefore, a low-configuration video analysis model may not be able to accurately detect all targets. n Indicates the number of detected objects as follows:

[0179] f a =[a,a1,…,a T ] (7);

[0180] Among them, the total number of targets a and the number of targets in each category a1 to a T ;gather Represents all target categories that can be detected, all category labels in one frame

[0181] B. Bounding box size: The size of the target will also significantly affect the accuracy of video analysis. For example, when dealing with small targets, low resolution will lead to lower analysis accuracy. Calculating all class labels in a frame is The bounding box size of the target is stored in vector B t In , each component represents the size of a target. Considering that the number of targets is dynamic, B t The dimension is not fixed, so use B t Statistics are used to represent target size information as follows:

[0182]

[0183] in, is the vector of the bounding box size; B t Represents a vector consisting of the bounding box sizes of all targets with category label t in a frame; min(B t )、max(B t )、mean(B t ) and var(B t ) represent B t The minimum, maximum, mean, and variance of .

[0184] C. Confidence score: The confidence score reflects the model's certainty about each detected object. A lower confidence score may indicate uncertainty about some detections in the scene, which may indicate that a higher-configuration model may be needed. The definition is similar to that of Describes the confidence score. In addition, An additional value is included, which represents the sum of the confidence scores of all objects with class label t, as shown below:

[0185]

[0186] Among them, C t is a vector of confidence scores of all targets with category labels t in a frame; sum(C t ) represents the sum of confidences of all objects with category label t; min(C t )、max(C t )、mean(C t ) and var(C t ) represent C t The minimum, maximum, mean, and variance of .

[0187] By concatenating the above vectors, we can get the detection feature f of a video frame. d , as shown below:

[0188]

[0189] The visual features, current model size, and detection features are concatenated together to fully represent a video frame. Before concatenation, these features are made to have similar dimensions through a linear projection layer, as shown below:

[0190] f=[f v ,f m ,f d ] (11);

[0191] Among them, f v Represents the visual features of a single video frame; f m Indicates that a single hot vector is used to encode the currently selected video analysis model, f d Represents the detection features of a video frame.

[0192] Step S5: Predict the video analysis accuracy based on the Transformer model.

[0193] The Transformer-based video analysis accuracy prediction model can predict the analysis accuracy of the current video under all configurations, which is the prerequisite for subsequently establishing and solving the optimization problem of minimizing the total system cost.

[0194] Because video streams are naturally continuous, changes in video content and the movement of objects can aid in analyzing accurate predictions. Existing methods have explored motion features such as speed, size change rate, and aspect ratio change rate. However, the selection and representation of these motion features are far from optimal. Considering the inherent spatiotemporal dependencies of video streams, a Transformer model is used to directly analyze feature sequences. The feature sequence is a sequence composed of the fused multi-features extracted for each video frame in step S4, arranged in chronological order.

[0195] Transformer is a sequence-to-sequence deep learning model that can effectively capture the temporal dependency information in video feature sequences. Figure 3 As shown in (B) in the figure, in order to explore the potential implicit features in the video frame sequence, the Transformer model is used to directly capture the temporal dependency information in the time window F of the feature sequence, as shown below:

[0196] F=f (1) ,f (2) ,…,f (β) (12);

[0197] H = Encoder(F) (13);

[0198] z = Pooling(H) (14);

[0199] a t =MLP(z),t=1,…,T (15);

[0200] a=[a1,…,a T ] (16);

[0201] Here, F represents the time window that directly captures the feature sequence using the Transformer model; the parameter β is the length of the window. This approach enables the model to learn and extract relevant features from the data, rather than relying solely on a predefined set of features.

[0202] By leveraging the self-attention mechanism, the Transformer model is able to capture local and global dependencies and contextual relationships in the data, making it very effective in processing complex temporal pattern tasks. The Transformer architecture consists of an encoder and a decoder. The Transformer encoder is used to extract feature representations. The encoder consists of multiple Transformer encoder layers, and positional encoding is added to provide the model with position information for each element in the sequence.

[0203] After the Transformer encoder, an average pooling operation is applied to aggregate features in the sequence and output a feature representation. The resulting feature vector is fed into the output layer, which consists of multiple multi-layer perceptrons (MLPs). Each MLP is responsible for predicting the accuracy of a specific object detection class under different configurations. Each MLP consists of two fully connected layers, mapping the aggregated features to the final accuracy prediction.

[0204] Step S6: Based on system modeling, resource constraints and prediction accuracy, an optimization problem for minimizing the total system cost is established.

[0205] S61. Establish user delay constraints.

[0206] The latency requirement constraints for each request are as follows:

[0207]

[0208] in, Indicates the service delay per frame when using configuration δ; Indicates the maximum end-to-end delay requirement.

[0209] Let n represent the edge server providing services for camera k, that is, n = o(k). Service delay per frame when using configuration δ As shown below:

[0210]

[0211] in, is the computational delay of the video analysis model for each frame, It is the transmission delay from the camera to the edge server. If the request is offloaded to the cloud, additional network delay will be added, which is the transmission delay from the edge to the cloud. and a fixed network propagation delay The composition takes into account the inherent latency of the public Internet connection between the edge servers and the cloud center.

[0212] Computational latency of the video analysis model per frame By video analysis model m δ , request execution position p δ and allocated GPU computing resources Decisions are as follows:

[0213]

[0214] The function T(·) is obtained through offline analysis.

[0215] Assuming that the bandwidth of the communication link between the camera and its edge server is evenly distributed, the transmission delay from camera k to edge server n is determined by the following formula:

[0216]

[0217] Similarly, the transmission delay from edge server n to the cloud can be calculated using the following formula:

[0218]

[0219] To prevent frame loss, the throughput of the selected model must be greater than the input frame rate. This constraint is expressed as:

[0220]

[0221] S62. Establish user accuracy constraints.

[0222] Video analysis accuracy refers to the average object detection accuracy across all frames within a small time interval α. The object accuracy for each video frame is evaluated by the average intersection-over-union (IoU) between the object detection results and the ground-truth annotations. Due to the dynamic adjustment of FAPS, only some frames may be analyzed. For unanalyzed frames, the object position in the previously analyzed frame is used as the current detection result.

[0223] The prediction accuracy of the selected configuration for each request Its user accuracy requirements should be met as follows:

[0224]

[0225] in, Indicates the prediction accuracy of the selected configuration for each request; Indicates the minimum accuracy requirement; the prediction accuracy is provided by the video analysis accuracy prediction module based on the Transformer model in step S5.

[0226] S63. Establish a problem statement.

[0227] The goal of this method is to minimize the total cost for edge service providers while meeting the accuracy, latency, and resource constraints required for edge video analysis. Let C be the total cost of the edge service provider, which includes internal cost I and external cost E, and is defined as follows:

[0228] Internal cost I comes from operating expenses and consists of three parts: transmission cost within the edge network, GPU memory cost of all edge servers, and GPU computing cost. Transmission cost is proportional to the amount of data transmitted, while GPU memory and computing cost are related to total memory usage and GPU computing resource consumption, respectively. t 、l u and l g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second. The internal cost per second is as follows:

[0229]

[0230] Among them, i trans is the internal transmission cost; i mem is the internal memory cost; i gpu is the internal GPU cost.

[0231] The emerging cloud server-less computing paradigm provides edge service providers with a pay-as-you-go billing model, i.e., external cost E. In this invention, the edge video analysis system needs to pay the cloud service provider for the resource usage of network bandwidth, GPU computing, and memory. Similar to the internal cost coefficient, the cloud computing coefficient ∈ t ,∈ u and ∈ g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second. The external cost per second is as follows:

[0232]

[0233] Among them, etrans is the external transmission cost; e mem is the external memory cost; e gpu is the external GPU cost. Based on this, the total system cost minimization problem can be expressed as:

[0234]

[0235] stC1,C2,C3,C4,C5,C6,C7,C8(27);

[0236]

[0237] Since the objective function and constraints C4 and C6 both contain quadratic terms This problem can be classified as a quadratically constrained quadratic programming (QCQP) problem. The complexity of solving the QCQP problem through semidefinite relaxation (SDR) is as follows:

[0238]

[0239] Where d is the variable dimension and ω is the accuracy of the solution. In this method, if the total number of video analysis requests is recorded as Q, the total number of configurations is recorded as H, and the total number of cameras is recorded as K, then d = H 2 QK (usually more than 10 7 ), making the problem unsolvable in a reasonable amount of time. Therefore, a more efficient algorithm is needed.

[0240] Step S7: Solve the system total cost minimization problem in step S5 based on a heuristic algorithm.

[0241] An efficient heuristic algorithm is proposed to solve the above system total cost minimization problem by coordinating configuration selection, request merging and GPU allocation.

[0242] The principle of heuristic algorithm is as follows Figure 4 As shown in Figure 1. The key principle of the proposed heuristic strategy is to start from an initial solution and gradually reduce the cost. In each iteration, the configuration selection x is first updated through a greedy idea (Lines 6-12), and then the corresponding GPU computing resource g is allocated (Line 13). If the current objective has not been significantly improved within θ iterations and all constraints are satisfied, the loop is terminated and the solution is output (Lines 15-18).

[0243] In order to meet the user's accuracy requirements, a configuration set Δ that meets the accuracy requirements is established in advance for each request q q (row 1), where In the subsequent phase, only each request Δ q Configuration within.

[0244] (1) Improvement of the solution based on the penalty function idea.

[0245] Inspired by the penalty function method for solving constrained optimization problems, the efficiency index φ is defined as the current target and constraint violation term σ(P L +P R ), as follows:

[0246]

[0247] Among them, σ is the penalty coefficient, P L represents the penalty term for delay, P R The penalty term representing the resource constraint.

[0248] The performance metric φ is zero for constraint violations when it is within the set of feasible solutions and becomes positive when it is outside the set. Therefore, constraints are penalized when they are not satisfied, and the performance metric is consistent with the actual target value when all constraints are satisfied. σ is initially set to a small value and then increased by a factor of p in each iteration (line 21), thereby gradually imposing a stronger penalty on constraint violations. As σ becomes larger, the penalty term becomes increasingly significant outside the set of feasible solutions, which gradually forces all constraints to be satisfied. As a result, the solution found will gradually converge to the solution of the original constrained problem.

[0249] Penalty term P for delay constraints and resource constraints L and P R , as shown below:

[0250]

[0251] in, Indicates the currently used bandwidth from the camera to the edge server; Indicates the currently used bandwidth from edge server to cloud server; Indicates the GPU resources currently used; represents the current memory resources used; the constraint violation term is zero when the performance metric φ is within the feasible solution set and becomes positive when it is outside the feasible solution set. Due to the different units and magnitudes of these metrics, the penalty term is defined as the normalized difference between the actual value and the constraint value. The penalty is zero when the corresponding constraint is fully satisfied.

[0252] Using a greedy strategy, the solution to x is gradually improved. In each iteration, the current solution is updated to the one with the minimum performance metric, and the system state is adjusted accordingly, continuing to explore new solutions in subsequent iterations. This process continues until all constraints are satisfied and no further improvements can be found.

[0253] Since there are a total of H QThere are many possible x, and it is very expensive to calculate the performance indicators of all possible solutions. Therefore, a multi-granularity solution update strategy is proposed, in which only one request or all the requested configurations on a single camera are updated in each iteration. Define φ q,δ To switch from the current configuration decision x to the new decision x (q,δ) The performance index of (q,δ) = φ means that only the configuration of request q is updated to configuration δ, while retaining the existing configurations of other requests. Similarly, φ k,δ Indicates switching from x to x (k,δ) The performance index of (k,δ) = means to update the configuration of all requests on camera k to configuration δ, and keep the configuration of requests on other cameras unchanged. Then, from {φ q,δ} and {φ k,δ This strategy reduces the search space to HQ + Hk, where Q is the total number of video analysis requests, H is the total number of configurations, and K is the total number of cameras.

[0254] This multi-granularity solution update strategy has two advantages: First, the strategy fully considers the merging of requests. Switching from x to x (k,δ) It is a coarse-grained step that changes the configuration of all requests on a camera and merges them into one request. Switching from x to x (q,δ) It is a fine-grained step that adjusts the configuration of only one request and considers merging the current request into other requests. Secondly, this strategy speeds up the convergence of the algorithm because the configurations of multiple requests can be decided simultaneously in one iteration.

[0255] (2) Two-stage decision making of configuration selection and GPU allocation.

[0256] First determine the current configuration selection decision x, and then use the greedy algorithm to determine the allocation of GPU computing resources g.

[0257] Since increasing GPU computing resources will reduce computing latency, a greedy strategy is used to allocate the minimum amount of GPU resources that meets the latency requirements for a specific request under its selected configuration. However, for some requests, if the latency requirements are too strict or the current configuration is too high, even allocating all GPU computing resources may not be enough to meet the latency requirements. In this case, Just set it to 1.

[0258] like Figure 5 As shown in Figure 1, we first use x to update the intermediate request merging variable s (line 1), and then calculate the delay from the camera to the edge server and from the edge server to the cloud center (line 2). Next, we calculate the GPU computing resource allocation for each pair of k and δ. If there is no request to select configuration δ on camera k, then Set to 0 (line 16); otherwise, for merge requests The requests in (including all requests for configuration δ on camera k) are combined to determine the maximum computational latency that can be tolerated by the merged requests (lines 6-8). If the latency requirement cannot be met even if all GPU computing resources are allocated, Set to 1 and mark the constraint satisfaction flag η as False (lines 9-11). Otherwise, Set it to the minimum value that satisfies the latency requirement and throughput constraint, and set η to True (Lines 12-14).

[0259] Therefore, the present invention adopts the above-mentioned request scheduling method for multi-task edge video analysis, aiming to minimize the total system cost through dynamic configuration adjustment and GPU resource allocation under the premise of meeting user accuracy and latency requirements; capture the current video content and inter-frame time dependency through a prediction module based on the Transformer model, and accurately predict the relationship between configuration and analysis accuracy in real time; adopt video analysis request merging technology to efficiently utilize limited network and computing resources, realize the sharing of network transmission and model calculation, and reduce resource consumption; based on the configuration-accuracy analysis results, use heuristic algorithms to efficiently schedule video analysis requests, dynamically adjust video configuration, model selection, request merging, task deployment and GPU resource allocation to optimize system performance and reduce costs.

[0260] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A request scheduling method for multi-task edge video analysis, characterized in that: The following steps are involved: Step S1: Build a multi-task video analysis system based on edge computing: Multi-task edge video analysis adopts a cloud-edge-end collaborative architecture, which mainly consists of three layers: end device camera, edge server and cloud center; Step S2: Establishing a configuration selection model considering multi-task request merging: Based on the multi-task video analysis system in step S1, a multi-task request merging model is constructed for communication and computing resources to optimize the configuration selection process; Step S3: GPU resource modeling based on GPU virtualization technology: logically partitioning the physical GPU based on GPU virtualization technology to achieve dynamic allocation of GPU computing resources and video memory resources, and establishing a corresponding resource constraint model; Step S4: Based on the video analysis model, perform multi-feature fusion video frame feature extraction: Based on the visual features of the video analysis model and the detection features based on the video analysis results, the video content and the detection results are comprehensively considered to fully reflect the current video characteristics; wherein, the intermediate feature map of the video analysis model is used as the visual feature, and the video analysis results are extracted as the detection feature; Step S5: Predicting video analysis accuracy based on the Transformer model: Using the Transformer model to process the time series composed of the video frame features in step S4, capturing sequence features and temporal dependencies, and then processing the feature representations extracted by the Transformer through a multi-layer perceptron to predict the relationship between the camera configuration and analysis accuracy in the current scene; Step S6: Establish an optimization problem for minimizing the total system cost: define the internal cost of the edge computing network and the external cost of task offloading based on serverless cloud computing, consider the latency and accuracy requirements, and construct a quadratically constrained quadratic programming problem for minimizing the total system cost; Step S7: Solve the total system cost minimization problem in step S5 based on a heuristic algorithm: Based on the predicted configuration-accuracy relationship, adopt a multi-granularity heuristic solution improvement algorithm, improve the feasibility and quality of the solution simultaneously through the penalty function idea, and combine it with a greedy algorithm to achieve optimal allocation of GPU resources.

2. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: A system framework for a request scheduling method for multi-task edge video analysis, consisting of a configuration selection and GPU resource modeling module, a video analysis accuracy prediction module based on the Transformer model, an optimization problem module for minimizing the total system cost, and a request scheduling algorithm module based on heuristic ideas. Among them, the configuration selection and GPU resource modeling modules establish a multi-task edge video analysis architecture with cloud-edge-end collaboration, and design a configuration selection model and GPU resource allocation model that considers the merging of multi-task requests; Based on Transforme r The model's video analysis accuracy prediction module predicts the analysis accuracy corresponding to each camera configuration in the current scene based on the relationship between video content and video frame changes; The optimization problem module for minimizing the total system cost is based on the system model and accuracy information, taking into account the latency and accuracy requirements, and constructing a quadratic constrained quadratic programming problem for minimizing the total system cost. A multi-granularity heuristic solution improvement algorithm is designed based on the request scheduling module of the heuristic algorithm. The feasibility and quality of the solution are improved through the penalty function idea, and the greedy algorithm is combined to achieve the optimal allocation of GPU resources.

3. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S1, the workflow of the multi-task video analysis system based on edge computing is as follows: A global scheduler schedules video analysis requests, optimizing configuration selection, request merging, and GPU resource allocation. It treats detection tasks for different categories of targets as separate video analysis requests. Among them, the scheduler relies on four inputs: the accuracy of each video stream provided online by the video analysis accuracy prediction module; the data size of different frames obtained through offline analysis and the GPU memory usage of different video analysis models; user-defined accuracy and latency requirements; and real-time resource consumption of bandwidth, GPU and video memory.

4. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S2, a configuration selection model considering multi-task request merging is established, as follows: The configuration options that can be adjusted in a video analysis request are represented as a 4-tuple δ(r δ ,fδ,m δ , p δ ), the set of all available configurations is represented by Δ = {δ}; where the variable r δ Indicates video resolution; f δ Indicates the number of frames per second for video analysis FAPS; m δ represents the video analysis model; p δ Indicates the requested execution location. When p δ = 0 means the request is processed locally on the edge server. δ =1 means the request is offloaded to the cloud for processing; To implement configuration selection by adjusting the configuration tuple for each request, a binary variable is introduced. in Indicates that request q selects configuration δ. Since a video analysis request can only select one configuration, the following conditions must be met: in, represents the set of requests on all K cameras, Represents the camera collection of the end device; In order to save communication and computing resources, a video analysis request merging scheme is adopted to introduce intermediate binary variables. The request merging process is controlled by the "Boolean OR" operation; where k is the camera number; when When at least one of is true, as shown below: in, represents the request set on camera k; Therefore, the bandwidth resource constraints between the camera and the edge server and the bandwidth resource constraints between the edge server and the cloud center are as follows: Among them, b δ Indicates the data size of one frame when using configuration δ, Indicates the total bandwidth from the camera to the edge server. represents the total bandwidth from the edge server to the cloud center; the fixed function o(·) defines the mapping between the camera and the edge server, where the camera Connecting to the Edge Server Represents a collection of edge servers.

5. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S3, based on the edge computing-based multi-task video analysis system architecture model and the multi-task video analysis request merging model, GPU computing resource and GPU memory resource models and constraints are established, and GPU resource modeling based on GPU virtualization is performed, as follows: (1) GPU computing resources: Let discrete variables Indicates the GPU computing resource allocation for each merge request; G is the total number of computing resource slices; the GPU computing capacity constraint of each edge server can be expressed as: in, represents the total GPU computing resources of edge server n; (2) GPU memory resources: For GPU memory usage, the memory usage required by each video analysis model is obtained through offline analysis; the GPU memory resource constraint can be expressed as: Among them, uδ represents the video memory usage of the video analysis model when using configuration δ; Indicates the total GPU memory resources of edge server n.

6. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S4, based on the video analysis model, multi-feature fusion video frame feature extraction is performed, where the feature vector of each frame consists of three parts: visual features, current video analysis model size, and detection features, as follows: (1) Visual features: Reuse the intermediate feature maps generated by the backbone of the video analysis model to obtain the visual feature map, and then use the global average pooling (GAP) operation to reduce the spatial dimension; use repeated padding operations to fill all vectors to the dimension that matches the vector generated by the largest video analysis model, and use f v Represents the visual features of a single video frame; Use one-hot vector x Encode the currently selected video analysis model, recorded as f m ; where f m As f v Indicator of filling mode; (2) Detection features: The target detection results output by the video analysis model include category labels, bounding boxes, and confidence scores, which are used to characterize video features as follows: A. Category label: Use vector f a Indicates the number of detected objects as follows: f a =[a, a1,..., a T ] (7); Among them, the total number of targets a and the number of targets in each category a1 to a T ;gather Represents all target categories that can be detected, all category labels in one frame B. Bounding box size: Use B t The statistics of represent the target size information, as shown below: in, is the vector of the bounding box size; B t Represents a vector consisting of the bounding box sizes of all targets with category label t in a frame; min(B t )、max(B t )、mean(B t ) and var(B t ) represent B t the minimum, maximum, mean, and variance of ; C. Confidence score: reflects the model's certainty about each detected target, using the statistical vector Describe the confidence scores as follows: Among them, C t is a vector of confidence scores of all targets with category labels t in a frame; sum(C t ) represents the sum of confidences of all objects with category label t; min(C t )、max(C t )、mean(C t ) and var(C t ) represent C t the minimum, maximum, mean, and variance of ; By concatenating the above vectors, we can get the detection feature f of a video frame. d , as shown below: The visual features, current model size, and detection features are concatenated together to fully represent a video frame; before concatenation, these features are made to have similar dimensions through a linear projection layer, as shown below: f=[f v ,f m ,f d ] (11); Among them, f v Represents the visual features of a single video frame; f m Indicates that a single hot vector is used to encode the currently selected video analysis model, f d Represents the detection features of a video frame.

7. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S5, the video analysis accuracy is predicted based on the Transformer model as follows: To explore the potential implicit features in the video frame sequence, the Transformer model is used to directly capture the temporal dependency information in the time window F of the feature sequence, as shown below: F=f (1) ,f (2) ,...,f (β) (12); H=Encoder(F)(13); z = Pooling(H)(14); a t =MLP(z),t=1,...,T (15); a=[a1,...,a T ] (16); Where F represents the time window of the feature sequence directly captured by the Transformer model; the parameter β is the length of the window; by utilizing the self-attention mechanism, the Transformer model captures local and global dependencies and contextual relationships in the data, making it very effective in processing complex temporal pattern tasks; Transforme r The architecture consists of an encoder and a decoder; using Transforme r The encoder is composed of multiple Transformer encoder layers to extract feature representations, and position encoding is added to provide the model with the position information of each element in the sequence; After the Transformer encoder, an average pooling operation is applied to aggregate the features in the sequence and output a feature representation. The resulting feature vector is input to the output layer, which consists of multiple multi-layer perceptrons (MLPs). Each MLP is responsible for predicting the accuracy of a specific target detection category under different configurations. Each MLP consists of two fully connected layers that map the aggregated features to the final accuracy prediction.

8. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S6, based on system modeling, resource constraints, and prediction accuracy, an optimization problem for minimizing the total system cost is established, as follows: S61. Establish user delay constraints, and the delay requirement constraints for each request are as follows: in, Indicates the service delay per frame when using configuration δ; Indicates the maximum end-to-end delay requirement; To prevent frame loss, the throughput of the selected model must be greater than the input frame rate, so the constraint expression is: S62. Establish user accuracy constraint: the prediction accuracy of the selected configuration for each request Its user accuracy requirements should be met as follows: in, Indicates the prediction accuracy of the selected configuration for each request; Indicates the minimum accuracy requirement; the prediction accuracy is provided by the video analysis accuracy prediction module based on the Transformer model in step S5; S63. Problem Statement: The goal is to minimize the total cost for the edge service provider while satisfying the accuracy, latency, and resource constraints required for edge video analysis. Let C be the total cost of the edge service provider, which includes internal costs and external costs E. The specific definitions are as follows: Internal costs, including three parts: transmission costs within the edge network, GPU memory costs of all edge servers, and GPU computing costs; usage coefficient ι t 、ι u and ι g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second; the internal cost per second is as follows: Among them, i trans is the internal transmission cost; i mem is the internal memory cost; i gpu is the internal GPU cost; External costs E include: the edge video analysis system needs to pay fees to the cloud service provider for network bandwidth, GPU computing and memory resource usage; cloud computing coefficient ∈ t ,∈ u and ∈ g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second; the external cost per second is as follows: Among them, e trans is the external transmission cost; e mem is the external memory cost; e gpu Cost of external GPU; Based on this, the system total cost minimization problem is expressed as: stC1, C2, C3, C4, C5, C6, C7, C8(27); Where G is the total number of computing resource slices; The above problem is classified as a quadratically constrained quadratic programming (QCQP) problem. The complexity of solving the QCQP problem by semidefinite relaxation SDR is as follows: Where d is the variable dimension and ω is the accuracy of the solution.

9. The method for request scheduling for multi-task edge video analysis according to claim 1, characterized in that: In step S7, based on a heuristic algorithm, the total system cost minimization problem in step S5 is solved by coordinating configuration selection, request merging, and GPU allocation, as follows: The key principle of the heuristic strategy is to start from an initial solution and gradually reduce the cost. In each iteration, the configuration selection x is first updated through a greedy approach, and then the corresponding GPU computing resources g are allocated. If the current objective has not been significantly improved within θ iterations and all constraints are satisfied, the loop is terminated and the solution is output. In order to meet the user's accuracy requirements, a configuration set Δ that meets the accuracy requirements is established in advance for each request q q ,in In the subsequent phase, only each request Δ q Configuration within; (1) Improvement of the solution based on the penalty function idea; Inspired by the penalty function method for solving constrained optimization problems, the efficiency index φ is defined as the current target and constraint violation term σ(P L +P R ), as follows: Among them, σ is the penalty coefficient, P L represents the penalty term for delay, P R Penalty term representing resource constraints; Penalty term P for delay constraints and resource constraints L and P R , as shown below: in, Indicates the currently used bandwidth from the camera to the edge server; Indicates the currently used bandwidth from edge server to cloud server; Indicates the GPU resources currently used; Represents the currently used memory resources; when the performance index φ is within the feasible solution set, the constraint violation item is zero, and becomes positive when it exceeds the feasible solution set; Based on the multi-granularity solution update strategy, each iteration only updates one request or the configuration of all requests on a single camera; define φ q，δ To switch from the current configuration decision x to the new decision x (q，δ) The performance index of (q，δ) Indicates that only the configuration of request q is updated to configuration δ, while retaining the existing configurations of other requests; φ k，δ Indicates switching from x to x (k，δ) The performance index of (k，δ) Indicates that the configuration of all requests on camera k is updated to configuration δ, and the configuration of requests on other cameras remains unchanged; then, from {φ q，δ } and {φ k，δ }, and select the solution with the smallest performance index from the union of the two. This strategy reduces the search space to HQ+HK, where Q is the total number of video analysis requests, H is the total number of configurations, and K is the total number of cameras. (2) Two-stage decision-making for configuration selection and GPU allocation: First, determine the current configuration selection decision x, and then use the greedy algorithm to determine the allocation of GPU computing resources g; First, use x to update the intermediate request merge variable s, and then calculate the delay from the camera to the edge server and from the edge server to the cloud center; Next, calculate the GPU computing resource allocation for each pair of k and δ If there is no request to select configuration δ on camera k, then Set to 0; otherwise for merge requests Requests in, determine the maximum computational delay that can be tolerated by merging requests; If the latency requirement cannot be met even if all GPU computing resources are allocated, Set to 1 and mark the constraint satisfaction flag η as False; otherwise Set it to the minimum value that satisfies the latency requirement and throughput constraint, and set η to True.

Images