Request scheduling method for multi-task edge video analysis

Abstract

The invention belongs to the technical field of edge video analysis, and discloses a multi-task edge video analysis-oriented request scheduling method, which comprises the following steps of: constructing a multi-task video analysis system based on edge calculation; establishing a configuration selection model considering multi-task request merging; gPU resource modeling is carried out based on a GPU virtualization technology; performing multi-feature fusion video frame feature extraction based on the video analysis model; performing video analysis accuracy prediction based on a Transform model; establishing a system total cost minimization optimization problem; and solving the total cost minimization problem of the system based on a heuristic algorithm. According to the multi-task edge video analysis-oriented request scheduling method, on the premise of meeting user accuracy and time delay requirements, the total cost of a system is minimized, the relation between configuration and analysis accuracy is accurately predicted in real time, limited network and computing resources are efficiently utilized, network transmission sharing is achieved, and resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of edge video analysis, and in particular to a request scheduling method for multi-task edge video analysis. Background Art

[0002] Currently, more than one billion cameras deployed around the world generate massive amounts of video data every day, which are widely used in fields such as public safety, traffic monitoring, and autonomous driving. However, traditional manual video analysis methods usually rely on video playback for post-analysis, which has the disadvantages of low accuracy, high false negative rate, and inability to respond in real time. The development of artificial intelligence and computer vision technology has made highly accurate intelligent video analysis possible. These methods are usually based on computationally complex deep neural network reasoning and are difficult to run directly on camera devices. Traditional intelligent video analysis mainly adopts centralized cloud computing architecture, which has problems such as network congestion, high response latency, and single point failure. With the development of edge computing technology, intelligent video analysis methods are gradually evolving towards edgeization. By performing computing tasks at the edge of the network close to the data generation location, the response speed can be significantly improved and the network pressure can be relieved.

[0003] Existing edge computing-based intelligent video analysis systems usually only target a single task, such as vehicle tracking, traffic accident detection, or pedestrian counting. However, in actual applications, there are often multiple video analysis requirements on the same camera. Deploying an independent system for each task will lead to cost and resource waste. Multi-task edge video analysis is an emerging architecture that supports executing multiple tasks simultaneously on the same video stream, effectively reducing the overhead of repeated deployment.

[0004] In edge video analysis, the key to improving system performance and achieving efficient and high-quality data analysis is to adaptively adjust configuration parameters and optimize task scheduling based on dynamic computing and communication resources. Adjustable configuration parameters include video resolution, frame rate, deep learning model selection, and task execution location. Request scheduling for edge video analysis is usually modeled as a mathematical optimization problem. Before optimization, it is necessary to obtain the video analysis accuracy of each request under various configuration parameters in advance, which is called "configuration-accuracy analysis."

[0005] However, existing multi-task edge video analysis methods still face the following challenges:

[0006] (1) Configuration-accuracy analysis suffers from inaccuracy and high computational cost: Existing methods usually ignore the dynamic changes of video content such as illumination changes and object speed, and most of them rely on offline or periodic methods, making it difficult to achieve an ideal balance between accuracy and efficiency.

[0007] (2) There is high computational redundancy when processing a large number of video analysis requests: Multiple analysis requests for the same video may involve repeated network transmission and deep neural network inference, and existing methods fail to fully share network and computing resources.

[0008] (3) Complexity of joint configuration adjustment and GPU resource allocation: Existing studies often ignore the importance of GPU allocation in improving system performance. The complex correlation between multiple configuration parameters further increases the difficulty of decision-making in joint configuration adjustment and GPU resource allocation. Summary of the invention

[0009] The purpose of the present invention is to provide a request scheduling method for multi-task edge video analysis, aiming to minimize the total system cost through dynamic configuration adjustment and GPU resource allocation under the premise of meeting the accuracy and latency requirements of users; capture the current video content and inter-frame time dependency through a prediction module based on a Transformer model, and accurately predict the relationship between configuration and analysis accuracy in real time; adopt video analysis request merging technology to efficiently utilize limited network and computing resources, realize the sharing of network transmission and model calculation, and reduce resource consumption; based on the configuration-accuracy analysis results, use a heuristic algorithm to efficiently schedule video analysis requests, dynamically adjust video configuration, model selection, request merging, task deployment and GPU resource allocation, so as to optimize system performance and reduce costs.

[0010] To achieve the above object, the present invention provides a request scheduling method for multi-task edge video analysis, comprising the following steps:

[0011] Step S1, building a multi-task video analysis system based on edge computing: Multi-task edge video analysis adopts a cloud-edge-end collaborative architecture, which is mainly composed of three levels, namely, end device camera, edge server and cloud center;

[0012] Step S2, establishing a configuration selection model considering multi-task request merging: based on the multi-task video analysis system of step S1, construct a multi-task request merging model for communication and computing resources to optimize the configuration selection process;

[0013] Step S3, GPU resource modeling based on GPU virtualization technology: logically partitioning the physical GPU based on GPU virtualization technology, realizing dynamic allocation of GPU computing resources and video memory resources, and establishing a corresponding resource constraint model;

[0014] Step S4, based on the video analysis model, perform video frame feature extraction with multi-feature fusion: based on the visual features of the video analysis model and the detection features based on the video analysis results, comprehensively consider the video content and the detection results to fully reflect the current video characteristics; wherein the intermediate feature map of the video analysis model is used as the visual feature, and the video analysis results are extracted as the detection feature;

[0015] Step S5, predicting the video analysis accuracy based on the Transformer model: using the Transformer model to process the time series composed of the video frame features in step S4, capturing the sequence features and time dependency, and then processing the feature representation extracted by the Transformer through a multi-layer perceptron to predict the relationship between the camera configuration and the analysis accuracy in the current scene;

[0016] Step S6, establishing an optimization problem for minimizing the total system cost: defining the internal cost of the edge computing network and the external cost of task offloading based on serverless cloud computing, considering the latency and accuracy requirements, and constructing a quadratic constrained quadratic programming problem for minimizing the total system cost;

[0017] Step S7, based on the heuristic algorithm, solve the total cost minimization problem of the system in step S5: based on the predicted configuration-accuracy relationship, adopt a multi-granularity heuristic solution improvement algorithm, improve the feasibility and quality of the solution at the same time through the penalty function idea, and combine the greedy algorithm to achieve optimal allocation of GPU resources.

[0018] Preferably, a system framework of a request scheduling method for multi-task edge video analysis is composed of a configuration selection and GPU resource modeling module, a video analysis accuracy prediction module based on a Transformer model, an optimization problem module for minimizing the total cost of the system, and a request scheduling algorithm module based on heuristic thinking;

[0019] Among them, the configuration selection and GPU resource modeling modules establish a multi-task edge video analysis architecture with cloud-edge-end collaboration, and design a configuration selection model and GPU resource allocation model that considers the merging of multi-task requests;

[0020] The Transformer-based video analysis accuracy prediction module predicts the analysis accuracy of each camera configuration in the current scene based on the relationship between video content and video frame changes.

[0021] The optimization problem module for minimizing the total system cost is based on the system model and accuracy information, comprehensively considers the delay and accuracy requirements, and constructs a quadratic constrained quadratic programming problem for minimizing the total system cost;

[0022] A multi-granularity heuristic solution improvement algorithm is designed based on the request scheduling module of the heuristic algorithm. The feasibility and quality of the solution are improved through the penalty function idea, and the greedy algorithm is combined to achieve the optimal allocation of GPU resources.

[0023] Preferably, the workflow of the multi-task video analysis system based on edge computing is:

[0024] Video analysis requests are scheduled through a global scheduler to optimize configuration selection, request merging, and GPU resource allocation; detection tasks for different categories of targets are treated as different task video analysis requests;

[0025] Among them, the scheduler relies on four inputs, namely the accuracy of each video stream provided online by the video analysis accuracy prediction module; the data size of different frames obtained through offline analysis and the GPU memory usage of different video analysis models; user-defined accuracy and latency requirements; and real-time resource consumption of bandwidth, GPU and video memory.

[0026] Preferably, in step S2, a configuration selection model considering merging of multi-task requests is established, specifically as follows:

[0027] The adjustable configuration options of the video analysis request are represented as a 4-tuple δ(r δ ,f δ ,m δ ,p δ ), the set of all available configurations is represented by Δ = {δ}; where the variable r δ Indicates the video resolution; f δ Indicates the number of video analysis frames per second FAPS; m δ represents the video analysis model; p δ Indicates the requested execution location. When p δ = 0 means the request is processed locally on the edge server. δ =1 means the request is offloaded to the cloud for processing;

[0028] To implement configuration selection by adjusting the configuration tuple for each request, a binary variable is introduced in Indicates that request q selects configuration δ. Since only one configuration can be selected for a video analysis request, the following conditions must be met:

[0029]

[0030] in, represents the set of requests on all K cameras, Represents the camera collection of the terminal device;

[0031] In order to save communication and computing resources, a video analysis request merging scheme is adopted to introduce intermediate binary variables. The request merging process is controlled by the "Boolean OR" operation; where k is the camera number; when When at least one of is true, as shown below:

[0032]

[0033] in, represents the request set on camera k;

[0034] Therefore, the bandwidth resource constraints between the camera and the edge server and the bandwidth resource constraints between the edge server and the cloud center are as follows:

[0035]

[0036] Among them, b δ Indicates the data size of one frame when using configuration δ, Indicates the total bandwidth from the camera to the edge server. represents the total bandwidth from the edge server to the cloud center; the fixed function o(·) defines the mapping between the camera and the edge server, where the camera Connecting to the Edge Server {1,2,…,N} represents the set of edge servers.

[0037] Preferably, in step S3, based on the multi-task video analysis system architecture model based on edge computing and the multi-task video analysis request merging model, GPU computing resource and GPU memory resource models and constraints are established, and GPU resource modeling based on GPU virtualization is performed, as follows:

[0038] (1) GPU computing resources: Let discrete variables Indicates the GPU computing resource allocation for each merge request; G is the total number of computing resource slices; the GPU computing capacity constraint of each edge server can be expressed as:

[0039]

[0040] in, represents the total GPU computing resources of edge server n;

[0041] (2) GPU memory resources: For GPU memory usage, the memory usage required by each video analysis model is obtained through offline analysis; the GPU memory resource constraint can be expressed as:

[0042]

[0043] Among them, u δ Indicates the video memory usage of the video analysis model when using configuration δ; Indicates the total GPU memory resources of edge server n.

[0044] Preferably, in step S4, based on the video analysis model, video frame feature extraction with multi-feature fusion is performed, wherein the feature vector of each frame is composed of three parts, namely, visual features, current video analysis model size and detection features, as follows:

[0045] (1) Visual features: Reuse the intermediate feature maps generated by the backbone of the video analysis model to obtain the visual feature map, and then use the global average pooling (GAP) operation to reduce the spatial dimension; use repeated padding operations to fill all vectors to the dimension that matches the largest vector generated by the video analysis model, and use f v Represents the visual features of a single video frame; uses a one-hot vector to encode the currently selected video analysis model, denoted as f m ; where f m As f v Indicator of filling mode;

[0046] (2) Detection features: The object detection results output by the video analysis model include category labels, bounding boxes, and confidence scores, which are used to characterize video features as follows:

[0047] A. Category label: Use vector f a Indicates the number of detected objects as follows:

[0048] f a =[a,a1,…,a T ] (7);

[0049] Among them, the total number of targets A and the number of targets in each category a1 to a T ;gather Represents all detectable target categories, all category labels in one frame

[0050] B. Bounding box size: Use B t The statistics of represent the target size information, as follows:

[0051]

[0052] in, is the vector of the bounding box size; B t Represents a vector of bounding box sizes of all targets with category label t in a frame; min(B t)、max(B t )、mean(B t ) and var(B t ) represent B t The minimum, maximum, mean and variance of

[0053] C. Confidence score: reflects the model's certainty about each detected target, using the statistical vector Describe the confidence scores as follows:

[0054]

[0055] Among them, C t is a vector of confidence scores of all targets with category label t in a frame; sum(C t ) represents the sum of confidences of all objects with category label t; min(C t )、max(C t )、mean(C t ) and var(C t ) represent C t The minimum, maximum, mean and variance of

[0056] By concatenating the above vectors, we can get the detection feature f of a video frame. d , as shown below:

[0057]

[0058] The visual features, current model size, and detection features are concatenated together to fully represent a video frame; before concatenation, these features are made to have similar dimensions through a linear projection layer, as shown below:

[0059] f=[f v ,f m ,f d ] (11);

[0060] Among them, f v Represents the visual features of a single video frame; f m Indicates the use of a one-hot vector to encode the currently selected video analysis model, f d Represents the detected features of a video frame.

[0061] Preferably, in step S5, the video analysis accuracy prediction is performed based on the Transformer model, as follows:

[0062] To explore the potential implicit features in the video frame sequence, the Transformer model is used to directly capture the time dependency information in the time window F of the feature sequence, as shown below:

[0063] F=f (1) ,f (2) ,…,f (β) (12);

[0064] H = Encoder (F) (13);

[0065] z = Pooling(H) (14);

[0066] a t =MLP(z),t=1,…,T (15);

[0067] a=[a1,…,a T ] (16);

[0068] Where F represents the time window of the feature sequence directly captured by the Transformer model; the parameter β is the length of the window; by using the self-attention mechanism, the Transformer model captures local and global dependencies and contextual relationships in the data, making it very effective in processing complex temporal pattern tasks;

[0069] The Transformer architecture consists of an encoder and a decoder. The Transformer encoder is used to extract feature representations. The encoder consists of multiple Transformer encoder layers, and position encoding is added to provide the model with the position information of each element in the sequence.

[0070] After the Transformer encoder, an average pooling operation is applied to aggregate the features in the sequence and output the feature representation. The resulting feature vector is input to the output layer, which consists of multiple multi-layer perceptrons (MLPs); each MLP is responsible for predicting the accuracy of a specific target detection category under different configurations; each MLP consists of two fully connected layers that map the aggregated features to the final accuracy prediction.

[0071] Preferably, in step S6, based on system modeling, resource constraints and prediction accuracy, an optimization problem of minimizing the total system cost is established, as follows:

[0072] S61. Establish user delay constraints and delay requirement constraints for each request, as shown below:

[0073]

[0074] in, Indicates the service delay per frame when using configuration δ; Indicates the maximum end-to-end delay requirement;

[0075] To prevent frame loss, the throughput of the selected model must be greater than the rate of the input frames, so the constraint expression is:

[0076]

[0077] S62. Establish user accuracy constraint: the prediction accuracy of the selected configuration for each request Its user accuracy requirements should be met as follows:

[0078]

[0079] in, Indicates the prediction accuracy of the selected configuration for each request; represents the minimum accuracy requirement; the prediction accuracy is provided by the video analysis accuracy prediction module based on the Transformer model in step S5;

[0080] S63. Problem statement: The goal is to minimize the total cost for the edge service provider while satisfying the accuracy, latency, and resource constraints required for edge video analysis. Let C be the total cost of the edge service provider, including internal cost I and external cost E, which are defined as follows:

[0081] The internal cost I includes three parts: the transmission cost within the edge network, the GPU memory cost of all edge servers, and the GPU computing cost; the usage coefficient ι t , u and g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second; the internal cost per second is as follows:

[0082]

[0083] Among them, i trans is the internal transmission cost; i mem is the internal memory cost; i gpu is the internal GPU cost;

[0084] The external cost E includes: the edge video analysis system needs to pay the cloud service provider for the network bandwidth, GPU computing and memory resource usage; the cloud computing coefficient ∈ t ,∈ u and ∈ g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second; the external cost per second is as follows:

[0085]

[0086] Among them, e trans is the external transmission cost; mem is the external memory cost; e gpu Cost of external GPU;

[0087] Based on this, the system total cost minimization problem is expressed as:

[0088]

[0089] stC1,C2,C3,C4,C5,C6,C7,C8(27);

[0090]

[0091] Where G is the total number of computing resource slices;

[0092] The above problem is classified as a quadratic constrained quadratic programming (QCQP) problem. The complexity of solving the QCQP problem by semidefinite relaxation SDR is as follows:

[0093]

[0094] Where d is the variable dimension and ω is the precision of the solution.

[0095] Preferably, based on a heuristic algorithm, the total system cost minimization problem in step S5 is solved by coordinating configuration selection, request merging, and GPU allocation, as follows:

[0096] The key principle of the heuristic strategy is to start from an initial solution and gradually reduce the cost. In each iteration, the configuration selection x is first updated through a greedy idea, and then the corresponding GPU computing resource g is allocated. If the current goal has not been significantly improved within θ iterations and all constraints are satisfied, the loop is terminated and the solution is output.

[0097] In order to meet the user's accuracy requirements, a configuration set Δ that meets the accuracy requirements is established in advance for each request q q ,in In the subsequent phase, only each request Δ q Configuration within;

[0098] (1) Improvement of the solution based on the penalty function idea;

[0099] Inspired by the penalty function method for solving constrained optimization problems, the performance index φ is defined as the current target and the constraint violation term σ(P L +P R ), as follows:

[0100]

[0101] Among them, σ is the penalty coefficient, P L represents the penalty term for delay, P R The penalty term represents the resource constraint;

[0102] The penalty term P for delay constraints and resource constraints L and P R , as shown below:

[0103]

[0104] in, Indicates the currently used bandwidth from camera to edge server; Indicates the currently used edge server to cloud server bandwidth; Indicates the GPU resources currently used; Represents the memory resources currently used; when the performance index φ is in the feasible solution set, the constraint violation item is zero, and becomes positive when it exceeds the feasible solution set;

[0105] Based on the multi-granularity solution update strategy, each iteration only updates one request or the configuration of all requests on a single camera; define φ q,δ To switch from the current configuration decision x to the new decision x (q,δ) The performance index of (q,δ) Indicates that only the configuration of request q is updated to configuration δ, while retaining the existing configurations of other requests; φ k,δ Indicates switching from x to x (k,δ) The performance index of (k,δ) = means updating the configuration of all requests on camera k to configuration δ, and keeping the configuration of requests on other cameras unchanged; then, from {φ q,δ} and {φ k,δ}, and select the solution with the smallest performance index; this strategy reduces the search space to HQ+HK; where Q is the total number of video analysis requests, H is the total number of configurations, and K is the total number of cameras;

[0106] (2) Two-stage decision-making for configuration selection and GPU allocation: First, determine the current configuration selection decision x, and then use the greedy algorithm to determine the allocation of GPU computing resources g;

[0107] First, use x to update the intermediate request merge variable s, and then calculate the delay from the camera to the edge server and from the edge server to the cloud center;

[0108] Next, the GPU computing resource allocation for each pair of k and δ is calculated If there is no request to select configuration δ on camera k, then Set to 0; otherwise for merge requests Requests in the merged request, determine the maximum computational delay that can be tolerated;

[0109] If the latency requirement cannot be met even if all GPU computing resources are allocated, Set to 1, and mark the constraint satisfaction flag η as False; otherwise Set it to the minimum value that satisfies the latency requirement and throughput constraint, and set η to True.

[0110] Therefore, the present invention adopts the above-mentioned request scheduling method for multi-task edge video analysis, and the beneficial effects are as follows:

[0111] (1) The edge computing architecture of the present invention for multi-task video analysis can effectively reduce the overhead caused by repeated deployment and improve resource utilization efficiency compared with the traditional single-task video analysis system;

[0112] (2) The configuration-accuracy relationship prediction model based on the Transformer model can comprehensively consider the relationship between video content and video frame changes to achieve efficient and high-precision video analysis accuracy prediction;

[0113] (3) The present invention realizes the merging of video analysis requests to fully share network and computing resources, effectively addressing the problem of high computing redundancy when processing a large number of video analysis requests;

[0114] (4) The heuristic algorithm for edge video analysis request scheduling in this invention effectively solves the complex joint optimization problem of configuration selection and GPU resource allocation, and can significantly reduce the total system cost while meeting the user's accuracy and latency requirements;

[0115] (5) The present invention provides ideas for the allocation and scheduling of discrete resources, broadening the application scenarios of resource management in edge computing;

[0116] (6) The architecture and method proposed in the present invention support a wide range of video analysis application scenarios, such as intelligent transportation, smart cities, industrial monitoring, etc., and have strong versatility.

[0117] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 It is a schematic diagram of the architecture and workflow of the multi-task edge video analysis system of the present invention;

[0119] Figure 2 It is a schematic diagram of a module of a request scheduling method for multi-task edge video analysis of the present invention;

[0120] Figure 3: is a schematic diagram of the structure of the video analysis accuracy prediction model based on the Transformer model of the present invention; wherein (A) is the feature vector structure of each frame; (B) is the process of directly capturing the feature sequence by the Transformer model;

[0121] Figure 4 is a heuristic algorithm diagram for video analysis request scheduling of the present invention;

[0122] Figure 5 It is a greedy algorithm diagram of GPU allocation of the present invention. DETAILED DESCRIPTION

[0123] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0124] like Figure 1 As shown, the present invention provides a request scheduling method for multi-task edge video analysis, comprising the following steps:

[0125] Step S1, building a multi-task video analysis system based on edge computing: Multi-task edge video analysis adopts a cloud-edge-end collaborative architecture, which is mainly composed of three levels, namely, end device camera, edge server and cloud center;

[0126] Step S2, establishing a configuration selection model considering multi-task request merging: based on the multi-task video analysis system of step S1, construct a multi-task request merging model for communication and computing resources to optimize the configuration selection process;

[0127] Step S3, GPU resource modeling based on Graphics Processing Unit (GPU) virtualization technology: logically partition the physical GPU based on GPU virtualization technology, realize dynamic allocation of GPU computing resources and video memory resources, and establish a corresponding resource constraint model;

[0128] Step S4, based on the video analysis model, performing video frame feature extraction with multi-feature fusion: based on the visual features of the video analysis model and the detection features based on the video analysis results, the video content (such as lighting, scene) and the detection results (such as target category, size) are comprehensively considered to fully reflect the current video characteristics; wherein the intermediate feature map of the video analysis model is used as the visual feature, and the video analysis results are extracted as the detection feature;

[0129] Step S5, predicting the video analysis accuracy based on the Transformer model: using the Transformer model to process the time series composed of video frame features, capturing sequence features and time dependencies, and then processing the feature representation extracted by the Transformer through a multi-layer perceptron to predict the relationship between the camera configuration and the analysis accuracy in the current scene;

[0130] Step S6, establishing an optimization problem for minimizing the total system cost: defining the internal cost of the edge computing network and the external cost of task offloading based on serverless cloud computing, considering the latency and accuracy requirements, and constructing a quadratic constrained quadratic programming problem for minimizing the total system cost;

[0131] Step S7, efficient request scheduling based on heuristic algorithm: Based on the predicted configuration-accuracy relationship, a multi-granularity heuristic solution improvement algorithm is adopted, the feasibility and quality of the solution are improved simultaneously through the penalty function idea, and the greedy algorithm is combined to achieve optimal allocation of GPU resources.

[0132] like Figure 2 As shown, the system framework of the request scheduling method for multi-task edge video analysis of the present invention is mainly composed of a configuration selection and GPU resource modeling module, a video analysis accuracy prediction module based on the Transformer model, an optimization problem module for minimizing the total cost of the system, and an efficient request scheduling algorithm module based on heuristic thinking.

[0133] Among them, the configuration selection and GPU resource modeling modules established a multi-task edge video analysis architecture with cloud-edge-end collaboration, designed a configuration selection model and GPU resource allocation model that considered the merging of multi-task requests, and provided system model support for subsequent optimization problems.

[0134] The video analysis accuracy prediction module based on the Transformer model predicts the analysis accuracy corresponding to each camera configuration in the current scene through the relationship between video content and video frame changes, providing accuracy information support for the formation of optimization problems.

[0135] The optimization problem module for minimizing the total system cost is based on the system model and accuracy information, comprehensively considers the delay and accuracy requirements, and constructs a quadratic constrained quadratic programming problem for minimizing the total system cost.

[0136] A multi-granularity heuristic solution improvement algorithm is designed based on the efficient request scheduling module of the heuristic algorithm. The feasibility and quality of the solution are improved simultaneously through the penalty function idea, and the greedy algorithm is combined to achieve the optimal allocation of GPU resources.

[0137] Example

[0138] The following example specifically illustrates a request scheduling method for multi-task edge video analysis of the present invention, and the specific processing flow calculation description is as follows:

[0139] Step S1: Build a multi-task video analysis system based on edge computing.

[0140] First, we adopt the cloud-edge-end collaborative architecture to build a multi-task video analysis system based on edge computing, which mainly consists of three levels: end device camera, edge server and cloud center, such as Figure 1 shown.

[0141] Device camera collection Indicates that multiple video analysis requests can be executed on a camera at the same time. The requests on camera k can be represented as a set Any request With its user-specified maximum end-to-end delay Requirements and minimum accuracy requirements To support its task. The request set on all K cameras is expressed as

[0142] Edge Server Collection They are located at the edge of the network and provide timely services to nearby cameras. Each edge server can provide services to multiple end-device cameras, while a camera is only connected to one edge server. The mapping between cameras and edge servers is defined by a fixed function o(·), where camera Connecting to the Edge Server

[0143] The cloud center is interconnected with the edge servers through the public Internet to process video analysis requests that exceed the computing capacity of the edge servers.

[0144] Secondly, the workflow of the edge computing-based multi-task video analysis system is as follows: schedule video analysis requests through a global scheduler to optimize configuration selection, request merging, and GPU resource allocation. The scheduler relies on four inputs: the accuracy of each video stream provided online by the video analysis accuracy prediction module; the data size of different frames obtained through offline analysis and the GPU memory usage of different video analysis models; the accuracy and latency requirements defined by the user; and the real-time resource consumption of bandwidth, GPU, and video memory.

[0145] Considering that target detection is the basic module of most video analysis applications, the detection tasks of different categories of targets (such as vehicle detection, pedestrian detection, etc.) are regarded as different task video analysis requests.

[0146] Step S2: Establish a configuration selection model that considers the merging of multiple task requests.

[0147] Based on the multi-task video analysis system in step S1, a configuration selection model considering the merging of multi-task requests is established. The adjustable configuration options of the video analysis request can be expressed as a 4-tuple δ(r δ ,f δ ,mδ ,p δ ), the set of all available configurations is represented by Δ = {δ}; where the variable r δ Indicates the video resolution; f δ Indicates the number of video analysis frames per second (Frame Analyzed PerSecond, FAPS); m δ represents the video analysis model; p δ Indicates the request execution location, p δ =0 means the request is processed locally on the edge server; p δ =1 means the request is offloaded to the cloud for processing.

[0148] In order to save computing resources, only a part of the frames can be selected for video analysis due to the temporal correlation between consecutive frames. FAPS is adjusted according to the accuracy requirement. If the accuracy requirement is not high, a lower FAPS can be used.

[0149] Configuration selection is achieved by adjusting the configuration tuple for each request. To this end, a binary variable in Indicates that request q selects configuration δ. Since a video analysis request can only select one configuration, the following conditions must be met:

[0150]

[0151] In order to save communication and computing resources, a video analysis request merging scheme is adopted, that is, if multiple requests issued to the same camera select the same configuration (i.e. resolution, frame rate, model, and execution location), these requests can be merged into one. However, although request merging can save resource consumption, this merging process may reduce the performance of video analysis. Let's explain it with an example:

[0152] Assume that there are two requests on the same camera, "Request A" has high accuracy requirements and needs to select a high configuration, while "Request B" has high latency requirements and needs to select a low configuration. If "Request B" is merged into "Request A", the service latency of "Request B" will increase and may even exceed its latency constraint, mainly because the video analysis process using the high configuration takes longer in both transmission and processing time. Similarly, merging "Request A" into "Request B" requires sacrificing accuracy. Therefore, request merging must be carefully designed.

[0153] Introducing intermediate binary variables The request merging process is controlled by the "Boolean OR" operation. Where k is the camera number and δ is the configuration index. When at least one of is true, as shown below:

[0154]

[0155] Therefore, the bandwidth resource constraints between the camera and the edge server and the bandwidth resource constraints between the edge server and the cloud center are as follows:

[0156]

[0157] Among them, b δ Indicates the data size of one frame when using configuration δ, Indicates the total bandwidth from the camera to the edge server. Indicates the total bandwidth from edge servers to the cloud center.

[0158] Step S3: Perform GPU resource modeling based on GPU virtualization technology.

[0159] Video analysis based on deep learning usually has a large demand for GPU resources, and its performance is often limited by the bottleneck of GPU resources. Based on the multi-task video analysis system architecture model based on edge computing and the multi-task video analysis request merging model, the GPU computing resource and GPU memory resource models and constraints are established, and GPU resource modeling based on GPU virtualization is performed.

[0160] (1) GPU computing resources.

[0161] The Multi-Process Service (MPS) technology developed by NVIDIA can process multiple CUDA applications in parallel on a single GPU, thereby achieving efficient division and management of GPU resources. With MPS, the physical GPU computing power obtained by each application is specified, including the percentage of active threads (indicating GPU computing power) and fixed device memory limits (indicating GPU video memory resources). MPS technology allows a physical GPU to be divided into multiple virtual GPUs, each of which is dedicated to running a video analysis model instance for a single video request. This design reduces frequent switching and preemption between GPU tasks, thereby improving the parallelism, isolation, and stability of the service.

[0162] Let discrete variables Indicates the GPU computing resource allocation for each merge request. G is the total number of computing resource slices. The GPU computing capacity constraint of each edge server can be expressed as:

[0163] ;

[0164] in, Represents the total GPU computing resources of edge server n.

[0165] (2) GPU video memory resources.

[0166] For GPU memory usage, the memory usage required by each video analysis model is obtained through offline analysis. The GPU memory resource constraint can be expressed as:

[0167]

[0168] Among them, u δ Indicates the video memory usage of the video analysis model when using configuration δ; Indicates the total GPU memory resources of edge server n.

[0169] Step S4: extracting video frame features by multi-feature fusion based on the video analysis model.

[0170] Traditional methods based on offline analysis and online analysis need to try all configuration combinations on a small representative video clip and calculate the accuracy to infer the accuracy for a period of time in the future. This type of method consumes a lot of resources and a lot of time during analysis. The present invention adopts a video analysis accuracy prediction method based on the Transformer deep learning model, which consumes less resources and has high accuracy during online prediction. The video frame feature extraction of multi-feature fusion can comprehensively characterize the current video characteristics and provide support for the subsequent video analysis accuracy prediction based on the Transformer model.

[0171] The Transformer model has difficulty processing raw video streams directly, mainly due to the huge amount of video data and the sparsity of useful information in the raw data. The end-to-end Transformer model that learns directly from raw video data requires a complex architecture, high computing power requirements, and long inference latency, especially on edge servers with limited resources. To solve this problem, a video frame feature extraction method with multi-feature fusion is designed to make the Transformer model more lightweight and obtain better prediction results. The feature vector of each frame consists of three parts: visual features, current video analysis model size, and detection features, such as Figure 3 As shown in (A), the details are as follows:

[0172] (1) Visual features.

[0173] Video content has a significant impact on the accuracy of video analysis, such as changes in lighting conditions (day or night), scene activities (busy or idle), and camera angles. Existing studies usually use independent convolutional neural networks (CNNs) to capture the relationship between video content and accuracy, such as models such as Resnet and Darknet. In contrast, the present invention reuses the intermediate feature maps generated by the backbone of the video analysis model, which can save additional CNN calculations.

[0174] After obtaining these visual feature maps, a global average pooling (GAP) operation is used to reduce the spatial dimension. GAP calculates the average of all values ​​in each feature map channel and reduces the height and width to one value per channel. Since the video analysis model is dynamically adjusted according to the scheduling results of the video request scheduler, the number of channels in the feature map and the dimension of the vector after GAP will also change. To this end, a repeated padding operation is used to pad all vectors to the dimension that matches the vector generated by the largest video analysis model, using f v Represents the visual features of a single video frame.

[0175] The dimension of the visual feature map will change according to the currently selected video analysis model, resulting in a dynamic change in the number of repeated fillings after the GAP. Therefore, a one-hot vector is used to encode the currently selected video analysis model, denoted as f m This vector can be used as f v Indicator of fill mode.

[0176] (2) Detection features.

[0177] The object detection results output by the video analysis model include category labels, bounding boxes, and confidence scores. This information can also be used to characterize video features, as follows:

[0178] A. Category label: The number of targets has an important impact on the accuracy of video analysis. Intuitively, a large number of targets indicates that the current scene may be very crowded and there may be serious occlusions between targets. Therefore, a low-configuration video analysis model may not be able to accurately detect all targets. Use the vector f n Indicates the number of detected objects as follows:

[0179] f a =[a,a1,…,a T ] (7);

[0180] Among them, the total number of targets a and the number of targets in each category a1 to a T ;gather Represents all detectable target categories, all category labels in one frame

[0181] B. Bounding box size: The size of the object can also significantly affect the accuracy of video analysis. For example, when dealing with small objects, low resolution leads to lower analysis accuracy. The calculation of all class labels in a frame is The bounding box size of the target is stored in vector B t In B, each component represents the size of a target. Considering that the number of targets is dynamic, t The dimension is not fixed, so use B t Statistics are used to represent target size information, as shown below:

[0182]

[0183] in, is the vector of the bounding box size; B t Represents a vector of bounding box sizes of all targets with category label t in a frame; min(B t )、max(B t )、mean(B t ) and var(B t ) represent B t The minimum, maximum, mean, and variance of .

[0184] C. Confidence score: The confidence score reflects how certain the model is about each detected object. A lower confidence score may indicate that some detections in the scene are uncertain, which indicates that a more sophisticated model may be needed. The definition is similar to that of Describes the confidence score. In addition, An additional value is included, which represents the sum of the confidence scores of all objects with class label t, as shown below:

[0185]

[0186] Among them, C t is a vector of confidence scores of all targets with category label t in a frame; sum(C t ) represents the sum of confidences of all objects with category label t; min(C t )、max(C t )、mean(C t ) and var(C t ) represent C t The minimum, maximum, mean, and variance of .

[0187] By concatenating the above vectors, we can get the detection feature f of a video frame. d , as shown below:

[0188]

[0189] The visual features, current model size, and detection features are concatenated together to fully represent a video frame. Before concatenation, these features are made to have similar dimensions through a linear projection layer, as shown below:

[0190] f=[f v ,f m ,f d ] (11);

[0191] Among them, f v Represents the visual features of a single video frame; f m Indicates the use of a one-hot vector to encode the currently selected video analysis model, f d Represents the detected features of a video frame.

[0192] Step S5: Predict the video analysis accuracy based on the Transformer model.

[0193] The Transformer-based video analysis accuracy prediction model can predict the analysis accuracy of the current video under all configurations, which is the prerequisite for the subsequent establishment and solution of the optimization problem of minimizing the total cost of the system.

[0194] Since video streams are continuous in nature, changes in video content and movement of objects may help to analyze accuracy prediction. Existing methods have explored motion features such as speed, size change rate, and aspect ratio change rate, but the selection and representation of these motion features are far from optimal. Considering the spatiotemporal dependency of video streams, the Transformer model is used to directly analyze the feature sequence, which is a sequence composed of the fused multi-features of each video frame extracted in step S4 in the time order of the video frames.

[0195] Transformer is a sequence-to-sequence deep learning model that can effectively capture the temporal dependency information in video feature sequences. Figure 3 As shown in (B) in Figure 1, in order to explore the potential implicit features in the video frame sequence, the Transformer model is used to directly capture the time dependency information in the time window F of the feature sequence, as shown below:

[0196] F=f (1) ,f (2) ,…,f (β) (12);

[0197] H = Encoder (F) (13);

[0198] z = Pooling(H) (14);

[0199] a t =MLP(z),t=1,…,T (15);

[0200] a=[a1,…,a T ] (16);

[0201] Among them, F represents the time window of the feature sequence directly captured by the Transformer model; the parameter β is the length of the window. This method enables the model to learn and extract relevant features from the data instead of just relying on a predefined feature set.

[0202] By utilizing the self-attention mechanism, the Transformer model is able to capture local and global dependencies and contextual relationships in the data, making it very effective in processing complex temporal pattern tasks. The Transformer architecture consists of an encoder and a decoder. The Transformer encoder is used to extract feature representations. The encoder consists of multiple Transformer encoder layers, and position encoding is added to provide the model with position information of each element in the sequence.

[0203] After the Transformer encoder, an average pooling operation is applied to aggregate the features in the sequence and output the feature representation. The resulting feature vector is input to the output layer, which consists of multiple multi-layer perceptrons (MLPs). Each MLP is responsible for predicting the accuracy of a specific object detection category under different configurations. Each MLP consists of two fully connected layers that map the aggregated features to the final accuracy prediction.

[0204] Step S6: Based on system modeling, resource constraints and prediction accuracy, an optimization problem of minimizing the total system cost is established.

[0205] S61. Establish user delay constraints.

[0206] The latency requirement constraints for each request are as follows:

[0207]

[0208] in, Indicates the service delay per frame when using configuration δ; Indicates the maximum end-to-end delay requirement.

[0209] Let n represent the edge server that provides services for camera k, that is, n = o(k). Service latency per frame when using configuration δ As shown below:

[0210]

[0211] in, is the computational delay of the video analysis model for each frame, It is the transmission delay from the camera to the edge server. If the request is offloaded to the cloud, additional network delay will be added, which is the transmission delay from the edge to the cloud. and a fixed network propagation delay The composition takes into account the inherent latency of the public Internet connection between the edge servers and the cloud center.

[0212] Computational latency of each frame of video analysis model By video analysis model m δ , request execution position p δ And the allocated GPU computing resources Decide as follows:

[0213]

[0214] The function T(·) is obtained by offline analysis.

[0215] Assuming that the bandwidth of the communication link between the camera and its edge server is evenly distributed, the transmission delay from camera k to edge server n is determined by the following formula:

[0216]

[0217] Similarly, the transmission delay from edge server n to the cloud can be calculated by the following formula:

[0218]

[0219] To prevent frame loss, the throughput of the selected model must be greater than the input frame rate. This constraint is expressed as:

[0220]

[0221] S62. Establish user accuracy constraints.

[0222] The video analysis accuracy refers to the average object detection accuracy of all frames within a small time interval α. The object accuracy of each video frame is evaluated by the average intersection over union (IoU) between the object detection result and the true annotation. Due to the dynamic adjustment of FAPS, only some frames may be analyzed. For frames that are not analyzed, the position of the object in the previous analyzed frame is used as the current detection result.

[0223] The prediction accuracy of the selected configuration for each request Its user accuracy requirements should be met as follows:

[0224]

[0225] in, Indicates the prediction accuracy of the selected configuration for each request; represents the minimum accuracy requirement; the prediction accuracy is provided by the video analysis accuracy prediction module based on the Transformer model in step S5.

[0226] S63. Establish problem statement.

[0227] The goal of this method is to minimize the total cost for edge service providers while meeting the accuracy, latency, and resource constraints required for edge video analysis. Let C be the total cost of the edge service provider, including internal cost I and external cost E, which are defined as follows:

[0228] Internal cost I comes from operating expenses and includes three parts: transmission cost within the edge network, GPU memory cost of all edge servers, and GPU computing cost. Transmission cost is proportional to the amount of data transmitted, while GPU memory and computing cost are related to total memory usage and GPU computing resource consumption, respectively. t , l u and l g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second. The internal cost per second is as follows:

[0229]

[0230] Among them, i trans is the internal transmission cost; i mem is the internal memory cost; i gpu is the internal GPU cost.

[0231] The emerging cloud server-less computing paradigm provides edge service providers with a pay-as-you-go billing method, namely, external cost E. In this invention, the edge video analysis system needs to pay the cloud service provider for the resource usage of network bandwidth, GPU computing, and memory. Similar to the internal cost coefficient, the cloud computing coefficient ∈ t ,∈ u and ∈ g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second. The external cost per second is as follows:

[0232]

[0233] Among them, etrans is the external transmission cost; mem is the external memory cost; e gpu is the external GPU cost. Based on this, the system total cost minimization problem can be expressed as:

[0234]

[0235] stC1,C2,C3,C4,C5,C6,C7,C8(27);

[0236]

[0237] Since the objective function and constraints C4 and C6 both contain quadratic terms This problem can be classified as a quadratically constrained quadratic programming (QCQP) problem. The complexity of solving the QCQP problem through semidefinite relaxation (SDR) is as follows:

[0238]

[0239] Where d is the variable dimension and ω is the precision of the solution. In this method, if the total number of video analysis requests is recorded as Q, the total number of configurations is recorded as H, and the total number of cameras is recorded as K, then d = H 2 QK (usually more than 10 7 ), making the problem unsolvable in a reasonable amount of time. Therefore, a more efficient algorithm is needed.

[0240] Step S7: Solve the system total cost minimization problem in step S5 based on a heuristic algorithm.

[0241] An efficient heuristic algorithm is proposed to solve the above system total cost minimization problem by coordinating configuration selection, request merging and GPU allocation.

[0242] The principle of heuristic algorithm is as follows Figure 4 As shown. The key principle of the proposed heuristic strategy is to start from an initial solution and gradually reduce the cost. In each iteration, the configuration selection x is first updated through the greedy idea (lines 6-12), and then the corresponding GPU computing resource g is allocated (line 13). If the current goal has not been significantly improved within θ iterations and all constraints are satisfied, the loop is terminated and the solution is output (lines 15-18).

[0243] In order to meet the user's accuracy requirements, a configuration set Δ that meets the accuracy requirements is established in advance for each request q q (row 1), where In the subsequent phase, only each request Δ q Configuration within.

[0244] (1) Improvement of the solution based on the penalty function idea.

[0245] Inspired by the penalty function method for solving constrained optimization problems, the performance index φ is defined as the current target and the constraint violation term σ(P L +P R ), as follows:

[0246]

[0247] Among them, σ is the penalty coefficient, P L represents the penalty term for delay, P R The penalty term representing the resource constraint.

[0248] The constraint violation term is zero when the performance indicator φ is in the feasible solution set and becomes positive when it is outside the feasible solution set. Therefore, the constraints are penalized when they are not satisfied, and the performance indicator is consistent with the actual target value when all constraints are satisfied. σ is set to a small value at the beginning and then increased by a factor of p in each iteration (line 21), thereby gradually applying stronger penalties to constraint violations. As σ becomes larger, the penalty term becomes more and more significant outside the feasible solution set, which encourages all constraints to be gradually satisfied. Therefore, the solution found will gradually converge to the solution of the original constraint problem.

[0249] The penalty term P for delay constraints and resource constraints L and P R , as shown below:

[0250]

[0251] in, Indicates the currently used bandwidth from camera to edge server; Indicates the currently used edge server to cloud server bandwidth; Indicates the GPU resources currently used; represents the memory resources currently used; the constraint violation term is zero when the performance index φ is in the feasible solution set and becomes positive when it exceeds the feasible solution set. Due to the different measurement units and orders of magnitude of these indicators, the penalty term is defined as the normalized difference between the actual value and the constraint value. When the corresponding constraint is fully satisfied, the penalty is zero.

[0252] The solution to x is gradually improved through a greedy strategy. In each iteration, the current solution is updated to the solution with the minimum performance index, and then the system state is adjusted accordingly, and new solutions are continued to be explored in subsequent iterations. This process continues until all constraints are satisfied and no further improvements can be found.

[0253] Since there are a total of H QThere are many possible x, and it is very expensive to calculate the performance indicators of all possible solutions. Therefore, a multi-granular solution update strategy is proposed, where only one request or the configuration of all requests on a single camera is updated in each iteration. Define φ q,δ To switch from the current configuration decision x to the new decision x (q,δ) The performance index of (q,δ) = means that only the configuration of request q is updated to configuration δ, while retaining the existing configurations of other requests. Similarly, φ k,δ Indicates switching from x to x (k,δ) The performance index of (k,δ) = means updating the configuration of all requests on camera k to configuration δ, and keeping the configuration of requests on other cameras unchanged. Then, from {φ q,δ} and {φ k,δ}, select the solution with the smallest performance index. This strategy reduces the search space to HQ+Hk; where Q is the total number of video analysis requests, H is the total number of configurations, and K is the total number of cameras.

[0254] This multi-granularity solution update strategy has two advantages: First, the strategy fully considers the merging of requests. (k,δ) is a coarse-grained step that changes the configuration of all requests on a camera and merges them into one request. Switching from x to x (q,δ) It is a fine-grained step that adjusts the configuration of only one request and considers merging the current request into other requests. Secondly, this strategy speeds up the convergence of the algorithm because the configurations of multiple requests can be decided simultaneously in one iteration.

[0255] (2) Two-stage decision making of configuration selection and GPU allocation.

[0256] First determine the current configuration selection decision x, and then use the greedy algorithm to determine the allocation of GPU computing resources g.

[0257] Since increasing GPU computing resources will reduce computing latency, a greedy strategy is used to allocate the minimum amount of GPU resources that meet latency requirements for a specific request under its selected configuration. However, for some requests, if the latency requirements are too strict or the current configuration is too high, even allocating all GPU computing resources may not be able to fully meet the latency requirements. In this case, Just set it to 1.

[0258] like Figure 5 As shown in Figure 1, we first use x to update the intermediate request merge variable s (line 1), and then calculate the delay from the camera to the edge server and from the edge server to the cloud center (line 2). Next, we calculate the GPU computing resource allocation for each pair of k and δ. If there is no request to select configuration δ on camera k, then Set to 0 (line 16); otherwise, for merge requests The requests in (including all requests for configuration δ on camera k) determine the maximum computational latency that can be tolerated by the merged requests (lines 6-8). If the latency requirement cannot be met even if all GPU computing resources are allocated, Set to 1, and mark the constraint satisfaction flag n as False (lines 9-11). Otherwise, Set it to the minimum value that satisfies the latency requirement and throughput constraint, and set η to True (Lines 12-14).

[0259] Therefore, the present invention adopts the above-mentioned request scheduling method for multi-task edge video analysis, aiming to minimize the total system cost through dynamic configuration adjustment and GPU resource allocation under the premise of meeting the user's accuracy and latency requirements; capture the current video content and inter-frame time dependency through a prediction module based on the Transformer model, and accurately predict the relationship between configuration and analysis accuracy in real time; adopt video analysis request merging technology to efficiently utilize limited network and computing resources, realize the sharing of network transmission and model calculation, and reduce resource consumption; based on the configuration-accuracy analysis results, use heuristic algorithms to efficiently schedule video analysis requests, dynamically adjust video configuration, model selection, request merging, task deployment and GPU resource allocation to optimize system performance and reduce costs.

[0260] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A request scheduling method for multi-task edge video analysis, characterized in that: The following steps are involved: Step S1, building a multi-task video analysis system based on edge computing: Multi-task edge video analysis adopts a cloud-edge-end collaborative architecture, which is mainly composed of three levels, namely, end device camera, edge server and cloud center; Step S2, establishing a configuration selection model considering multi-task request merging: based on the multi-task video analysis system of step S1, construct a multi-task request merging model for communication and computing resources to optimize the configuration selection process; Step S3, GPU resource modeling based on GPU virtualization technology: logically partitioning the physical GPU based on GPU virtualization technology, realizing dynamic allocation of GPU computing resources and video memory resources, and establishing a corresponding resource constraint model; Step S4, based on the video analysis model, perform video frame feature extraction with multi-feature fusion: based on the visual features of the video analysis model and the detection features based on the video analysis results, comprehensively consider the video content and the detection results to fully reflect the current video characteristics; wherein the intermediate feature map of the video analysis model is used as the visual feature, and the video analysis results are extracted as the detection feature; Step S5, predicting the video analysis accuracy based on the Transformer model: using the Transformer model to process the time series composed of the video frame features in step S4, capturing the sequence features and time dependencies, and then processing the feature representation extracted by the Transformer through a multi-layer perceptron to predict the relationship between the camera configuration and the analysis accuracy in the current scene; Step S6, establishing an optimization problem for minimizing the total system cost: defining the internal cost of the edge computing network and the external cost of task offloading based on serverless cloud computing, considering the latency and accuracy requirements, and constructing a quadratic constrained quadratic programming problem for minimizing the total system cost; Step S7, based on the heuristic algorithm, solve the total cost minimization problem of the system in step S5: based on the predicted configuration-accuracy relationship, adopt a multi-granularity heuristic solution improvement algorithm, improve the feasibility and quality of the solution at the same time through the penalty function idea, and combine the greedy algorithm to achieve optimal allocation of GPU resources.

2. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: A system framework for request scheduling method for multi-task edge video analysis, which consists of a configuration selection and GPU resource modeling module, a video analysis accuracy prediction module based on the Transformer model, an optimization problem module for minimizing the total system cost, and a request scheduling algorithm module based on heuristic ideas; Among them, the configuration selection and GPU resource modeling modules establish a multi-task edge video analysis architecture with cloud-edge-end collaboration, and design a configuration selection model and GPU resource allocation model that considers the merging of multi-task requests; Based on Transforme r The model's video analysis accuracy prediction module predicts the analysis accuracy corresponding to each camera configuration in the current scene based on the relationship between video content and video frame changes; The optimization problem module for minimizing the total system cost is based on the system model and accuracy information, comprehensively considers the delay and accuracy requirements, and constructs a quadratic constrained quadratic programming problem for minimizing the total system cost; A multi-granularity heuristic solution improvement algorithm is designed based on the request scheduling module of the heuristic algorithm. The feasibility and quality of the solution are improved through the penalty function idea, and the greedy algorithm is combined to achieve the optimal allocation of GPU resources.

3. A request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S1, the workflow of the multi-task video analysis system based on edge computing is: Video analysis requests are scheduled through a global scheduler to optimize configuration selection, request merging, and GPU resource allocation; detection tasks for different categories of targets are treated as different task video analysis requests; Among them, the scheduler relies on four inputs, namely the accuracy of each video stream provided online by the video analysis accuracy prediction module; the data size of different frames obtained through offline analysis and the GPU memory usage of different video analysis models; user-defined accuracy and latency requirements; and real-time resource consumption of bandwidth, GPU and video memory.

4. A request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S2, a configuration selection model considering the merging of multi-task requests is established, as follows: The adjustable configuration options of the video analysis request are represented as a 4-tuple δ(r δ ,fδ,m δ , p δ ), the set of all available configurations is represented by Δ = {δ}; where the variable r δ Indicates the video resolution; f δ Indicates the number of video analysis frames per second FAPS; m δ represents the video analysis model; p δ Indicates the requested execution location. When p δ = 0 means the request is processed locally on the edge server. δ =1 means the request is offloaded to the cloud for processing; To implement configuration selection by adjusting the configuration tuple for each request, a binary variable is introduced in Indicates that request q selects configuration δ. Since only one configuration can be selected for a video analysis request, the following conditions must be met: in, represents the set of requests on all K cameras, Represents the camera collection of the terminal device; In order to save communication and computing resources, a video analysis request merging scheme is adopted to introduce intermediate binary variables. The request merging process is controlled by the "Boolean OR" operation; where k is the camera number; when When at least one of is true, as shown below: in, represents the request set on camera k; Therefore, the bandwidth resource constraints between the camera and the edge server and the bandwidth resource constraints between the edge server and the cloud center are as follows: Among them, b δ Indicates the data size of one frame when using configuration δ, Indicates the total bandwidth from the camera to the edge server. represents the total bandwidth from the edge server to the cloud center; the fixed function o(·) defines the mapping between the camera and the edge server, where the camera Connecting to the Edge Server Represents a collection of edge servers.

5. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S3, based on the edge computing-based multi-task video analysis system architecture model and the multi-task video analysis request merging model, GPU computing resource and GPU memory resource models and constraints are established to perform GPU resource modeling based on GPU virtualization, as follows: (1) GPU computing resources: Let discrete variables Indicates the GPU computing resource allocation for each merge request; G is the total number of computing resource slices; the GPU computing capacity constraint of each edge server can be expressed as: in, represents the total GPU computing resources of edge server n; (2) GPU memory resources: For GPU memory usage, the memory usage required by each video analysis model is obtained through offline analysis; the GPU memory resource constraint can be expressed as: Among them, uδ represents the video memory usage of the video analysis model when using configuration δ; Indicates the total GPU memory resources of edge server n.

6. A request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S4, based on the video analysis model, multi-feature fusion video frame feature extraction is performed, where the feature vector of each frame consists of three parts, namely, visual features, current video analysis model size and detection features, as follows: (1) Visual features: Reuse the intermediate feature maps generated by the backbone of the video analysis model to obtain the visual feature map, and then use the global average pooling (GAP) operation to reduce the spatial dimension; use repeated padding operations to fill all vectors to the dimension that matches the largest vector generated by the video analysis model, and use f v Represents the visual features of a single video frame; Use one-hot vector x Encode the currently selected video analysis model, recorded as f m ; where f m As f v Indicator of filling mode; (2) Detection features: The object detection results output by the video analysis model include category labels, bounding boxes, and confidence scores, which are used to characterize video features as follows: A. Category label: Use vector f a Indicates the number of detected objects as follows: f a =[a, a1,..., a T ] (7); Among them, the total number of targets a and the number of targets in each category a1 to a T ;gather Represents all detectable target categories, all category labels in one frame B. Bounding box size: Use B t The statistics of represent the target size information, as follows: in, is the vector of the bounding box size; B t Represents a vector of bounding box sizes of all targets with category label t in a frame; min(B t )、max(B t )、mean(B t ) and var(B t ) represent B t The minimum, maximum, mean and variance of C. Confidence score: reflects the model's certainty about each detected target, using the statistical vector Describe the confidence scores as follows: Among them, C t is a vector of confidence scores of all targets with category label t in a frame; sum(C t ) represents the sum of confidences of all objects with category label t; min(C t )、max(C t )、mean(C t ) and var(C t ) represent C t The minimum, maximum, mean and variance of By concatenating the above vectors, we can get the detection feature f of a video frame. d , as shown below: The visual features, current model size, and detection features are concatenated together to fully represent a video frame; before concatenation, these features are made to have similar dimensions through a linear projection layer, as shown below: f=[f v ,f m ,f d ] (11); Among them, f v Represents the visual features of a single video frame; f m Indicates the use of a one-hot vector to encode the currently selected video analysis model, f d Represents the detected features of a video frame.

7. A request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S5, the video analysis accuracy is predicted based on the Transformer model, as follows: To explore the potential implicit features in the video frame sequence, the Transformer model is used to directly capture the time dependency information in the time window F of the feature sequence, as shown below: F=f (1) ,f (2) ,...,f (β) (12); H = Encoder (F) (13); z = Pooling(H)(14); a t =MLP(z),t=1,...,T (15); a=[a1,...,a T ] (16); Where F represents the time window of the feature sequence directly captured by the Transformer model; the parameter β is the length of the window; by using the self-attention mechanism, the Transformer model captures local and global dependencies and contextual relationships in the data, making it very effective in processing complex temporal pattern tasks; Transform r The architecture consists of an encoder and a decoder; using Transforme r The encoder is used to extract feature representations. The encoder consists of multiple Transformer encoder layers, and position encoding is added to provide the model with the position information of each element in the sequence; After the Transformer encoder, an average pooling operation is applied to aggregate the features in the sequence and output the feature representation. The resulting feature vector is input to the output layer, which consists of multiple multi-layer perceptrons (MLPs); each MLP is responsible for predicting the accuracy of a specific target detection category under different configurations; each MLP consists of two fully connected layers that map the aggregated features to the final accuracy prediction.

8. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S6, based on system modeling, resource constraints and prediction accuracy, an optimization problem of minimizing the total system cost is established, as follows: S61. Establish user delay constraints and delay requirement constraints for each request, as shown below: in, Indicates the service delay per frame when using configuration δ; Indicates the maximum end-to-end delay requirement; To prevent frame loss, the throughput of the selected model must be greater than the rate of input frames, so the constraint expression is: S62. Establish user accuracy constraint: the prediction accuracy of the selected configuration for each request Its user accuracy requirements should be met as follows: in, Indicates the prediction accuracy of the selected configuration for each request; represents the minimum accuracy requirement; the prediction accuracy is provided by the video analysis accuracy prediction module based on the Transformer model in step S5; S63. Establish problem statement: Its goal is to minimize the total cost for the edge service provider while meeting the accuracy, latency and resource constraints required for edge video analysis; let C be the total cost of the edge service provider, including internal cost and external cost E, which is specifically defined as follows: Internal costs, including three parts: transmission costs within the edge network, GPU memory costs of all edge servers, and GPU computing costs; usage coefficient ι t , u and g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second; the internal cost per second is as follows: Among them, i trans is the internal transmission cost; i mem is the internal memory cost; i gpu is the internal GPU cost; The external cost E includes: the edge video analysis system needs to pay the cloud service provider for the network bandwidth, GPU computing and memory resource usage; the cloud computing coefficient ∈ t ,∈ u and ∈ g They represent the transmission cost per gigabit, the memory cost per gigabit per second, and the cost of using the entire GPU computing resources per second; the external cost per second is as follows: Among them, e trans is the external transmission cost; mem is the external memory cost; e gpu Cost of external GPU; Based on this, the system total cost minimization problem is expressed as: stC1, C2, C3, C4, C5, C6, C7, C8(27); Where G is the total number of computing resource slices; The above problem is classified as a quadratic constrained quadratic programming (QCQP) problem. The complexity of solving the QCQP problem by semidefinite relaxation SDR is as follows: Here, d is the variable dimension and ω is the precision of the solution.

9. The request scheduling method for multi-task edge video analysis according to claim 1, characterized in that: In step S7, based on the heuristic algorithm, the total cost minimization problem of the system in step S5 is solved by coordinating configuration selection, request merging and GPU allocation, as follows: The key principle of the heuristic strategy is to start from an initial solution and gradually reduce the cost. In each iteration, the configuration selection x is first updated through a greedy idea, and then the corresponding GPU computing resource g is allocated. If the current goal has not been significantly improved within θ iterations and all constraints are satisfied, the loop is terminated and the solution is output. In order to meet the user's accuracy requirements, a configuration set Δ that meets the accuracy requirements is established in advance for each request q q ,in In the subsequent phase, only each request Δ q Configuration within; (1) Improvement of the solution based on the penalty function idea; Inspired by the penalty function method for solving constrained optimization problems, the performance index φ is defined as the current target and the constraint violation term σ(P L +P R ), as follows: Among them, σ is the penalty coefficient, P L represents the penalty term for delay, P R The penalty term represents the resource constraint; Penalty term P for delay constraint and resource constraint L and P R , as shown below: in, Indicates the currently used bandwidth from camera to edge server; Indicates the currently used edge server to cloud server bandwidth; Indicates the GPU resources currently used; Represents the memory resources currently used; when the performance index φ is in the feasible solution set, the constraint violation item is zero, and becomes positive when it exceeds the feasible solution set; Based on the multi-granularity solution update strategy, each iteration only updates one request or the configuration of all requests on a single camera; define φ q，δ To switch from the current configuration decision x to the new decision x (q，δ) The performance index of (q，δ) Indicates that only the configuration of request q is updated to configuration δ, while retaining the existing configurations of other requests; φ k，δ Indicates switching from x to x (k，δ) The performance index of (k，δ) = means updating the configuration of all requests on camera k to configuration δ, and keeping the configuration of requests on other cameras unchanged; then, from {φ q，δ } and {φ k，δ }, and select the solution with the smallest performance index; this strategy reduces the search space to HQ+HK; where Q is the total number of video analysis requests, H is the total number of configurations, and K is the total number of cameras; (2) Two-stage decision-making for configuration selection and GPU allocation: First, determine the current configuration selection decision x, and then use the greedy algorithm to determine the allocation of GPU computing resources g; First, use x to update the intermediate request merge variable s, and then calculate the delay from the camera to the edge server and from the edge server to the cloud center; Next, the GPU computing resource allocation for each pair of k and δ is calculated If there is no request to select configuration δ on camera k, then Set to 0; otherwise for merge requests Requests in the merged request, determine the maximum computational delay that can be tolerated; If the latency requirement cannot be met even if all GPU computing resources are allocated, Set to 1, and mark the constraint satisfaction flag η as False; otherwise Set it to the minimum value that satisfies the latency requirement and throughput constraint, and set η to True.

Images