A method and apparatus for processing multiple real-time video streams

Abstract

The application provides a kind of multi-path real-time video stream processing method and device, comprising: obtaining the video stream to be processed, determine the influence of video stream on the quality of service of edge platform, based on the influence of video stream on the quality of service of edge platform determine whether to allow video stream access, when allowing, determine the video frame request set of each video stream;Obtain the environmental state information of edge platform, based on environmental state information determine the optimal batch processing time window, optimal frame rate time window, optimal resolution and optimal concurrency corresponding to each time step;Based on the two-stage task type of each video stream and video frame request set generate batch processing queue, based on optimal batch processing time window, optimal frame rate time window, optimal resolution to batch processing queue dynamic batch processing;Based on the dynamic batch processing result of each video stream generates concurrent processing queue, based on the optimal concurrency determined to concurrent processing queue concurrent processing.The method optimizes the quality of service of edge platform.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for processing multiple real-time video streams. Background Technology

[0002] Ubiquitous cameras and IoT platforms generate massive amounts of data, especially video data. Simultaneously, the rapid development of computer vision technology has spurred numerous edge video analytics applications, such as real-time anomaly detection in autonomous vehicles and pedestrian tracking in urban surveillance. Many video processing tasks in these applications are two-stage tasks, which can be divided into a GPU processing stage and a CPU processing stage. In the first stage, GPU computing resources are used to run deep neural network (DNN) models, obtaining intermediate results by calculating visual information from video frames. In the second stage, CPU computing resources are used to analyze and filter the intermediate results to obtain the final result.

[0003] The two-stage tasks described above can be processed in the cloud. However, to achieve the required high real-time performance, high throughput, and high privacy, using a cloud computing architecture will inevitably significantly increase the system's operation and deployment costs. In some current practices, to reduce the bandwidth required for cloud transmission and improve data privacy and security, edge servers are often used to process video streams. For edge platforms, ensuring service quality (QoS) metrics directly perceived by users, such as latency and accuracy, is particularly important. However, edge servers typically have fixed resources, making dynamic scaling difficult, while simultaneously processing multiple video streams. To ensure real-time service, edge servers could be dedicated to a single tenant, but this wastes valuable edge computing resources. Supporting multi-tenancy and sharing computing resources among multiple applications accessing the edge can reduce the cost per application; however, the dynamically changing request load on a multi-tenant platform will inevitably affect the edge platform's throughput and QoS.

[0004] Improving the performance of video processing tasks, such as throughput, latency, accuracy, and energy consumption, on resource-constrained edge platforms is a crucial issue in edge computing. Most research focuses on inference optimization and multi-tenant scheduling of deep neural networks (DNNs) on edge platforms. Regarding DNN inference optimization, some representative studies have experimentally obtained the execution characteristics of DNNs on GPUs and proposed performance optimization methods such as model selection, batch processing, and adaptive resolution. Examples include using model compression and model selection techniques to optimize GPU resource utilization, using video frame filtering techniques to optimize the video frame selection process to reduce the number of frames input to the DNN for inference, using batch processing techniques for video streams to leverage multi-core parallelism of GPUs to improve throughput, and using multiple model instances for concurrent processing to increase throughput. However, most of these methods only focus on optimizing DNN inference on edge servers and resolving inference resource conflicts. On edge servers, the CPU typically also handles post-processing of DNN inference and request / response tasks. If CPU resources reach a bottleneck, even with optimization strategies for DNN inference and GPU resources, throughput and QoS may still fall short of expectations. Furthermore, using a single optimization strategy cannot fully utilize the valuable computing resources on edge servers. In multi-tenant scheduling, some methods focus on the task scheduling problem of edge platforms under multi-tenant conditions, optimizing throughput, latency, and energy consumption according to different scheduling strategies. For example, distributed greedy scheduling algorithms with task deadlines maximize throughput while minimizing latency; soft real-time task scheduling mechanisms based on earliest deadline priority effectively improve the throughput of edge servers; and reinforcement learning algorithms such as A3C optimize the energy consumption of edge devices and the latency of request responses. However, for two-stage tasks, scheduling strategies that only consider single-stage tasks cannot fully utilize the limited resources on edge servers to improve execution performance. This may manifest as request backlog or poor QoS performance in high-traffic scenarios. Therefore, optimizing the quality of service (QoS) of edge platforms, while ensuring that their throughput can adapt to dynamically changing loads, is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a method and apparatus for processing multiple real-time video streams to solve one or more problems existing in the prior art.

[0006] According to one aspect of the present invention, a method for processing multiple real-time video streams is disclosed, the method comprising:

[0007] The process involves acquiring a video stream to be processed, determining the impact of the video stream on the service quality of the edge platform, determining whether to allow the video stream to access based on the impact of the video stream on the service quality of the edge platform, and determining the video frame request set for each video stream when the video stream is allowed to access.

[0008] Obtain the environmental status information of the edge platform, and determine the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency for each time step based on the environmental status information;

[0009] A batch processing queue is generated based on the two-stage task type of each video stream and the video frame request set of each video stream. The batch processing queue is dynamically batch processed based on the optimal batch processing time window, optimal frame rate time window and optimal resolution corresponding to each determined time step.

[0010] A concurrent processing queue is generated based on the dynamic batch processing results of each video stream, and concurrent processing is performed on the concurrent processing queue based on the determined optimal concurrency degree corresponding to each time step.

[0011] In some embodiments of the present invention, determining the impact of the video stream on the quality of service of the edge platform, and determining whether to allow the video stream to access based on the impact of the video stream on the quality of service of the edge platform, includes:

[0012] The available computing resources of the edge platform, as well as the two-stage task type and soft cutoff time of each video stream, are obtained. Based on the available computing resources of the edge platform, the two-stage task type and soft cutoff time of each video stream, a trained neural network model is used to predict the quality of service (QoS) value of the edge platform. Based on the QoS value of the edge platform, it is determined whether the video stream is allowed to access.

[0013] In some embodiments of the present invention, the quality of service (QoS) value of the edge platform is predicted using a trained neural network model based on the available computing resources of the edge platform and the two-stage task type and soft cutoff time of each video stream, including:

[0014] Determine the number of video streams corresponding to each type of two-stage task;

[0015] The average soft cutoff time is calculated based on the soft cutoff time of each video stream.

[0016] The number of video streams, average soft cutoff time, and available computing resources of the edge platform corresponding to each type of two-stage task are input into the trained neural network model to obtain the service quality value of the edge platform.

[0017] In some embodiments of the present invention, determining the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency for each time step based on the environmental state information includes: determining the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency for each time step using a trained multi-agent reinforcement learning model based on the environmental state information.

[0018] In some embodiments of the present invention, the method further includes: constructing an initial multi-agent reinforcement learning model, and pre-training the initial multi-agent reinforcement learning model to obtain a trained multi-agent reinforcement learning model.

[0019] In some embodiments of the present invention, pre-training the initial multi-agent reinforcement learning model to obtain a trained multi-agent reinforcement learning model includes:

[0020] A sample set is constructed, wherein each sample data in the sample set includes environmental state sample data and joint action sample data. The environmental state sample data includes resource consumption information sample data, available resource information sample data of the edge platform, attribute information sample data of all frame requests in the batch processing queue, and attribute information sample data of all dynamic batch processing results in the concurrent processing queue. The joint action sample data includes batch processing time window sample data, frame rate time window sample data, resolution sample data, and concurrency sample data.

[0021] Construct the global reward function, the model loss function, and the state transition function;

[0022] Obtain sample data of the current state, determine the reward value corresponding to the sample data of the current state based on the global reward function, and determine the new state based on the state transition function;

[0023] Based on the reward value corresponding to the current state and the new state, the model parameters of the initial multi-agent reinforcement learning model are updated through the model loss function.

[0024] In some embodiments of the present invention, the global reward function is:

[0025] r = Q s -αlog(1+max(0,T g -n s ,T c -n s ));

[0026]

[0027] Q r =k·V(l,RDT) s,h );

[0028]

[0029] Among them, Q s Quality of service for edge platforms; Q r Indicates the quality of service for a single request and response; n s Indicates the total number of requests invoked within a time step; l represents the request-response delay; Tg This represents the number of new frame requests added to the batch processing queue from the start to the end of dynamic batch processing; T c This represents the number of newly added dynamic batch processing results in the concurrent processing queue from the start to the end of concurrent processing; k∈[0,1] represents the precision of the request response; V represents the time utility function; RDT s Indicates the soft cutoff time for frame requests; RDT h This represents the hard cutoff time for the frame request; α represents the throughput penalty coefficient greater than 0.

[0030] In some embodiments of the present invention, dynamic batch processing is performed on the batch processing queue based on the determined optimal batch processing time window, optimal frame rate time window, and optimal resolution corresponding to each time step, including:

[0031] All frame requests within the optimal batch processing time window are selected from the batch processing queue to generate the first frame request set;

[0032] Based on the optimal frame rate time window, redundant requests are removed from the first frame request set to generate a second frame request set;

[0033] Adjust the resolution of each frame request in the second frame request set to the optimal resolution.

[0034] According to another aspect of the present invention, a multi-channel real-time video stream processing system is also disclosed. The system includes a processor and a memory, wherein computer instructions are stored in the memory, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any of the above embodiments.

[0035] According to another aspect of the present invention, a computer-readable storage medium is also disclosed having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in any of the above embodiments.

[0036] In the above embodiments of the present invention, the access of the video stream is first determined based on its impact on the service quality of the edge platform. After access is allowed, the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency are further determined based on the environmental state information of the edge platform. Subsequently, dynamic batch processing and concurrent processing are performed based on the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency of each time step. This method makes full use of the limited heterogeneous computing resources on the edge platform, improves the maximum throughput that the edge platform can bear, and effectively improves the service quality of the edge platform while ensuring that the throughput can meet the dynamically changing request load.

[0037] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the written description, claims, and drawings.

[0038] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. The components in the drawings are not drawn to scale but are merely illustrative of the principles of the invention. For ease of illustration and description of certain parts of the invention, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to the invention. In the drawings:

[0040] Figure 1 This is a flowchart illustrating a multi-channel real-time video stream processing method according to an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the architecture of a multi-channel real-time video stream processing system according to an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0043] It should be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0044] It should be emphasized that the term "including / comprises / has" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0045] To reduce the bandwidth required for cloud transmission and improve data privacy and security, multi-tenant edge servers are typically used to process real-time video streams. However, the dynamically changing request load on multi-tenant edge platforms inevitably affects throughput and QoS (Quality of Service), especially in high-traffic scenarios, easily leading to request backlog or poor QoS performance. Therefore, this application addresses the throughput and QoS issues when processing two-stage tasks on multi-tenant edge platforms by designing a learnable, adaptive, and QoS-aware multi-channel real-time video stream processing method. This method can utilize limited edge computing resources to optimize QoS as much as possible while ensuring that the throughput of the edge platform can adapt to dynamically changing loads.

[0046] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0047] Figure 1 This is a flowchart illustrating a multi-channel real-time video stream processing method according to an embodiment of the present invention. (Refer to...) Figure 1 The method may include at least steps S10 to S40.

[0048] Step S10: Obtain the video stream to be processed, determine the impact of the video stream on the service quality of the edge platform, determine whether to allow the video stream to access based on the impact of the video stream on the service quality of the edge platform, and determine the video frame request set for each video stream when the video stream is allowed to access.

[0049] The arrival of new video streams may overload edge servers, leading to lower-than-expected service quality. Therefore, this application performs service quality prediction on newly arrived video streams. This step can be implemented using the access control module in a multi-channel real-time video stream processing system. First, users sharing the computing resources of the edge platform can access video streams generated by IoT devices and request the execution of specified two-stage tasks. When a new video stream arrives, the access control module first predicts the impact of the video stream's access on QoS, thereby deciding whether to allow the video stream to access. Allowed video streams will generate a series of consecutive video frame requests. These video frame requests, along with their accompanying information, will be considered as a series of requests, i.e., a set of video frame requests formed by a series of requests. The accompanying information for the i-th video frame request may include, for example, the task type... Input resolution is Generation time is Relative soft cutoff time Relative hard cutoff time Corresponding video stream number and the frame number in the corresponding video stream That is, the i-th request ri It can be represented as Understandably, the relative soft cutoff time and relative hard cutoff time are the same for all video frames requested in the same video stream.

[0050] For example, determining the impact of the video stream on the service quality of the edge platform, and determining whether to allow the video stream to access based on the impact of the video stream on the service quality of the edge platform, includes: obtaining the available computing resources of the edge platform and the two-stage task type and soft cutoff time of each video stream; predicting the service quality value of the edge platform using a trained neural network model based on the available computing resources of the edge platform and the two-stage task type and soft cutoff time of each video stream; and determining whether to allow the video stream to access based on the service quality value of the edge platform.

[0051] Furthermore, based on the available computing resources of the edge platform and the two-stage task type and soft cutoff time of each video stream, the service quality value of the edge platform is predicted by a trained neural network model, including: determining the number of video streams corresponding to each type of two-stage task; calculating the average soft cutoff time based on the soft cutoff time of each video stream; and inputting the number of video streams corresponding to each type of two-stage task, the average soft cutoff time, and the available computing resources of the edge platform into the trained neural network model to obtain the service quality value of the edge platform.

[0052] In the above embodiment, a lightweight neural network consisting of two fully connected layers is used as a prediction module to predict the impact of video stream access on the quality of service (QoS) of the edge platform. When a video stream generated by an edge camera is accessed, the available computing resources of the edge platform, the number of video streams for each type of dual-stage task, and the average soft cutoff time are first obtained. The input layer of the lightweight neural network then receives the obtained available computing resources, the number of video streams for each type of dual-stage task, and the average soft cutoff time, allowing the output layer to output a stable QoS. When determining whether to allow the video stream access based on the predicted QoS, the predicted QoS is specifically compared with a preset QoS threshold. If it is lower than the preset threshold, the video stream access is rejected.

[0053] Step S20: Obtain the environmental status information of the edge platform, and determine the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency for each time step based on the environmental status information.

[0054] In this step, at the beginning of each scheduling time step, the optimal batch processing time window, frame rate time window, resolution, and concurrency are determined for each time step based on the environmental state information of the edge platform. In other words, this step transforms the multi-objective optimization problem into a scheduling problem that includes the batch processing time window size, frame rate time window size, resolution, and CPU processing stage concurrency. In the corresponding multi-channel real-time video stream processing system, this step is implemented based on the scheduling module.

[0055] Specifically, the scheduling in this application is complex because it involves four dimensions: batch size (corresponding to the batch processing time window), frame rate, resolution, and CPU processing stage concurrency, and is affected by multiple interrelated variables. Specifically, the selection of batch size, frame rate, resolution, and CPU processing stage concurrency depends on the task's computational requirements, the available computing resources of the edge platform, and the attributes of the requests. This type of scheduling task is non-monotonic; the impact of each dimension on the result cannot be quantified in advance, and the influence between dimensions is difficult to estimate beforehand. Compared to heuristic algorithms and single-agent reinforcement learning algorithms, the inventors found that multi-agent deep reinforcement learning is more suitable for this type of task. However, the inventors found that directly using a multi-agent reinforcement learning algorithm (WQMIX algorithm) to adjust the batch size and frame rate would increase the exploration cost of the scheduling module. This is because adjusting the frame rate may affect the batch aggregation speed, thereby increasing the batch processing waiting time. Therefore, this application no longer considers "how many requests should be processed in one batch," but instead considers "how long a batch of video frame requests should be processed in one batch," that is, instead of directly adjusting the batch size during batch processing, the size of the batch processing time window is adjusted. Therefore, this application uses a multi-agent reinforcement learning algorithm to determine the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency for each time step, so as to optimize QoS as much as possible while ensuring that the throughput can meet the dynamically changing load.

[0056] In addition, this method sets a frame rate time window to control the frame rate, meaning that during packet processing, only the first frame of each video stream within each frame rate time window can be retrieved. This multi-channel real-time video stream processing method, by setting the batch processing time window size and the frame rate time window size, ensures that the batch processing waiting time will not be too long due to frame rate adjustments, and makes the adjusted frame rate intervals as uniform as possible, thereby improving QoS stability.

[0057] For example, determining the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency for each time step based on the environmental state information includes: determining the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency for each time step using a trained multi-agent reinforcement learning model based on the environmental state information. In this model, the WQMIX algorithm introduces a weight function based on the QMIX algorithm, which has the following advantages: (1) The WQMIX algorithm maps monotonic functions to non-monotonic functions, thereby efficiently processing non-monotonic tasks; (2) By weighting each action, the algorithm can be prevented from falling into suboptimal strategies.

[0058] In one embodiment, the method further includes: constructing an initial multi-agent reinforcement learning model, and pre-training the initial multi-agent reinforcement learning model to obtain a trained multi-agent reinforcement learning model.

[0059] Further, pre-training the initial multi-agent reinforcement learning model to obtain a trained multi-agent reinforcement learning model includes: constructing a sample set, wherein each sample data in the sample set includes environmental state sample data and joint action sample data; the environmental state sample data includes resource consumption information sample data, available resource information sample data of the edge platform, attribute information sample data of all frame requests in the batch processing queue, and attribute information sample data of all dynamic batch processing results in the concurrent processing queue; the joint action sample data includes batch processing time window sample data, frame rate time window sample data, resolution sample data, and concurrency sample data; constructing a global reward function, a model loss function, and a state transition function; obtaining sample data of the current state, determining the reward value corresponding to the sample data of the current state based on the global reward function, and determining a new state based on the state transition function; and updating the model parameters of the initial multi-agent reinforcement learning model based on the determined reward value corresponding to the current state and the new state through the model loss function.

[0060] Specifically, this step models the problem as a discrete, partially observable Markov decision process (Dec-POMDP), and this fully cooperative multi-agent sequential decision-making task can be represented by a tuple G =<A,S,U,P,R,Ω,O,n.γ> To describe, in which:

[0061] In tuple G, A represents a finite set of n agents. In this application, A is a set of four agents: batch processing time window selection agent, frame rate time window selection agent, resolution selection agent, and concurrency selection agent.

[0062] In tuple G, S represents the global discrete state space, at each time step t. iThe four agents mentioned above will construct a state S. t (S t (∈S), periodically collect request information and resource utilization from the edge platform to describe the true state of the environment. t It consists of 4 parts: (1) Resource consumption information of all tasks running on the edge platform I m (2) Available resources M of the edge platform; (3) Attribute information of all video frame requests in the batch processing queue. (4) Attribute information of all dynamic batch processing results in the concurrent processing queue

[0063] In tuple G, U represents the set of discrete action spaces shared by agents, where each agent a∈A chooses an action u from U. a To form a joint action u∈U≡U n At each time step t i Joint Operation u t It can be represented as u t (w b ,w r ,f r ,n c ), where w b w is the batch processing time window size. r f is the frame rate time window size. r For resolution, n c This represents the maximum concurrency during the CPU processing phase.

[0064] In tuple G, P represents the state transition function, denoted as P(s′|s,u). According to the state transition function P(s), a state can be transitioned to the environment: S×U×S→[0,1].

[0065] In tuple G, R represents the global reward function shared by the agent, i.e., r = R(s,u): S × U → R. In this application, the optimization objective is to optimize QoS as much as possible while adapting throughput to dynamically changing request loads. For edge platforms, ensuring QoS metrics directly perceived by users, such as latency and accuracy, is particularly important; therefore, the quality Q of a single video frame request response is defined. r For Q r =k·V(l,RDT) s,h Where l represents the delay of the request-response interval, k∈[0,1] represents the precision of the request-response interval; V represents the time utility function (TUF); RDT s Indicates the soft cutoff time for frame requests; RDT h Indicates the hard cutoff time for the frame request.

[0066]

[0067] In this application, since it is batch processing, the average quality Q of the video frame request response in one scheduling is used. s QoS as an edge platform Q s For the quality of service of edge platforms, Q r Indicates the quality of service for a single request and response, n s This indicates the total number of requests invoked within a time step.

[0068] Furthermore, for the optimization objective in this step, throughput adaptation and memory constraints also need to be considered. Therefore, the specific optimization objective is defined as follows:

[0069] maxQ s

[0070] stT g >=n s

[0071] T c >=m s

[0072]

[0073] m i This represents the memory size occupied by video frame request i, where M represents the GPU memory limit of the edge platform; T g This represents the number of new frame requests added to the batch processing queue from the start to the end of dynamic batch processing; T c This represents the number of dynamically processed batch results added to the concurrent processing queue from the start to the end of concurrent processing. The first two constraints require that the throughput must be greater than or equal to the input rate of the video frame requests, while the last constraint constrains memory usage. Therefore, the global reward function of this application is: r = Q s -αlog(1+max(0,T g -n s ,T c -n s )); α represents the throughput penalty coefficient greater than 0, which represents the strictness of the throughput constraint.

[0074] In tuple G, Ω represents the set of local observations, which includes the surrogate local observation o. a Local observation value o a It is determined based on the observation function O(s,a).

[0075] In tuple G, γ is a discount factor used to balance long-term and short-term returns.

[0076] At each time step t in the Markov decision process i Each agent a∈A will receive a local observation O. a and select an action u a These actions constitute a joint action u∈U≡U n The joint action will change the system from the current state s according to the state transition function. t Guide to the next state s t+1 The environment will provide a global reward r = R(s, u). Each agent has a local action observation history τ. i ∈T≡(Ω×U) * It provides a potential random policy π a (u a |τ a The condition T×U→[0,1] is provided. τ represents the action observation history of all agents. Ultimately, the agents learn the joint policy π to maximize the global action value function: in k indicates the discount reward.

[0077] In some embodiments, this step employs the WQMIX algorithm, which incorporates a time window, to address the reward maximization problem in deep reinforcement learning. This algorithm consists of three parts: (1) That is, unrestricted joint action and feedforward networks; (2)Q tot That is, the utility Q of each agent. a (3) Weighting function w(s, u).

[0078] It is expressed as follows:

[0079]

[0080] in Similar to the Bellman optimal operator, It is trained by minimizing the following loss: in

[0081] Q tot It is expressed as follows:

[0082]

[0083] The following formula is trained by minimizing the following loss:

[0084]

[0085] Among them Π w This is the WQMIX projection operator.

[0086] In the above embodiments, a central weighting function is used to weight WQMIX:

[0087] in,

[0088] For example, the following is the entire process of a specific WQMIX algorithm scheduling request:

[0089]

[0090] In the above algorithm, the scheduling module in the multi-channel real-time video stream processing system first receives information about each video frame request in the batch processing queue and the dynamic batch processing results corresponding to each video frame request in the concurrent processing queue; before scheduling begins, all networks are initialized, including... and Q tot Since the relative deadlines of video frame requests from different video streams are inconsistent, it is impossible to specify a fixed time step for scheduling. Too small a time step would significantly impact the system's real-time performance. Therefore, this application selects the i-th scheduling time step t. i Minimum relative hard deadline in the scheduled requests This serves as the time step length for this scheduling. When the GPU is idle, the next scheduling will begin immediately; at each scheduling time step t... i At the start, the scheduling module selects an action u to determine the optimal batch processing time window size w. b Optimal frame rate time window size w r Optimal resolution f r Optimal concurrency n during CPU processing c Based on this, video frame requests are scheduled and executed in batches, and the agents in the DRL receive a reward r. t Meanwhile, the state changes from s t Transition to s t+1 The current state, action, reward, and next state are stored as action transitions in the replay buffer. In the middle, the scheduling module updates the network parameters and repeats the above process until the iteration ends.

[0091] Step S30: Generate a batch processing queue based on the dual-stage task type of each video stream and the video frame request set of each video stream, and perform dynamic batch processing on the batch processing queue based on the determined optimal batch processing time window, optimal frame rate time window, and optimal resolution corresponding to each time step.

[0092] This step can be implemented using the dynamic batch processing module of a multi-channel real-time video stream processing system. The dynamic batch processing module performs dynamic batch processing on the batch processing queue based on the optimal batch processing time window, optimal frame rate time window, and optimal resolution determined by the scheduling module. During dynamic batch processing, the batch processing module maintains a batch processing queue for each type of two-stage task and packages the video frame requests in each queue at each scheduling time step.

[0093] In one embodiment, dynamic batch processing is performed on the batch processing queue based on the determined optimal batch processing time window, optimal frame rate time window, and optimal resolution corresponding to each time step, including: selecting all frame requests within the optimal batch processing time window from the batch processing queue to generate a first frame request set; removing redundant requests from the first frame request set based on the optimal frame rate time window to generate a second frame request set; and adjusting the resolution of each frame request in the second frame request set to the optimal resolution.

[0094] For example, during the packaging process, the batch processing module retrieves all video frame requests within the optimal batch processing time window from the batch processing queue. Then, based on the optimal frame rate time window size, it removes redundant requests from all retrieved video frame requests and adjusts the resolution of all video frame requests after removing redundant requests to the selected optimal resolution before assigning them to the corresponding DNN models for execution. The dynamic batch processing module packages video frame requests based on their soft cutoff times; the smaller the soft cutoff time, the higher the priority of the request. Since video frame requests from the same video stream have the same relative soft cutoff time, they are processed in the order they were generated. After dynamic batch processing is completed, the results are placed into the corresponding concurrent processing queue according to the video stream number.

[0095] Step S40: Generate a concurrent processing queue based on the dynamic batch processing results of each video stream, and perform concurrent processing on the concurrent processing queue based on the determined optimal concurrency degree corresponding to each time step.

[0096] This step can be implemented using the concurrent processing module of a multi-channel real-time video stream processing system. The concurrent processing module maintains a concurrent processing queue for each video stream, storing dynamic batch processing results, according to the video stream number. Since the dynamic batch processing results in each concurrent processing queue are input in the order of the corresponding video frame requests, and the processing order of video frame requests ensures that data frames in the same video stream are processed sequentially according to their frame numbers, the dynamic batch processing results in each concurrent processing queue are also arranged in the order of the video frame requests in the corresponding video stream. The concurrency of the two-stage task in the CPU processing stage depends on the optimal concurrency determined by the scheduling module. For each type of two-stage task, the concurrent processing module will retrieve the smallest dynamic batch processing result from all queues for concurrent processing until the number of concurrent processing operations reaches the selected concurrency level. After the dynamic batch processing results are completed, the final request response will be returned. It is worth noting that if the task depends on the relationship between consecutive frames in the video stream, and the previous dynamic batch processing result from the same video stream in the task has not yet been completed, the dynamic batch processing result from the same video stream cannot be retrieved from the concurrent processing queue for processing.

[0097] Correspondingly, the present invention also discloses a multi-channel real-time video stream processing system, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any of the above embodiments.

[0098] Figure 2 This is a schematic diagram of the architecture of a multi-channel real-time video stream processing system according to an embodiment of the present invention. (Refer to...) Figure 2 The system includes at least an access control module, a batch processing module, a scheduling module, a concurrent processing module, and an execution record module. The access control module predicts the impact of the accessed video streams on service quality. The scheduling module, at the start of each scheduling time step, determines the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency level based on the environmental state information of the edge platform. It then feeds back the optimal batch processing time window, optimal frame rate time window, and optimal resolution to the batch processing module, and the optimal concurrency level to the concurrent processing module; that is, each type of two-stage task maintains a batch processing queue, and video frame requests of the same task type enter the same batch processing queue in the order of their generation time. Upon receiving the scheduling signal from the scheduling module, the batch processing module aggregates a batch of requests in each batch processing queue according to the task type. Then, DNN batch processing is performed, where b iThis represents the batch size for processing the i-th type of task. During aggregation, the batch processing module retrieves all video frame requests within the optimal batch processing time window from the batch processing queue. Then, based on the optimal frame rate time window size, it removes redundant requests from all retrieved data frame requests and adjusts the resolution of all video frame requests to the optimal resolution. Additionally, the system maintains a concurrent processing queue for each video stream based on its video stream number to receive the dynamic batch processing results obtained after DNN batch processing. The concurrent processing module retrieves dynamic batch processing results from the concurrent processing queue and performs concurrent processing on the CPU according to the task type, thereby obtaining the final result Z of the video frame request. k ={z1,z2,...,z k}, where k = ∑bi, and the concurrency of the task is the optimal concurrency. The execution recording module collects the running data of each video frame request in real time, including CPU utilization, GPU utilization, GPU memory usage, throughput, latency, and accuracy. Then, it feeds these results back to the scheduling module so that the scheduling module can adjust its next action by learning from the feedback provided by the execution recording module.

[0099] As can be seen from the above embodiments, the multi-channel real-time video stream processing method and apparatus of this application addresses the throughput and Quality of Service (QoS) issues when processing two-stage tasks on multi-tenant edge platforms. It integrates multiple optimization strategies to improve the maximum throughput of the edge platform when processing two-stage tasks. Furthermore, by combining deep reinforcement learning and time windows to jointly regulate multiple optimization strategies, the multi-objective optimization problem is transformed into a scheduling problem involving batch time window size, frame rate time window size, resolution, and CPU processing stage concurrency. This method and apparatus can optimize QoS while utilizing limited edge computing resources, ensuring that the throughput of the edge platform can adapt to dynamically changing loads. In addition, by predicting the impact of video stream access on QoS in advance, and thus predicting potential request overruns, the QoS of the edge platform is further guaranteed.

[0100] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments.

[0101] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0102] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0103] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for processing multiple real-time video streams, characterized in that... The method includes: The process involves acquiring a video stream to be processed, determining the impact of the video stream on the service quality of the edge platform, determining whether to allow the video stream to access based on the impact of the video stream on the service quality of the edge platform, and determining the video frame request set for each video stream when allowing access. The determination of the impact of the video stream on the service quality of the edge platform, and the determination of whether to allow the video stream to access based on the impact of the video stream on the service quality of the edge platform, includes: acquiring the available computing resources of the edge platform and the two-stage task type and soft cutoff time of each video stream; predicting the service quality value of the edge platform using a trained neural network model based on the available computing resources of the edge platform and the two-stage task type and soft cutoff time of each video stream; and determining whether to allow the video stream to access based on the service quality value of the edge platform. The environmental state information of the edge platform is obtained, and the optimal batch processing time window, optimal frame rate time window, optimal resolution and optimal concurrency degree corresponding to each time step are determined based on the environmental state information and the multi-agent reinforcement learning model trained according to the preset global reward function. A batch processing queue is generated based on the two-stage task type of each video stream and the video frame request set of each video stream. The batch processing queue is dynamically batch processed based on the optimal batch processing time window, optimal frame rate time window and optimal resolution corresponding to each determined time step. A concurrent processing queue is generated based on the dynamic batch processing results of each video stream, and concurrent processing is performed on the concurrent processing queue based on the determined optimal concurrency degree corresponding to each time step. The global reward function is as follows: ； ； ； in, For the quality of service of edge platforms; Indicates the quality of service for a single request and response; Indicates the total number of requests invoked within a time step; Indicates a delay in the request-response process; This indicates the number of new frame requests added to the batch processing queue from the start to the end of the dynamic batch processing execution. This indicates the number of new dynamic batch processing results added to the concurrent processing queue from the start to the end of concurrent processing execution; k ∈[0,1] represents the precision of the request response; V represents the time utility function; Indicates the soft cutoff time for the frame request; Indicates the hard cutoff time for the frame request; This represents the throughput penalty coefficient when it is greater than 0.

2. The multi-channel real-time video stream processing method according to claim 1, characterized in that... Based on the available computing resources of the edge platform and the two-stage task type and soft cutoff time of each video stream, a trained neural network model is used to predict the quality of service (QoS) value of the edge platform, including: Determine the number of video streams corresponding to each type of two-stage task; The average soft cutoff time is calculated based on the soft cutoff time of each video stream. The number of video streams, average soft cutoff time, and available computing resources of the edge platform corresponding to each type of two-stage task are input into the trained neural network model to obtain the service quality value of the edge platform.

3. The multi-channel real-time video stream processing method according to claim 1, characterized in that... The determination of the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency for each time step based on the environmental state information includes: determining the optimal batch processing time window, optimal frame rate time window, optimal resolution, and optimal concurrency for each time step using a trained multi-agent reinforcement learning model based on the environmental state information.

4. The multi-channel real-time video stream processing method according to claim 3, characterized in that... The initial multi-agent reinforcement learning model is pre-trained to obtain a trained multi-agent reinforcement learning model, including: A sample set is constructed, wherein each sample data in the sample set includes environmental state sample data and joint action sample data. The environmental state sample data includes resource consumption information sample data, available resource information sample data of the edge platform, attribute information sample data of all frame requests in the batch processing queue, and attribute information sample data of all dynamic batch processing results in the concurrent processing queue. The joint action sample data includes batch processing time window sample data, frame rate time window sample data, resolution sample data, and concurrency sample data. Construct the global reward function, the model loss function, and the state transition function; Obtain sample data of the current state, determine the reward value corresponding to the sample data of the current state based on the global reward function, and determine the new state based on the state transition function; Based on the reward value corresponding to the current state and the new state, the model parameters of the initial multi-agent reinforcement learning model are updated through the model loss function.

5. The multi-channel real-time video stream processing method according to claim 1, characterized in that... Dynamic batch processing is performed on the batch processing queue based on the determined optimal batch processing time window, optimal frame rate time window, and optimal resolution for each time step, including: All frame requests within the optimal batch processing time window are selected from the batch processing queue to generate the first frame request set; Based on the optimal frame rate time window, redundant requests are removed from the first frame request set to generate a second frame request set; Adjust the resolution of each frame request in the second frame request set to the optimal resolution.

6. A multi-channel real-time video stream processing system, the system comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images