Intelligent video resource scheduling management system and method

Abstract

The invention relates to the technical field of intelligent video processing and cloud computing, and discloses an intelligent video resource scheduling management system and method. The system comprises a resource monitoring module, an intelligent analysis module, a task scheduling module and a storage management module. The method comprises the following steps: monitoring load states of computing nodes, network bandwidth and storage resources in real time; based on the type and priority of the video task requested by the user, predicting and analyzing the resource demand through an intelligent algorithm; dynamically generating an optimal resource scheduling strategy, and distributing the task to the most suitable processing node; and hierarchical storage management is carried out on hot, warm and cold data. According to the method, the problems of non-uniform distribution of video processing resources, high response delay and low hardware resource utilization rate in the prior art are solved, the intelligence and optimization of resource scheduling are realized, and the system throughput and the task processing efficiency are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed computing and resource management, and particularly relates to an intelligent video resource scheduling management system and method for a massive video data processing scenario. BACKGROUND

[0002] With the popularity of 4K / 8K ultra-high-definition video, VR / AR content, large-scale video monitoring and real-time video communication, the demand for video data processing of enterprises has shown explosive growth. These processing tasks usually have the dual characteristics of computation-intensive and data-intensive, and need to be completed on a distributed cluster composed of heterogeneous computing devices (such as CPU, GPU, FPGA) and multi-level storage systems.

[0003] At present, the industry generally adopts a resource scheduling system based on rules or static priorities, such as a simple round-robin algorithm or a first-come-first-served algorithm based on task submission time. These prior arts have significant defects in daily use, as follows: Uneven resource utilization: static scheduling strategies cannot perceive real-time load changes of underlying hardware resources, which easily leads to some computing nodes being overloaded due to too many allocated tasks, and performance dropping sharply, while other nodes are in idle state, causing resource waste.

[0004] High response delay: high-priority urgent tasks (such as live streaming real-time processing) may be queued for a long time due to resource being occupied by a large number of low-priority tasks (such as batch archiving of historical videos), which cannot meet their stringent requirements for low delay.

[0005] Lack of intelligent prediction: existing systems cannot predict the actual resource consumption of tasks according to specific parameters (such as encoding format, resolution, duration) and historical data of video files, and can only perform coarse-grained resource allocation, which easily leads to insufficient allocation or resource surplus.

[0006] Storage performance bottleneck: video data has a huge volume, but different access modes (frequent access of hot data and infrequent access of cold data). Existing systems lack intelligent data lifecycle management, and all data are usually stored in the same performance level of storage, which leads to storage I / O becoming the overall performance bottleneck and high cost.

[0007] Therefore, there is an urgent need for a solution that can achieve fine-grained, intelligent and automated scheduling to improve the efficiency and economic benefits of the entire video processing cluster SUMMARY

[0008] The application aims to provide an intelligent video resource scheduling management system and method to solve the problems of uneven resource utilization, high response delay, lack of intelligent prediction and storage performance bottleneck in the background technology. The application has the advantages of high resource utilization, low response delay, intelligent prediction and cost-effective.

[0009] To achieve the above-mentioned purpose, the application provides the following technical solutions: The intelligent video resource scheduling management system and method comprises: 1. Resource perception unit: for real-time monitoring and collecting performance index data of all computing nodes, network devices and storage systems in the cluster, including but not limited to CPU / GPU utilization, memory capacity, network bandwidth occupancy, storage IOPS and delay.

[0010] Task analysis unit: for receiving the video processing task submitted by the user and analyzing the metadata information of the task, including task type, priority, encoding format, resolution, frame rate, duration and expected completion deadline of the video source file.

[0011] Intelligent decision engine: connected with the resource perception unit and the task analysis unit, for generating the optimal resource scheduling strategy and data placement strategy based on the current cluster resource state and task demand characteristics through the built-in machine learning prediction model and optimization algorithm.

[0012] Dynamic scheduling executor: connected with the intelligent decision engine, for executing the scheduling strategy, assigning the task to the most suitable computing node and calling the storage management interface to assign appropriate storage resources for the task.

[0013] Hierarchical storage manager: connected with the intelligent decision engine, for automatically managing the flow of data between the cache, high-performance storage layer and low-cost archiving layer according to the access mode and strategy of the data.

[0014] The resource perception unit adopts a monitoring system based on a time series database, which can continuously collect index data at a second level and provide historical performance trend analysis for the system.

[0015] The task analysis unit supports rich API interfaces, which can receive tasks submitted from a Web portal, a command line tool or an automated workflow engine.

[0016] The intelligent decision engine has a built-in machine learning prediction model, which is a gradient boosting decision tree model. The model is trained using historical task execution records to predict the consumption of computing and storage resources by new tasks.

[0017] The optimization algorithm adopted by the intelligent decision engine is an improved genetic algorithm, which minimizes the total task completion time, maximizes the cluster resource utilization rate and minimizes the network transmission overhead.

[0018] The dynamic scheduling executor has task preemption and recovery functions, and can pause the low-priority task currently running when a higher-priority task arrives, release the resources occupied by the low-priority task, and resume execution from the breakpoint of the low-priority task at a suitable time later.

[0019] The dynamic scheduling executor preferentially selects a computing node storing data required by the task (data locality) when assigning the task, so as to reduce network data transmission delay.

[0020] The hierarchical storage manager dynamically identifies data objects as hot data, warm data or cold data based on access frequency, recent access time and user-defined rules.

[0021] The hierarchical storage manager saves hot data in an ultra-high-speed storage pool based on NVMe SSD, saves warm data in a SAS / SATA SSD storage pool, and automatically migrates cold data to an archive pool based on HDD or object storage.

[0022] The system also provides a visual operation and maintenance center to display the cluster topology, resource level, task execution progress, scheduling decision log and system alarm information in a graphical and real-time manner.

[0023] The present application has the following beneficial effects: 1. Improve resource utilization: through real-time sensing and intelligent prediction, fine-grained and dynamic resource allocation is realized, resource islands are eliminated, and the average resource utilization rate of the cluster is improved from 30-40% in the traditional way to more than 70%.

[0024] Guarantee the performance of critical tasks: through the scheduling strategy and task preemption mechanism based on multi-objective optimization, it is ensured that high-priority tasks can immediately obtain the required resources, and the average response delay is reduced by more than 50%, effectively guaranteeing the service level agreement (SLA).

[0025] Reduce total cost of ownership (TCO): through intelligent data hierarchical storage, expensive high-performance storage resources are concentrated for service hotspot data, and data with low access frequency are automatically settled on low-cost storage media, which significantly reduces storage costs while ensuring performance.

[0026] Realize automation operation and maintenance: The whole system forms a closed loop of self-perception, self-decision and self-execution, greatly reduces the need for manual intervention of operation and maintenance personnel, reduces the risk of manual operation error, and improves operation and maintenance efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The system architecture module diagram of the embodiment one of the present application is shown in the figure. Figure 2 The task scheduling and resource management flow chart of the embodiment two of the present application is shown in the figure. Figure 3 The working flow chart of the intelligent decision engine in the embodiment three of the present application is shown in the figure. Figure 4 The data migration strategy diagram of the hierarchical storage management in the embodiment four of the present application is shown in the figure. Figure 5 The comparison curve diagram of the present application and prior art in the aspect of cluster CPU utilization is shown in the figure. Figure 6 The timing diagram of the task preemption and recovery process in the embodiment of the present application is shown in the figure.

[0028] In the figure: 1, resource perception unit; 2, task analysis unit; 3, intelligent decision engine; 4, dynamic scheduling executor; 5, computing node cluster; 6, visual operation and maintenance center. DETAILED DESCRIPTION

[0029] The technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. The described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application. EMBODIMENT

[0030] Please refer to Figure 1 The intelligent video resource scheduling management system provided by the present application comprises a resource perception unit 1, a task analysis unit 2, an intelligent decision engine 3, a dynamic scheduling executor 4 and a hierarchical storage manager 5. The resource perception unit 1 is responsible for monitoring the health state and performance index of the computing node cluster 6 and the multi-level storage resource pool 7. The task analysis unit 2 receives external task requests. The intelligent decision engine 3 is the core of the system, responsible for processing information and making decisions. The dynamic scheduling executor 4 is responsible for converting decisions into specific scheduling instructions. The hierarchical storage manager 5 is responsible for executing data migration strategies.

[0031] Compared with the prior art, the present application realizes fine and automated management of video processing resources by constructing an intelligent closed loop integrating monitoring, analysis, planning and execution. EMBODIMENT

[0032] Referring to Figure 2 and Figure 3 , the workflow of the intelligent decision engine 3 includes: receiving resource and task information; performing feature extraction and standardization; calling a machine learning model to predict task resource requirements; running an optimization algorithm to generate a scheduling strategy; and finally outputting the strategy to an execution unit.

[0033] In this embodiment, it should be noted that the optimization algorithm solves an NP-Hard problem, and the present application uses a heuristic algorithm to obtain an approximate optimal solution within a reasonable time to balance the solution quality and decision speed. Embodiment

[0034] Referring to Figure 4 and Figure 6 , the dynamic scheduling executor 4 has a task preemption and recovery function. When resources need to be freed for high-priority tasks, it sends a pause signal to the running low-priority task and requires it to save the current state (checkpoint), then releases the task resources and puts them back into the scheduling queue.

[0035] In this embodiment, it should be noted that the checkpoint data is saved in shared storage, so that the task can be resumed on any compatible node in the cluster, enhancing the fault tolerance and flexibility of the system.

[0036] Working principle: After the system starts, the resource perception unit 1 begins to continuously monitor the infrastructure. After the user submits a task, the task parsing unit 2 parses the task attributes. All this information is sent to the intelligent decision engine 3, which produces the best scheduling strategy through prediction and optimization calculation. The dynamic scheduling executor 4 allocates computing and network resources according to the strategy, while the hierarchical storage manager 5 ensures that the data is located on the correct storage level. Throughout the process, the visual operation and maintenance center 8 provides full visibility and controllability.

[0037] It should be noted that in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying that these entities or actions exist in any such actual relationship or order. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or apparatus.

[0038] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An intelligent video resource scheduling and management system, characterized in that, The system includes the following modules: The resource monitoring module is used to collect and perceive the resource status data of each computing node in the distributed computing environment in real time. The resource status data includes CPU utilization, GPU utilization, memory usage, network bandwidth utilization, and storage I / O performance. The intelligent analysis module is communicatively connected to the resource monitoring module. It is used to receive the resource status data and, based on the task type and priority of the video task to be scheduled, analyze and predict its required resource needs through a preset intelligent prediction model. The task scheduling module is communicatively connected to the intelligent analysis module and is used to receive the dynamic resource scheduling strategy generated by the intelligent analysis module and allocate the video task to the optimal computing node that conforms to the strategy for execution. The storage management module, which is communicatively connected to the intelligent analysis module, is used to implement hierarchical storage management of video data based on the access frequency and importance of the video data, and to schedule hot data, warm data and cold data to storage media with different performance.

2. The intelligent video resource scheduling and management system according to claim 1, characterized in that: The intelligent prediction model preset in the intelligent analysis module is a machine learning model. This model is trained using historical task execution data and resource consumption data to predict the computing, storage, and network bandwidth requirements of new video tasks.

3. The intelligent video resource scheduling and management system according to claim 2, characterized in that: The scheduling strategy generation process of the intelligent analysis module is as follows: with the optimization objectives of minimizing the total completion time of all video tasks and maximizing the utilization rate of cluster resources, a multi-objective optimization model is constructed, and a heuristic algorithm or metaheuristic algorithm is used to solve the model to generate the optimal task allocation scheme.

4. The intelligent video resource scheduling and management system according to claim 1, characterized in that: The task scheduling module also supports a preemptive scheduling mechanism. When a high-priority task enters the system, it can pause or reschedule a low-priority task that is currently running, in order to ensure the service quality of the high-priority task.

5. The intelligent video resource scheduling and management system according to claim 1, characterized in that: The storage management module's hierarchical storage strategy dynamically adjusts the data level based on the video data's most recent access time, access frequency, and the task's preset retention period, and automatically triggers data migration between different storage levels.

6. The intelligent video resource scheduling and management system according to claim 1, characterized in that: The system also includes a visual interactive interface for real-time display of global resource status, task queues, scheduling decision-making process, and system performance metrics.

7. A method for intelligent video resource scheduling and management based on the system described in any one of claims 1-6, characterized in that, Includes the following steps: S1: Real-time collection of resource status data of each node in the distributed computing environment; S2: Receive video processing tasks and parse their task type and priority; S3: Based on the current resource status and task requirements, generate dynamic resource scheduling strategies through intelligent prediction model analysis; S4: According to the scheduling strategy, the task is assigned to the optimal computing node for execution, and appropriate storage resources are allocated to the task; S5: Continuously monitor resource status during task execution and dynamically adjust task scheduling strategies and data storage locations based on status changes.

8. The intelligent video resource scheduling and management method according to claim 7, characterized in that: The intelligent prediction model in step S3 is a regression model trained based on historical data, used to predict task execution time and resource consumption; the dynamic resource scheduling strategy is generated by solving an optimization problem with the goal of minimizing task completion time and load balancing.

9. The intelligent video resource scheduling and management method according to claim 7, characterized in that: The specific basis for "assigning the task to the optimal computing node" in step S4 includes: the network latency between the computing node and the task request source, the current load status of the computing node, and the data locality required by the task.

10. The intelligent video resource scheduling and management method according to claim 7, characterized in that: The dynamic adjustment in step S5 includes: when a computing node is detected to have degraded performance or failed, automatically migrating the tasks on it to other healthy nodes; and triggering a resource preemption and rescheduling process when a higher priority task is submitted.

Images