Video coding and decoding GPU (Graphics Processing Unit) scheduling self-adaptive method suitable for various GPU types

Abstract

The invention belongs to the technical field of video coding and decoding scheduling, and particularly relates to a video coding and decoding GPU scheduling adaptive method suitable for multiple GPU types. Comprising the following steps: S1, releasing a coupling relationship between a coding and decoding task and an operating system and GPU types; S2, establishing a dynamic pluggable detection mechanism for multiple types of GPUs; s3, a resource management and health detection mechanism of the GPU; s4, carrying out self-adaption on multiple types of GPUs; the method can adapt to different operating systems and GPU types; the method has the advantages that the method is compatible with products of mainstream GPU manufacturers, the currently used GPU is automatically detected, a proper encoding and decoding scheme is automatically selected according to the characteristics of the GPU, and the problems of too high CPU utilization rate and GPU type compatibility are solved. Efficient, stable and general operation of a video coding and decoding technology in a multi-platform and multi-GPU environment is realized, and remarkable technical progress and application value are brought to the field of video processing.

Description

Technical Field

[0001] The present invention belongs to the technical field of video codec scheduling, and relates to the design of video codec application layer architecture, including decoding, encoding, bitstream format conversion, picture generation, image processing and other scenarios; specifically, it relates to a video codec GPU scheduling adaptive method applicable to multiple GPU types. Background Art

[0002] With the rapid development of information technology, video coding and decoding technology has become an indispensable key technology in the field of modern multimedia, and is widely used in many fields such as video conferencing systems, online video platforms, digital TV broadcasting, video surveillance, and various mobile video applications. In these application scenarios, efficient transmission and processing of video data is crucial to providing a high-quality user experience.

[0003] Since the amount of video data is usually large, in order to facilitate transmission over the network, the original image data (common formats such as YUV, RGB, etc.) will be encoded and processed at the video source end (usually the server end), that is, it will be compressed into video stream data in a specific format (such as the widely used H.264, H.265, etc.). This compression operation can significantly reduce the bandwidth resources required for network transmission, improve transmission efficiency, and enable smooth transmission of videos in different network environments. However, the encoded and compressed video stream data cannot be directly presented on the display device, and must be decoded at the user end (i.e., the client) to restore it to the original image data, thereby realizing video playback. In addition, the decoded image data also provides a basis for a series of other extended applications, such as users can save the video frames of interest as pictures, or use image recognition technology to analyze video content, and improve video quality through image optimization algorithms.

[0004] In some complex video processing scenarios, there is also a need for bitstream format conversion. For example, when a video source obtains video stream data from other video sources, its bitstream format, bit rate, resolution and other parameters may not match the requirements of the current video source. In this case, it is necessary to first decode the video stream data into original image data, and then re-encode it into a suitable video stream according to the target requirements.

[0005] Although video coding and decoding technology plays a core role in the multimedia field, both decoding and encoding operations involve a large number of complex computing tasks, which are typical computing-intensive tasks. Under the current technical architecture, these tasks are highly dependent on CPU resources, especially the encoding process, which consumes a lot of CPU computing resources. This situation greatly limits the overall performance and efficiency of the video processing system, especially when processing large-scale video data or multi-task concurrent processing scenarios.

[0006] Specifically, the current relevant technologies have the following obvious deficiencies: 1. Unbalanced Utilization of Hardware Resources In the field of transcoding services, most existing technical solutions only support the use of the CPU for encoding and decoding operations. Although the CPU has certain advantages in general computing, its performance bottleneck becomes increasingly obvious when faced with highly parallel computing tasks such as video encoding and decoding. This single reliance on the CPU cannot fully tap the potential of other components in the computer hardware system, especially the powerful parallel computing capabilities of the GPU are completely ignored, resulting in low overall system resource utilization and the inability to achieve efficient video transcoding processing.

[0007] The situation is not optimistic in terms of image generation services. Currently, most image generation services only support GPUs from specific manufacturers (such as NVIDIA). This means that when the GPU in the hardware device used by the user is not a product of this specific manufacturer, the image generation service will not work properly. This strict dependence on specific hardware greatly limits the versatility and flexibility of image generation services, and makes users subject to many restrictions when choosing hardware devices, and cannot freely match hardware components according to their actual needs and budget.

[0008] Even in the decoding and display process of the client, although the GPU is used in the stage of outputting the decoded image data to the display device, the decoding process itself still relies solely on the CPU for calculation. This incomplete GPU application method fails to fully tap the potential value of the GPU in the entire video decoding process, resulting in low GPU utilization efficiency on the client side and failure to provide substantial performance improvement for the decoding process, thus affecting the smoothness and response speed of the client video playback.

[0009] 2. CPU solution performance is limited Solutions that rely solely on the CPU for video encoding and decoding have obvious limitations in processing power. Taking the common Intel Xeon E3 (with 64 cores) CPU as an example, when performing specific video encoding and decoding tasks, such as decoding the H.265 bitstream with a 1080P input and encoding the decoded data into a 1080P H.264 bitstream, a single task consumes a very high amount of CPU resources, about 150% to 200%. If the CPU resources occupied by the operating system itself and other necessary background processes are taken into account, the number of tasks that this pure CPU solution can support running at the same time under full load is extremely limited, about only 25 to 30. This means that when faced with large-scale video processing needs or actual application scenarios with concurrent access by multiple users, the system with a pure CPU solution can easily fall into a performance bottleneck and cannot meet the actual business needs of efficient and fast processing of video data, seriously affecting the user experience.

[0010] 3. Lack of hardware adaptability Technical solutions that support GPUs from specific manufacturers have serious defects in hardware compatibility. Since these solutions are tightly bound to GPU products from specific manufacturers, they will not work properly when facing different hardware platforms or other non-specific GPUs selected by users. This not only limits the user's autonomy in hardware selection, but also greatly hinders the promotion and application of video codec technology in different hardware environments. In today's diverse hardware market environment, technical solutions that lack hardware adaptability are difficult to meet the needs of a wide range of users, which is not conducive to the further development and popularization of video codec technology.

[0011] In summary, with the increasing popularity of video applications and the continuous growth of video data volume, existing video coding and decoding technologies face many challenges in terms of resource utilization efficiency, performance, and hardware adaptability. Therefore, this field urgently needs an innovative solution that can effectively unify the coding and decoding functions and achieve good adaptability to different operating systems and various types of GPUs, so as to give full play to the overall performance of the hardware system, improve the efficiency and versatility of video coding and decoding, and meet the growing needs of multimedia applications. Summary of the invention

[0012] In view of the limitations of the prior art, the present invention proposes a method for adaptive GPU scheduling of video codecs applicable to multiple GPU types. The method is committed to integrating codec functions, and can adapt to different operating systems and different types of GPUs; the method for adaptive GPU scheduling of video codecs proposed in the patent of the present invention is compatible with the products of mainstream GPU manufacturers, and can automatically detect the GPU currently in use, and automatically select the appropriate codec solution based on the characteristics of the GPU. This innovation aims to solve the problems of excessive CPU utilization and GPU compatibility. It realizes efficient, stable and universal operation of video codec technology in a multi-platform, multi-GPU environment, bringing significant technological progress and application value to the field of video processing.

[0013] To achieve the above object, the technical solution adopted by the present invention is: a video encoding and decoding GPU scheduling adaptive method applicable to multiple GPU types, comprising the following steps: Step S1, releasing the coupling relationship between the encoding and decoding task and the operating system and the GPU type: the encoding and decoding task no longer interacts directly with the GPU, but an abstract layer is used as an intermediary between the two, and the abstract layer is responsible for GPU resource management and GPU task processing; by isolating the encoding and decoding task, it is not affected by the current operating system type, GPU type and number, and specific operations, thereby releasing the coupling relationship between the encoding and decoding task and the operating system and GPU type; Step S2, dynamic pluggable detection mechanism for multiple types of GPUs: dynamically load the GPU detection module or use the driver-related modules installed on the target machine to detect different types of GPUs in a pluggable manner; obtain their capability sets, and add the GPU to the resource management and health detection module; Step S3, GPU resource management and health detection mechanism: centrally manage the application and release of GPU resources, including applying for GPU resources from the resource management module when receiving a codec task; returning the GPU resources to the resource management module when the codec task is completed; and performing a health detection on the GPU when a GPU-related error occurs during the operation of the GPU; Step S4, adaptation of multiple types of GPUs; abstracting the encoding and decoding operations of different types of GPUs through the GPU resources applied for by the resource management module, and performing interactive operations with the GPU using the corresponding API according to the currently applied GPU type.

[0014] Furthermore, in step S1, the abstract layer isolates the operating system type of the encoding and decoding tasks by maintaining the operating system interface mapping table and the GPU driver abstract interface, so that the encoding and decoding tasks have a unified calling method under different operating systems, and there is no need to perform separate code adaptation for different operating systems.

[0015] Furthermore, in step S2, the dynamic pluggable detection mechanism for multiple types of GPUs includes the following steps: Step S21, reading detectable GPU type information from a preset configuration file for initialization; Step S22, dynamically loading a GPU detection module or a driver-related module installed on the target machine; if the loading fails, it is deemed that there is no need to detect the GPU type or the GPU does not exist, and the process is skipped; Step S23, if the loading is successful in step S22, then detect whether the corresponding type of GPU exists; if it exists, obtain its capability set and add the GPU to the resource management and health detection module; if it does not exist, it is deemed that there is no GPU of this type on the current machine; in this implementation step, if the corresponding detection module exists, it is used; if it does not exist, it is skipped; making the program more flexible as a whole, no longer restricted by the requirement that all detection modules must exist to start normally; the design of each detection module adopts a pluggable manner.

[0016] Furthermore, in step S3, the application of GPU resources is implemented as follows: Step S311: When the encoding and decoding task needs to apply for GPU resources, it no longer directly applies to the GPU, but obtains them from the GPU resource management module; Step S312, determining whether there are free GPU resources in the current GPU resource management module, and if there are available GPU resources, marking the GPU resources as in use and returning them to the encoding and decoding task for use; executing the encoding and decoding task through GPU task processing; Step S313: In step S312, if the current resource management module has no available idle resources, then select the GPU with the lowest utilization rate from all current GPUs, apply for one channel of resources of the GPU, and return it to the encoding and decoding task for use; execute the encoding and decoding task through GPU task processing; Step S314: If there are no available GPU resources or the GPU resource application fails when the GPU resource management module applies for GPU resources, the encoding and decoding task is notified to use the CPU to execute the task.

[0017] Furthermore, in step S3, the application and release of GPU resources is implemented as follows: Step S321: When the encoding and decoding task is completed, the GPU resource is returned to the GPU resource management module; Step S322: The GPU resource management module marks the resource path as idle, and marks the current time as the idle start time of the resource path; Step S323: Add the resource to the GPU timing detection queue at the same time.

[0018] Furthermore, the GPU timing detection is implemented as follows: in the GPU timing detection queue, it is determined whether the idle time of each GPU resource has timed out; if the idle time of the GPU resource has timed out, the GPU resource management module is notified to release the resource.

[0019] Furthermore, the implementation method of GPU health detection is as follows: when a GPU-related error occurs during the encoding and decoding process, the GPU health detection logic needs to be triggered. At this time, all resources belonging to the GPU need to be recovered and the GPU needs to be removed from the resource management module, and an attempt is made to reset the GPU: if the status returns to normal after the reset, the GPU is added back to the resource management module; if the status remains the same after the reset, the GPU is abandoned.

[0020] Furthermore, in step S4, the corresponding API includes that under the Linux system, NvidiaGPU uses a specific encoding and decoding API in the image processing library CUDA in its driver, AMDGPU uses the relevant API in the VDPAU standard library, and IntelGPU or core graphics uses the corresponding API in the VAAPI standard library; under the Windows system, the GPU operation API in DirectX9 is uniformly used, and the abstract layer performs necessary parameter conversion and error handling when calling these APIs to ensure stable interaction of encoding and decoding tasks with different GPUs.

[0021] Furthermore, it also includes adaptive implementation methods for various types of GPUs and CPUs, specifically: The GPU resources applied for by the GPU resource management module abstract the encoding and decoding operations of different types of GPUs, and interact with the GPU using the corresponding API according to the currently applied GPU type; The encoding and decoding task no longer interacts directly with the GPU. It does not need to care about the specific GPU type and its specific interaction method. Instead, it uses the GPU resources that have been applied for: the encoding and decoding task inputs the bitstream data to the resource and specifies the output method in the form of a callback function. The resource processes the input bitstream data through the corresponding API and outputs it to the encoding and decoding task through the established output method. If the GPU resource application fails, it means that the GPU load is too high or there is no available GPU. In this case, the CPU will automatically perform the encoding and decoding tasks.

[0022] The present invention adopts the above technical solution, which has at least the following beneficial effects: 1. Efficient GPU resource utilization and task acceleration: The multi-type GPU dynamic detection and resource management mechanism adopted by the present invention can accurately identify and adapt to various types of GPUs commonly seen on the market (such as NVIDIA, AMD, Intel, etc.). In a system with GPU hardware, the powerful parallel computing capability of the GPU is fully utilized, and complex video encoding and decoding tasks are efficiently allocated to the GPU for processing. For example, when processing the encoding and decoding of high-definition video (1080P and above resolution), the GPU can process multiple pixels or data blocks in parallel, which significantly shortens the processing time compared to the traditional pure CPU solution, and greatly improves the overall performance of the system. Taking actual testing as an example, when processing the same video encoding and decoding task, the processing speed after using GPU acceleration can be increased by several times or even dozens of times compared to using only the CPU, which greatly improves the throughput of the system, especially in a multi-tasking environment. The advantage is more obvious, and more video data can be processed at the same time to meet a large number of concurrent video processing requirements. Even in the case of intensive tasks, such as when a video processing server simultaneously processes video editing, transcoding or playback requests from many users, the mechanism can reasonably allocate GPU resources to ensure that each task can obtain the required computing resources in a timely manner, avoid performance degradation caused by resource competition, always maintain an efficient and stable operating state, and effectively improve the user experience.

[0023] 2. Flexible CPU-GPU adaptive switching When the system detects that there is no GPU hardware, it can automatically and seamlessly switch to the CPU to execute tasks, ensuring that the system can run video encoding and decoding work normally under different hardware configurations. This adaptive ability makes the system widely applicable, whether it is on professional workstations equipped with high-performance GPUs or on ordinary devices with only basic CPU computing capabilities, it can provide stable and reliable video processing services. In some scenarios where real-time requirements are not high but computing resources are limited, such as when a low-configuration laptop performs simple video playback or light editing, even without GPU support, the system can minimize CPU consumption and avoid system freezes while ensuring the normal operation of basic functions through the optimized CPU execution solution, extending device battery life (for mobile devices such as laptops) and improving the overall efficiency of the device.

[0024] GPU health detection ensures reliable system operation The innovative health detection mechanism constantly monitors the status of the GPU during operation. Once a GPU-related error occurs during the encoding and decoding process, the deep health detection logic is immediately triggered. For example, when it is detected that the GPU temperature is too high, the core frequency fluctuates abnormally, there is a video memory error, or other signs of hardware failure, the system quickly reclaims the resources occupied by all tasks being executed on the GPU, removes it from the resource management module, and promptly attempts to reset the operation. If the GPU status returns to normal after the reset, the system will carefully re-add it to the resource management module and closely monitor its operating status in subsequent task allocation to ensure its stable operation. If multiple reset attempts fail, the faulty GPU will be decisively abandoned to avoid serious consequences such as system crashes or data loss due to the continued operation of the faulty GPU, effectively ensuring the stability and reliability of the entire video encoding and decoding system, reducing system downtime caused by hardware failures, and providing users with continuous and uninterrupted video processing services.

[0025] 4. Optimize resource management to ensure mission continuity The resource management module carefully manages the application and release of GPU resources during normal GPU operation. When a codec task applies for GPU resources, it intelligently allocates resources based on the current resource usage, giving priority to tasks with high priority or long waiting times, ensuring that the task can obtain the required computing power in a timely manner. After the task is completed, the resources are quickly recovered and marked as idle, and the timing detection mechanism is started to release the idle resources for a long time in a timely manner to optimize resource utilization efficiency.

[0026] When a GPU fails, the resource management module works together with the health detection mechanism to quickly adjust the resource allocation strategy and smoothly migrate the tasks originally assigned to the failed GPU to other available GPUs (if any) or CPUs for continued execution, ensuring the continuity of video encoding and decoding tasks and avoiding interruption of the entire task process due to a single GPU failure, thereby ensuring the normal operation of the business and improving the robustness of the system.

[0027] 5. Unified interface simplifies the development process By building an abstraction layer between the codec task and the GPU, the present invention realizes the unification of the codec interface. When developing video codec applications, developers do not need to delve into the complex underlying architecture and driver details of different types of GPUs, nor do they need to write specific codes for different operating systems. Efficient interaction with various GPUs can be achieved by programming based on the standardized interface provided by the abstraction layer, which greatly reduces the difficulty and workload of development.

[0028] This unified interface design makes the development process more efficient and stable, reducing development errors and compatibility issues caused by differences in different GPUs or operating systems. Developers can focus more on optimizing video encoding and decoding algorithms and innovating application functions, shortening the development cycle, speeding up the time to market, and improving the company's competitiveness in the field of video processing.

[0029] 6. Wide adaptation reduces maintenance costs and hardware dependence The dynamic detection and resource management mechanism of multiple types of GPUs ensures the system's wide adaptability to different GPU models and different operating systems (such as Linux, Windows, etc.). Whether it is large-scale deployment of video processing services in enterprise-level data centers or individual users using video editing software on different devices, the present invention can automatically adapt to the hardware environment without the need for users to manually perform complex configuration and adjustments.

[0030] This high degree of adaptability reduces the system's dependence on specific hardware and reduces system maintenance costs caused by hardware upgrades or replacements. When users replace GPU devices or upgrade operating systems, the system can automatically detect and adapt to the new environment and continue to run stably without the need to redevelop or modify software on a large scale, extending the service life of the system and protecting users' investments. It also provides a solid foundation for the widespread application of video processing technology on different hardware platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 It is a schematic diagram of the overall architecture of the present invention.

[0033] Figure 2 It is a task processing schematic diagram of the present invention.

[0034] Figure 3 It is a GPU detection flow chart of the present invention.

[0035] Figure 4 This is a flowchart of GPU resource application according to the present invention.

[0036] Figure 5 This is a flow chart of GPU resource release according to the present invention.

[0037] Figure 6 This is a flow chart of GPU resource timing detection of the present invention.

[0038] Figure 7 This is a flow chart of GPU health detection of the present invention. DETAILED DESCRIPTION

[0039] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0040] Embodiment 1: This embodiment provides a method for adaptively scheduling video encoding and decoding GPUs applicable to multiple GPU types, including the following steps: Step S1, decoupling the encoding and decoding task from the operating system and the GPU type: the encoding and decoding task no longer interacts directly with the GPU, but an abstract layer is used as an intermediary between the two, such as Figure 1The figure shows a schematic diagram of the overall architecture of this embodiment, in which the abstract layer is responsible for GPU resource management and GPU task processing; by isolating the encoding and decoding tasks, they are not affected by the current operating system type, GPU type and number, and specific operations, thereby releasing the coupling relationship between the encoding and decoding tasks and the operating system and GPU type; the operating system in the overall architecture of this embodiment includes a Linux system and / or a Windows system; the machine device includes a GPU driver, and is provided with N GPU units, where N is a natural number greater than or equal to 1.

[0041] Step S2, dynamic pluggable detection mechanism for multiple types of GPUs: dynamically load the GPU detection module or use the driver-related modules installed on the target machine to detect different types of GPUs in a pluggable manner; obtain their capability sets, and add the GPU to the resource management and health detection module; Step S3, GPU resource management and health detection mechanism: centrally manage the application and release of GPU resources, including applying for GPU resources from the resource management module when receiving a codec task; returning the GPU resources to the resource management module when the codec task is completed; and performing a health detection on the GPU when a GPU-related error occurs during the operation of the GPU; Step S4, adaptation of multiple types of GPUs; abstracting the encoding and decoding operations of different types of GPUs through the GPU resources applied for by the resource management module, and performing interactive operations with the GPU using the corresponding API according to the currently applied GPU type.

[0042] As an implementation method, in this embodiment, in step S1, the abstract layer implements operating system type isolation for encoding and decoding tasks by maintaining the operating system interface mapping table and the GPU driver abstract interface, so that the encoding and decoding tasks have a unified calling method under different operating systems, and there is no need to perform separate code adaptation for different operating systems.

[0043] like Figure 3 As shown, as an implementation mode, in this embodiment, in step S2, the dynamic pluggable detection mechanism of multiple types of GPUs includes the following steps: Step S21, reading detectable GPU type information from a preset configuration file for initialization; Step S22, dynamically loading a GPU detection module or a driver-related module installed on the target machine; if the loading fails, it is deemed that there is no need to detect the GPU type or the GPU does not exist, and the process is skipped; Step S23, if the loading is successful in step S22, then detect whether the corresponding type of GPU exists; if it exists, obtain its capability set and add the GPU to the resource management and health detection module; if it does not exist, it is deemed that there is no GPU of this type on the current machine; in this implementation step, if the corresponding detection module exists, it is used; if it does not exist, it is skipped; making the program more flexible as a whole, no longer restricted by the requirement that all detection modules must exist to start normally; the design of each detection module adopts a pluggable manner.

[0044] like Figure 4 As shown, as an implementation method, in this embodiment, in step S3, the application of GPU resources is implemented as follows: Step S311: When the encoding and decoding task needs to apply for GPU resources, it no longer directly applies to the GPU, but obtains them from the GPU resource management module; Step S312: determine whether there are free GPU resources in the current GPU resource management module. If there are available GPU resources, mark the GPU resources as in use and return the GPU resources to the encoding and decoding task for use; execute the encoding and decoding task through GPU task processing; Step S313: In step S312, if the current resource management module has no available idle resources, then select a GPU with the lowest utilization rate from all current GPUs, try to apply for resources on the GPU, that is, apply for one channel of resources of the GPU, and return it to the encoding and decoding task for use; execute the encoding and decoding task through GPU task processing; Step S314: If there are no available GPU resources or the GPU resource application fails when the GPU resource management module applies for GPU resources, the encoding and decoding task is notified to use the CPU to execute the task.

[0045] like Figure 5 As shown, as an implementation mode, in this embodiment, in step S3, the application release of GPU resources is implemented as follows: Step S321: When the encoding and decoding task is finished, the encoding and decoding task releases resources and returns the GPU resources to the GPU resource management module; Step S322: The GPU resource management module marks the resource path as idle, and marks the current time as the idle start time of the resource path; Step S323: Add the resource to the GPU timing detection queue at the same time.

[0046] like Figure 6As shown, as an implementation method, in this embodiment, the implementation method of GPU timing detection is: in the GPU timing detection queue, determine whether the idle time of each GPU resource has timed out; if the idle time of the GPU resource has timed out, notify the GPU resource management module to release the resource.

[0047] like Figure 7 As shown, as an implementation method, in this embodiment, the implementation method for GPU health detection is: when a GPU-related error occurs during the encoding and decoding process, it is necessary to trigger the GPU health detection logic. At this time, it is necessary to recycle all resources belonging to the GPU and remove the GPU from the resource management module, and try to reset the GPU: if the status returns to normal after the reset, the GPU is added back to the resource management module; if the status remains the same after the reset, the GPU is abandoned.

[0048] like Figure 2 As shown, as an implementation mode, in this embodiment, in step S4, the corresponding API includes that under the Linux system, NvidiaGPU uses the specific encoding and decoding API in the image processing library CUDA in its driver, AMDGPU uses the relevant API in the VDPAU standard library, and IntelGPU or core graphics uses the corresponding API in the VAAPI standard library; under the Windows system, the GPU operation API in DirectX9 is uniformly used, and the abstract layer performs necessary parameter conversion and error handling when calling these APIs to ensure stable interaction of encoding and decoding tasks with different GPUs.

[0049] As an implementation method, this embodiment also includes adaptive implementation methods of multiple types of GPUs and CPUs, specifically: The GPU resources applied for by the GPU resource management module abstract the encoding and decoding operations of different types of GPUs, and interact with the GPU using the corresponding API according to the currently applied GPU type; The encoding and decoding task no longer interacts directly with the GPU. It does not need to care about the specific GPU type and its specific interaction method. Instead, it uses the GPU resources that have been applied for: the encoding and decoding task inputs the bitstream data to the resource and specifies the output method in the form of a callback function. The resource processes the input bitstream data through the corresponding API and outputs it to the encoding and decoding task through the established output method. If the GPU resource application fails, it means that the GPU load is too high or there is no available GPU. In this case, the CPU will automatically perform the encoding and decoding tasks.

[0050] Embodiment 2 1. System initialization phase GPU detection module loading and GPU identification When the system starts, it first enters the GPU detection process. According to the preset detection order, try to load the detection modules for different types of GPUs. For example, for NVIDIA GPU, try to load the detection module for detecting NVIDIA GPU. If the module is loaded successfully, it communicates with the GPU hardware through its built-in detection algorithm to determine whether this type of GPU exists in the target machine. If it does, further obtain the detailed capability set information of the GPU, including key parameters such as the number of computing cores, video memory capacity, supported instruction sets, and maximum clock frequency, and record this information in the system's GPU information library. At the same time, mark the GPU as available and include it in the GPU resource management module to prepare for subsequent task allocation.

[0051] If a GPU detection module fails to load, the system does not immediately report an error and stop, but records the loading failure information and continues to try to load the next detection module. For example, if the NVIDIAGPU detection module fails to load, the system then tries to load the AMDGPU detection module. If all the detection modules that come with this program fail to load, the system will further check the driver-related modules installed on the target machine. If a driver-related module that can be used to detect a specific type of GPU is found, it is used to perform GPU detection operations. If there are no available driver-related modules on the target machine or the GPU cannot be detected through these modules, the GPU of that type is marked as unavailable, and the subsequent detection process is skipped, and the detection of other possible GPU types continues.

[0052] Abstract layer initialization and system decoupling After completing the GPU detection, the system initializes the abstraction layer. The abstraction layer implements the isolation of the operating system type for the encoding and decoding tasks by loading the underlying interface drivers that interact with the GPU under different operating systems and establishing an internal interface mapping table. For example, under the Linux system, the abstraction layer loads the CUDA library interface driver that interacts with the NVIDIA GPU, the VDPAU standard library interface driver that interacts with the AMD GPU, and the VAAPI standard library interface driver that interacts with the Intel GPU or core graphics card; under the Windows system, the GPU interaction interface driver related to DirectX9 is loaded. In this way, the encoding and decoding tasks do not need to care about the differences in the underlying operating systems when calling the abstraction layer interface, thus achieving a unified operation method.

[0053] At the same time, the abstract layer builds GPU resource management and task processing logic. It maintains a GPU resource status table to record the resource usage of each GPU, such as the number of assigned tasks, remaining video memory, computing core utilization, and other information. For encoding and decoding tasks, the abstract layer provides a unified task submission interface. The task only needs to pass relevant parameters (such as bitstream data, task type, output requirements, etc.) to the abstract layer. The abstract layer will decide to assign the task to the appropriate GPU for execution based on the internal resource management strategy and GPU status information, thereby uncoupling the encoding and decoding task from the type and number of GPUs and specific operations.

[0054] 2. Encoding and decoding task execution phase Task submission and GPU resource application When a user initiates a codec task (such as video decoding and playback, video encoding conversion, bitstream format conversion, picture generation, image processing, etc.), the task is first submitted to the abstract layer. The abstract layer puts the task into a waiting queue based on the task type and priority (the priority can be preset or dynamically determined based on factors such as the task source and urgency).

[0055] The task scheduling module takes the task out of the waiting queue and applies for GPU resources from the GPU resource management module. After receiving the application, the GPU resource management module first queries the GPU resource status table maintained by itself to determine whether there are idle GPU resources that meet the task requirements. If there are idle resources, the resource management module marks the resource as in use and allocates it to the encoding and decoding task. At the same time, it binds the task to the allocated GPU resource and records the relevant resource allocation information (such as task number, GPU ID, allocation time, etc.). If there are no idle resources, the resource management module starts the resource evaluation algorithm, traverses all the managed GPUs in the system, obtains the real-time resource usage rate, task queue length, remaining video memory and other information of each GPU, selects the most suitable GPU according to the preset resource selection strategy (such as giving priority to GPUs with low load and not frequently used recently, or selecting the most suitable GPU architecture according to the task type, etc.), and then allocates one resource from the selected GPU to the encoding and decoding task.

[0056] If, during the GPU resource application process, the GPU resource management module finds that there are no available GPU resources (e.g., all GPUs are fully loaded and new tasks cannot be assigned) or fails to apply for GPU resources (e.g., encountering GPU hardware failure or driver abnormality), the task scheduling module is notified, and the task scheduling module transfers the task to the CPU execution queue and adjusts the task execution parameters (e.g., reducing the concurrency of tasks, optimizing memory usage strategies, etc.) to adapt to the CPU execution environment. At the same time, the GPU resource monitoring thread is started to continuously monitor the GPU resource status. When GPU resources become available and meet the task requirements, the task is migrated from the CPU execution queue back to the GPU execution queue according to the preset task migration strategy (e.g., migrating high-priority tasks first, based on the remaining execution time of the task, etc.), and GPU resources are re-applied for execution.

[0057] Execution of encoding and decoding tasks on GPU Once the encoding and decoding task successfully obtains GPU resources, the task and the GPU will exchange data and perform operations through the abstract layer. For video decoding tasks, the encoding and decoding task will input the bitstream data that conforms to a specific format (such as H.264, H.265, etc.) into the GPU resources according to the predetermined high-speed data transmission interface specifications (such as PCIe direct memory access, etc.). The GPU resources call the corresponding API through the abstract layer (under the Linux system, NvidiaGPU uses the specific decoding API in the image processing library CUDA in its driver, AMDGPU uses the decoding-related API in the VDPAU standard library, and IntelGPU or core graphics uses the decoding corresponding API in the VAAPI standard library; under the Windows system, the decoding operation API in DirectX9 is used uniformly) to perform decoding operations. During the decoding process, the GPU uses its parallel computing capabilities to quickly process the bitstream data and convert it into image data (such as YUV, RGB, etc.).

[0058] For video encoding tasks, the process is similar but the operation is reversed. The encoding and decoding task inputs the original image data into the GPU resources according to the predetermined interface specifications, and the GPU resources call the corresponding encoding API to encode the image data into a bitstream data in a specific format. During the encoding process, the GPU can process multiple image blocks in parallel according to the characteristics of the image content, optimize the encoding efficiency, and improve the encoding quality (such as reducing the bit rate, increasing the resolution, etc.). For other types of encoding and decoding tasks, such as image generation tasks (decoding the original bitstream data into image data and then encoding it into a picture format) and image processing tasks (filtering, scaling, rotating, etc. the decoded image data and then encoding it back to the original format or other formats), they also interact with the GPU through the abstract layer, using the computing power of the GPU to accelerate the task processing process.

[0059] During the task execution process, the abstraction layer is responsible for monitoring the task execution status and interacting with the GPU in real time, such as obtaining GPU processing progress, processing results and other information, and feeding this information back to the encoding and decoding task. If an API call error, data transmission error or other abnormal situation occurs during the interaction, the abstraction layer immediately starts the error handling mechanism. For example, it automatically tries to reinitialize the connection with the GPU and call the API again. If multiple attempts fail, detailed error information is recorded and the resource management module is notified. Depending on the error situation, the resource management module considers taking further measures, such as switching to other available GPUs or continuing to execute tasks on the CPU (if the task has been transferred to the CPU execution queue), to ensure that the task can be completed as smoothly as possible while ensuring the stability and reliability of the system.

[0060] 3. Mission completion and resource recovery phase GPU resource release When the encoding and decoding task is completed, the task sends a task end signal to the abstract layer. After receiving the signal, the abstract layer notifies the GPU resource management module to reclaim the GPU resources occupied by the task. The resource management module first releases the binding relationship between the task and the GPU resources, marks the GPU resources as idle, and updates the relevant information in the GPU resource status table, such as resource usage time, remaining video memory, etc. At the same time, the resource management module adds the resource to the GPU timing detection queue, preparing for subsequent idle resource management operations.

[0061] Resource idle detection and deep cleaning In the GPU timing detection queue, the system detects the idle resources in the queue at preset time intervals (such as every 5 minutes). For each idle GPU resource, the system checks whether its idle time exceeds the preset recycling threshold (such as 10 minutes). If the idle time does not exceed the threshold, the resource continues to remain in the timing detection queue waiting for the next detection. If the idle time exceeds the threshold, the system starts the resource deep cleaning program. The program performs a series of cleanup operations on idle resources, including clearing residual data in the video memory, resetting GPU-related status bits, and shutting down unnecessary GPU hardware function modules (such as shutting down some computing cores or video memory channels in power reduction mode), etc., to ensure that the resources are in a clean and available state. After the cleanup is completed, the resource management module officially releases the GPU resource and returns it to the system resource pool, which can be applied for by other subsequent encoding and decoding tasks.

[0062] GPU health recheck (optional) During the resource release process, the system can choose whether to perform a quick health check on the GPU that has just released resources (this step can be configured to be enabled or disabled based on system performance and requirements). If the health check is enabled, the system sends a series of simple test instructions to the GPU, such as executing a short-term computing task, checking the video memory read and write speed, etc., to quickly confirm whether the GPU is in a normal state after the task is executed. If an abnormal situation is detected (such as an error in the calculation result, a significant decrease in the video memory read and write speed, etc.), the system records the abnormal information and marks the GPU as suspicious. In the subsequent GPU resource allocation process, the resource management module will conduct more stringent monitoring of the GPU in the suspicious state, or, if there are other available GPUs, prioritize the allocation of tasks to other normal GPUs to avoid allocating tasks to GPUs that may have problems, further ensuring the stability and reliability of the system. If the health check result is normal, the GPU continues to remain in a normal and available state, waiting for the next task allocation.

[0063] Through the above specific implementation methods, each link and operation process of the present invention in practical application is elaborated in detail, from system initialization, task execution to task termination and resource recovery, and comprehensively demonstrates how to realize dynamic pluggable detection of multiple types of GPUs, resource management and health detection of GPUs, and adaptation of multiple types of GPUs and CPUs, ensuring the feasibility and effectiveness of the invention, and providing clear implementation guidance for technicians in related fields. In the actual implementation process, the parameters, algorithms and processes in the above implementation methods can be further optimized and adjusted according to specific needs and hardware environment to achieve the best performance and effect.

[0064] Although the embodiments of the present invention have been shown and described above, it will be appreciated that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, substitute and vary the above embodiments within the scope of the present invention.

Claims

1. A video encoding and decoding GPU scheduling adaptive method applicable to various GPU types, characterized by: The steps include: Step S1, releasing the coupling relationship between the encoding and decoding task and the operating system and the GPU type: the encoding and decoding task no longer interacts directly with the GPU, but an abstract layer is used as an intermediary between the two, and the abstract layer is responsible for GPU resource management and GPU task processing; by isolating the encoding and decoding task, it is not affected by the current operating system type, GPU type and number, and specific operations, thereby releasing the coupling relationship between the encoding and decoding task and the operating system and GPU type; Step S2, dynamic pluggable detection mechanism for multiple types of GPUs: dynamically load the GPU detection module or use the driver-related modules installed on the target machine to detect different types of GPUs in a pluggable manner; obtain their capability sets, and add the GPU to the resource management and health detection module; Step S3, GPU resource management and health detection mechanism: centrally manage the application and release of GPU resources, including applying for GPU resources from the resource management module when receiving a codec task; returning the GPU resources to the resource management module when the codec task is completed; and performing a health detection on the GPU when a GPU-related error occurs during the operation of the GPU; Step S4, adaptation of multiple types of GPUs; abstracting the encoding and decoding operations of different types of GPUs through the GPU resources applied for by the resource management module, and performing interactive operations with the GPU using the corresponding API according to the currently applied GPU type.

2. The method according to claim 1, characterized in that: In step S1, the abstract layer isolates the operating system type of the encoding and decoding tasks by maintaining the operating system interface mapping table and the GPU driver abstract interface, so that the encoding and decoding tasks have a unified calling method under different operating systems, and there is no need to perform separate code adaptation for different operating systems.

3. The method according to claim 1, characterized in that: In step S2, the dynamic pluggable detection mechanism for multiple types of GPUs includes the following steps: Step S21, reading detectable GPU type information from a preset configuration file for initialization; Step S22, dynamically loading a GPU detection module or a driver-related module installed on the target machine; if the loading fails, it is deemed that there is no need to detect the GPU type or the GPU does not exist, and the process is skipped; Step S23, if the loading is successful in step S22, then detect whether the corresponding type of GPU exists; if it exists, obtain its capability set and add the GPU to the resource management and health detection module; if it does not exist, it is deemed that there is no GPU of this type on the current machine; in this implementation step, if the corresponding detection module exists, it is used; if it does not exist, it is skipped; making the program more flexible as a whole, no longer restricted by the requirement that all detection modules must exist to start normally; the design of each detection module adopts a pluggable manner.

4. The method according to claim 3, characterized in that: In step S3, the application of GPU resources is implemented as follows: Step S311: When the encoding and decoding task needs to apply for GPU resources, it no longer directly applies to the GPU, but obtains them from the GPU resource management module; Step S312, determining whether there are free GPU resources in the current GPU resource management module, and if there are available GPU resources, marking the GPU resources as in use and returning them to the encoding and decoding task for use; executing the encoding and decoding task through GPU task processing; Step S313: In step S312, if the current resource management module has no available idle resources, then select the GPU with the lowest utilization rate from all current GPUs, apply for one channel of resources of the GPU, and return it to the encoding and decoding task for use; execute the encoding and decoding task through GPU task processing; Step S314: If there are no available GPU resources or the GPU resource application fails when the GPU resource management module applies for GPU resources, the encoding and decoding task is notified to use the CPU to execute the task.

5. The method according to claim 4, characterized in that: In step S3, the application and release of GPU resources is implemented as follows: Step S321: When the encoding and decoding task is completed, the GPU resource is returned to the GPU resource management module; Step S322: The GPU resource management module marks the resource path as idle, and marks the current time as the idle start time of the resource path; Step S323: Add the resource to the GPU timing detection queue at the same time.

6. The method according to claim 5, characterized in that: The implementation method of GPU timing detection is: in the GPU timing detection queue, determine whether the idle time of each GPU resource has timed out; if the idle time of the GPU resource has timed out, notify the GPU resource management module to release the resource.

7. The method according to any one of claims 3 to 6, characterized in that: The implementation method for GPU health detection is as follows: when a GPU-related error occurs during the encoding and decoding process, the GPU health detection logic needs to be triggered. At this time, all resources belonging to the GPU need to be recovered and the GPU needs to be removed from the resource management module, and an attempt is made to reset the GPU: if the status returns to normal after the reset, the GPU is added back to the resource management module; if the status remains the same after the reset, the GPU is abandoned.

8. The method according to claim 1, characterized in that: In step S4, the corresponding API includes that under Linux system, NvidiaGPU uses the specific encoding and decoding API in the image processing library CUDA in its driver, AMDGPU uses the relevant API in the VDPAU standard library, and IntelGPU or core graphics uses the corresponding API in the VAAPI standard library; under Windows system, the GPU operation API in DirectX9 is uniformly used, and the abstract layer performs necessary parameter conversion and error handling when calling these APIs to ensure stable interaction of encoding and decoding tasks with different GPUs.

9. The method according to any one of claims 1 to 6 or claim 8, characterized in that: It also includes adaptive implementations for various types of GPUs and CPUs, specifically: The GPU resources applied for by the GPU resource management module abstract the encoding and decoding operations of different types of GPUs, and interact with the GPU using the corresponding API according to the currently applied GPU type; The encoding and decoding task no longer interacts directly with the GPU. It does not need to care about the specific GPU type and its specific interaction method. Instead, it uses the GPU resources that have been applied for: the encoding and decoding task inputs the bitstream data to the resource and specifies the output method in the form of a callback function. The resource processes the input bitstream data through the corresponding API and outputs it to the encoding and decoding task through the established output method. If the GPU resource application fails, it means that the GPU load is too high or there is no available GPU. In this case, the CPU will automatically perform the encoding and decoding tasks.

Images