JPEG encoding and decoding method and system based on heterogeneous resource scheduling

Abstract

The present invention discloses a JPEG encoding and decoding method and system based on heterogeneous resource scheduling. The JPEG encoding and decoding method based on heterogeneous resource scheduling of the present invention includes the following steps: obtaining all JPEG encoding and decoding hardware resources in the system; realizing priority division of all JPEG encoding and decoding hardware resources under each given scheduling strategy; obtaining JPEG encoding and decoding tasks, determining the priority of JPEG encoding and decoding tasks calling JPEG encoding and decoding hardware resources according to the scheduling strategy of JPEG encoding and decoding tasks; calling JPEG encoding and decoding hardware resources using the determined priority to execute JPEG encoding and decoding tasks. The present invention aims to realize reasonable scheduling according to the use of heterogeneous resources for various heterogeneous hardware resources available in the system, such as CPU, GPU and hardware codec, etc., to make full use of the heterogeneous resources of the system, improve the efficiency of JPEG encoding and decoding, and avoid idle or overused resources.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a JPEG encoding and decoding method and system based on heterogeneous resource scheduling. Background Art

[0002] JPEG is a widely used image encoding format that can compress the original image into a smaller file while maintaining high image quality. Therefore, most images are still in JPEG format and are very popular. However, JPEG encoding and decoding requires a lot of computing resources. If only CPU resources are used to encode and decode it, a lot of CPU resources will be occupied. Nowadays, computer systems have more and more abundant resources, and there are more and more heterogeneous resources, such as GPU, hardware codec, TPU, etc. More and more researchers use CPU and various heterogeneous resources to solve various problems. Therefore, using heterogeneous resources to encode and decode JPEG has become a hot research field. Existing studies use only one heterogeneous resource alone. There are few studies on how to use multiple heterogeneous resources together, and the JPEG encoding and decoding problem based on heterogeneous resource scheduling is not considered. For example, use CPU plus one heterogeneous resource to encode and decode JPEG, or use the parallel computing power of GPU (CUDA or OPENCL) to encode and decode JPEG, or only use JPEG hardware codec to encode and decode it. But now computer system hardware resources are becoming more and more abundant, and there are more and more heterogeneous resources. For example, a graphics card usually contains GPU and hardware codec. In addition, CPU SIMD acceleration instructions can also accelerate JPEG encoding and decoding. However, current systems often only use one resource to encode and decode it, and do not fully utilize hardware resources.

[0003] The Chinese patent application document with application number CN201510531805.3 discloses a method and device for collaborative decoding of images based on CPU and GPU. The method includes: receiving an image coding stream, calling the CPU to perform header information parsing and tier-2 algorithm processing on the image coding stream to obtain a first processing result; extracting the image resolution value, judging whether the resolution value is higher than a preset resolution threshold, and if so, calling the GPU to perform tier-1 algorithm processing on the first processing result to obtain a second processing result; calling the GPU to sequentially perform inverse quantization processing, inverse wavelet transform and DC level conversion on the second processing result to obtain decoded image data. It can be seen that, on the one hand, the CPU load is reduced without affecting the decoding speed, and on the other hand, the idle GPU resources are fully utilized, thereby achieving load balancing of the CPU and GPU and increasing the overall throughput of the system. Although the scheme meets the requirements of using GPU heterogeneous resources by designing image collaborative decoding based on CPU and GPU, it only calls the GPU to perform tier-1 algorithm processing on the first processing result, and does not involve the problem of JPEG encoding and decoding based on heterogeneous resource scheduling.

[0004] The Chinese patent application document with application number CN201710031852.0 proposes a multi-dimensional scheduling system for heterogeneous resources, including: a cluster management module, a container management module, a resource coordination module, a hardware abstraction module, a relational database and a component communication module; the resource coordination module is connected to the cluster management module, the container management module, the hardware abstraction module, the relational database and the component communication module respectively; the resource coordination module includes a cluster resource allocation component and a host resource management component; and the cluster resource allocation component is used to manage each secondary scheduling framework and the host resource management component connected to the cluster, and allocate the resources on the host resource management component to the secondary scheduling framework according to a preset strategy; the host resource management component is used to receive and execute commands from the cluster resource allocation component, manage container tasks on the node, and allocate resources to each container; the cluster management module is used to manage cluster resources in the system; the container management module is used to manage container resources in the system; the hardware abstraction module is used to provide hardware support for the system, including physical servers and virtual servers; the component communication module is used to provide communication for the system. Although this solution meets the heterogeneous resource scheduling requirements for clusters by designing a multi-dimensional scheduling system for heterogeneous resources, it does not involve the issue of JPEG encoding and decoding based on heterogeneous resource scheduling. Summary of the invention

[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a JPEG encoding and decoding method and system based on heterogeneous resource scheduling are provided. The present invention aims to realize reasonable scheduling according to the usage of heterogeneous resources for various heterogeneous hardware resources available in the system, such as CPU, GPU and hardware codec, so as to make full use of the heterogeneous resources of the system, improve the efficiency of JPEG encoding and decoding, and avoid idle or excessive use of resources.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A JPEG encoding and decoding method based on heterogeneous resource scheduling includes the following steps:

[0008] S101, obtaining all JPEG encoding and decoding hardware resources in the system;

[0009] S102, implementing priority division of all JPEG encoding and decoding hardware resources under each given scheduling strategy;

[0010] S103, obtaining a JPEG codec task, and determining the priority of the JPEG codec task in calling the JPEG codec hardware resources according to the scheduling strategy of the JPEG codec task;

[0011] S104: Use the determined priority to call the JPEG encoding and decoding hardware resources to execute the JPEG encoding and decoding task.

[0012] Furthermore, the JPEG encoding and decoding hardware resources in step S101 include part or all of the CPU, GPU, and hardware codec.

[0013] Furthermore, when the CPU executes the JPEG encoding and decoding task, it calls SIMD instructions to perform JPEG encoding and decoding, when the GPU executes the JPEG encoding and decoding task, it uses CUDA or OPENCL to perform multi-path parallel JPEG encoding and decoding, and when the hardware codec executes the JPEG encoding and decoding task, it uses its own hardware codec to perform JPEG encoding and decoding.

[0014] Furthermore, the scheduling strategies in step S102 include two types: a high real-time scheduling strategy and a high throughput scheduling strategy. Under the high real-time scheduling strategy, the priority order of the CPU, GPU and hardware codec is: hardware codec>CPU>GPU; under the high throughput scheduling strategy, the priority order of the CPU, GPU and hardware codec is: GPU>CPU>hardware codec.

[0015] Furthermore, in step S103, when determining the priority of the JPEG codec task calling the JPEG codec hardware resources according to the scheduling strategy of the JPEG codec task, if the scheduling strategy of the JPEG codec task is a high real-time scheduling strategy, then the priority of the JPEG codec task calling the JPEG codec hardware resources is determined to be hardware codec, CPU, and GPU; if the scheduling strategy of the JPEG codec task is a high throughput scheduling strategy, then the priority of the JPEG codec task calling the JPEG codec hardware resources is determined to be GPU, CPU, and hardware codec.

[0016] Furthermore, in step S104, using the determined priority to call JPEG codec hardware resources to execute the JPEG codec task includes: decomposing the JPEG codec task into batch tasks, and obtaining the utilization rates of all JPEG codec hardware resources in the system for each batch task; determining whether the utilization rates of all JPEG codec hardware resources exceed the corresponding preset thresholds, and if they do not exceed the corresponding preset thresholds, directly scheduling the execution of this batch of tasks according to the priority of the JPEG codec task to call the JPEG codec hardware resources; otherwise, taking this batch of tasks as the next batch of tasks, and delaying for a preset period or waiting until the utilization rates of all JPEG codec hardware resources do not exceed the corresponding preset thresholds before using the determined priority to call the JPEG codec hardware resources to execute the JPEG codec task.

[0017] Further, when the determined priority is used to call the JPEG codec hardware resource to perform the JPEG codec task in step S104, it includes: firstly reading the picture of the JPEG codec task into the shared memory, and then calling the JPEG codec hardware resource to perform the JPEG codec task using the determined priority of the picture in the shared memory, specifically including:

[0018] S201, initializing a set of encoding and decoding hardware resources of JPEG encoding and decoding hardware resources and sorting them from high to low priority to create a high real-time picture data queue;

[0019] S202, putting the high real-time image data in the shared memory into a high real-time image data queue;

[0020] S203, obtaining the utilization rate of the hardware codec, and if the utilization rate of the hardware codec does not exceed the corresponding threshold, scheduling the high real-time image data queue to the hardware codec for encoding and decoding;

[0021] If the utilization of the hardware codec exceeds the set threshold, check the CPU utilization. If the CPU utilization does not exceed the corresponding threshold, schedule the high-real-time image data queue to the CPU to use SIMD instructions for encoding and decoding;

[0022] If the CPU utilization exceeds the corresponding threshold, check the GPU utilization. If the GPU utilization does not exceed the corresponding threshold, schedule the high-real-time image data queue to the GPU for encoding and decoding;

[0023] If the hardware codec utilization, CPU utilization, and GPU utilization all exceed the threshold, it will wait in the queue until the hardware resource utilization meets its requirements;

[0024] S206, when the encoding and decoding of the picture data is completed, the corresponding JPEG encoding and decoding hardware resources are released, waiting for the next JPEG encoding and decoding task, and jumping to step S201.

[0025] Furthermore, step S201 further includes: creating a high-throughput image data queue; step S202 further includes: placing multiple batches of image data in the shared memory into the high-throughput image data queue, and jumping to step S204; and after step S202, the following further includes:

[0026] S204, obtaining the number of GPU hardware cores and calculating the total number of GPU computing threads according to the number of GPU hardware cores, checking the GPU utilization, and calculating the remaining total number of threads according to the total number of GPU computing threads and utilization. If the remaining total number of threads is greater than 0, scheduling the high-throughput image data queue to the GPU for encoding and decoding;

[0027] If the total number of remaining threads is less than or equal to 0, the waiting time is delayed. If the waiting time does not exceed the maximum waiting time, the process of this step is repeated again. If the waiting time exceeds the maximum waiting time, the execution jumps to step S205;

[0028] S205, obtaining CPU utilization, if the CPU utilization does not exceed the corresponding threshold, scheduling the high throughput image data queue to the CPU for encoding and decoding using SIMD instructions;

[0029] If the CPU utilization exceeds the corresponding threshold, check the hardware decoder utilization. If the hardware decoder utilization does not exceed the corresponding threshold, schedule the high-throughput image data queue to the hardware codec for encoding and decoding.

[0030] If both the CPU utilization and the hardware codec utilization exceed the corresponding thresholds, the device waits in the queue until the hardware resource utilization meets the requirements.

[0031] The present invention also proposes a JPEG encoding and decoding system based on heterogeneous resource scheduling, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute any one of the JPEG encoding and decoding methods based on heterogeneous resource scheduling.

[0032] The present invention also proposes a computer-readable storage medium, in which a computer program / instruction is stored. The computer program / instruction is programmed or configured to execute any one of the JPEG encoding and decoding methods based on heterogeneous resource scheduling through a processor.

[0033] The present invention also proposes a computer program product, comprising a computer program / instruction, wherein the computer program / instruction is programmed or configured to execute any one of the JPEG encoding and decoding methods based on heterogeneous resource scheduling through a processor.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] The present invention divides the priorities of all JPEG codec hardware resources under each given scheduling strategy, determines the priority of the JPEG codec hardware resource called by the JPEG codec task according to the scheduling strategy of the JPEG codec task, and then uses the determined priority to call the JPEG codec hardware resource to execute the JPEG codec task. Compared with the method of using only one resource for coding and decoding in the current system, the present invention fully utilizes the hardware resources in the system, and determines the priority of the JPEG codec hardware resource by the scheduling strategy, so that the hardware resources can be more effectively allocated and utilized, and the idle or excessive use of resources can be avoided, thereby optimizing the system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 4 is a system architecture diagram of an embodiment of the present invention.

[0037] Figure 2 This is a diagram of the real-time priority of JPEG encoding and decoding hardware resources.

[0038] Figure 3 Schematic diagram of high throughput priority of JPEG encoding and decoding hardware resources.

[0039] Figure 4 The figure is a flow chart of a method according to an embodiment of the present invention.

[0040] Figure 5 This is a flow chart of heterogeneous resource scheduling of a high throughput strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Before introducing the specific implementation of the present invention, the relevant concepts are first explained:

[0042] GPU: A graphics processing unit (Graphics Processing Unit) is a specialized chip originally designed to accelerate computer graphics and image processing. GPUs are now also often used for parallel computing.

[0043] Heterogeneous resources: In the field of computer science and information technology, heterogeneous resources refer to computing resources with different characteristics, types or architectures. These resources can be hardware resources, such as different types of processors (CPU, GPU, hardware codec, etc.).

[0044] SIMD: Abbreviation for Single Instruction Multiple Data, which is a computer processing technology that allows a single instruction to perform an operation on multiple data elements simultaneously.

[0045] CUDA: CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA. It allows software developers and researchers to use NVIDIA GPUs (graphics processing units) for general computing, not just for graphics rendering.

[0046] OpenCL: OpenCL (Open Computing Language) is a framework for writing programs for heterogeneous platforms. This heterogeneous platform can be composed of CPU, GPU, DSP, FPGA or other types of processors and hardware accelerators. It provides a parallel computing mechanism based on task partitioning and data partitioning.

[0047] JPEG: JPEG (Joint Photographic Experts Group) is a widely used lossy compression standard method for image processing. The JPEG format is the most commonly used image file format with the suffix .jpg or .jpeg.

[0048] Embodiment 1

[0049] With the advent of the AI ​​era, image data often needs to be decoded from JPEG to RGB format, whether in training or inference. Therefore, in AI systems, the problem of how to fully utilize the computing power of various heterogeneous resources is becoming more and more prominent. Therefore, this embodiment proposes a JPEG encoding and decoding method based on heterogeneous resource scheduling to solve the utilization of computing power.

[0050] The method of this embodiment is applied to a JPEG codec heterogeneous system, which is composed of hardware resources and JPEG codec runtime. When an application on the system needs to encode and decode an image, it submits a task to the JPEG codec runtime. The JPEG codec runtime will dynamically combine the image data of the submitted tasks into batches according to the use of heterogeneous resources, and then schedule its tasks to run on different hardware resources according to the scheduling strategy, making full use of the system's hardware resources and significantly improving the throughput of JPEG codec.

[0051] like Figure 1 As shown, in this embodiment, the hardware resources of the JPEG codec heterogeneous system include GPU, hardware codec heterogeneous resources and CPU resources. GPU uses CUDA or OPENCL to perform a large amount of parallel computing, which is very suitable for JPEG codec computing tasks. Hardware codec is suitable for JPEG fast coding and decoding, and its advantages are fast coding and decoding speed and no consumption of other system hardware resources. In addition, the SIMD acceleration instructions of the CPU can also be used to encode and decode JPEG images. In summary, the system includes many hardware resources that can encode and decode JPEG.

[0052] In this embodiment, the JPEG codec runtime of the JPEG codec heterogeneous system reasonably schedules and distributes the codec tasks in the application, so as to fully utilize the hardware resources. Figure 1 As shown, its main modules are as follows:

[0053] The heterogeneous resource monitoring module is used to detect heterogeneous resource information and monitor hardware resource utilization, specifically to detect whether the system has heterogeneous resources and CPU acceleration instruction resources for accelerating JPEG encoding and decoding. Secondly, it monitors GPU utilization, hardware codec utilization, and CPU utilization.

[0054] The heterogeneous resource monitoring module first detects the CPU's SIMD instruction information, the GPU's computing power and memory information, and checks whether it has hardware codecs and supports JPEG codecs. Different hardware resources are prioritized based on real-time and high throughput. Figure 2 As shown in Figure 1, hardware codec > CPU SIMD acceleration instructions > GPU parallel computing. The high throughput priority is divided as follows: Figure 3 As shown in the figure, GPU parallel computing > CPU SIMD acceleration instructions > hardware codec. Secondly, heterogeneous resources monitor CPU utilization, GPU utilization, and hardware codec utilization in real time.

[0055] The shared memory management module is used to manage the CPU shared memory. The JPEG codec runtime creates and manages the shared memory. The JPEG image data is read in the application and stored in the shared memory, and then passed from the shared memory to the GPU and JPEG hardware codec. This avoids copying large amounts of data from the application to the JPEG codec runtime, and then from the JPEG codec runtime to the hardware resources.

[0056] The shared memory module mainly enables the upper-layer application to share a piece of memory with the JPEG codec runtime, reads the image into the shared memory, and then sends the image data in the shared memory to various hardware resources for encoding and decoding, avoiding repeated copying of a large amount of image data in the application and the JPEG codec runtime.

[0057] The image dynamic batch processing module is used to dynamically process image data in batches. The GPU and CPU hardware can perform parallel calculations at the same time and can perform encoding and decoding calculations on a batch of image data at the same time. The image batch processor dynamically forms batches of image data based on the usage of hardware resources.

[0058] The dynamic batch processing module dynamically divides the images into different batches, and then transmits the image data to the hardware resources for encoding and decoding; more specifically, the dynamic batch processing module first obtains the utilization rate of each hardware resource from the heterogeneous resource monitoring module. If the utilization rate of all hardware resources exceeds the threshold, the image data is placed in the next batch of data. If the utilization rate of the hardware resources does not exceed the threshold, the hardware resources with high throughput priority are given priority according to the throughput priority. After determining the hardware resources, the dynamic batch processing module will determine whether the batch data of the current hardware resources has sufficient capacity. If there is capacity, the current batch data will be placed, otherwise, the next batch data will be placed. Finally, the batch data is placed in the waiting queue to wait for the hardware resources to encode and decode.

[0059] The heterogeneous resource scheduling module is responsible for scheduling and distributing tasks. It can schedule the encoding and decoding tasks submitted by the application with different strategies to make full use of hardware resources.

[0060] The heterogeneous resource scheduling module can schedule the application's encoding and decoding tasks to run on various hardware resources according to different scheduling strategies; to be more specific, first, the upper-layer application sets the JPEG encoding and decoding priority. If the application requires low time consumption and high real-time performance for JPEG encoding and decoding, the task is set to a high-real-time scheduling strategy. If the application requires high throughput, the task is set to a high-throughput scheduling strategy. In the high-real-time strategy, the JPEG encoding and decoding tasks will be submitted to the hardware codec, CPU, and GPU first. In the high-throughput strategy, the JPEG encoding and decoding tasks will be submitted to the GPU, CPU, and hardware codec first.

[0061] like Figure 4 As shown, the method of this embodiment includes the following steps:

[0062] S101, obtaining all JPEG encoding and decoding hardware resources in the system;

[0063] S102, implementing priority division of all JPEG encoding and decoding hardware resources under each given scheduling strategy;

[0064] S103, obtaining a JPEG codec task, and determining the priority of the JPEG codec task in calling the JPEG codec hardware resources according to the scheduling strategy of the JPEG codec task;

[0065] S104: Use the determined priority to call the JPEG encoding and decoding hardware resources to execute the JPEG encoding and decoding task.

[0066] According to the foregoing content, in step S101 of this embodiment, the JPEG codec hardware resources are specifically detected by the heterogeneous resource monitoring module, and the JPEG codec hardware resources include part or all of the CPU, GPU and hardware codec. When the CPU performs the JPEG codec task, it calls SIMD instructions to perform JPEG codec, when the GPU performs the JPEG codec task, it uses CUDA or OPENCL to perform multi-channel parallel JPEG codec, and when the hardware codec performs the JPEG codec task, it uses its own hardware codec to perform JPEG codec.

[0067] According to the foregoing content, the scheduling strategies in step S102 of this embodiment include two types: a high real-time scheduling strategy and a high throughput scheduling strategy. Under the high real-time scheduling strategy, the priority ranking of the CPU, GPU and hardware codec is: hardware codec>CPU>GPU; under the high throughput scheduling strategy, the priority ranking of the CPU, GPU and hardware codec is: GPU>CPU>hardware codec.

[0068] In step S103 of this embodiment, when determining the priority of the JPEG codec task calling the JPEG codec hardware resources according to the scheduling strategy of the JPEG codec task, if the scheduling strategy of the JPEG codec task is a high real-time scheduling strategy, the priority of the JPEG codec task calling the JPEG codec hardware resources is determined to be hardware codec, CPU, and GPU; if the scheduling strategy of the JPEG codec task is a high throughput scheduling strategy, the priority of the JPEG codec task calling the JPEG codec hardware resources is determined to be GPU, CPU, and hardware codec.

[0069] According to the foregoing content, in step S104 of this embodiment, using the determined priority to call the JPEG codec hardware resources to execute the JPEG codec task includes: the dynamic batch processing module decomposes the JPEG codec task into batch tasks, and for each batch task, obtains the utilization rate of all the JPEG codec hardware resources in the system from the heterogeneous resource monitoring module; determines whether the utilization rate of all the JPEG codec hardware resources exceeds the corresponding preset threshold value, if it does not exceed the corresponding preset threshold value, the heterogeneous resource scheduling module directly schedules the execution of this batch of tasks according to the priority of the JPEG codec task to call the JPEG codec hardware resources, otherwise, the current batch of tasks is used as the next batch of tasks, and after a preset period of delay or waiting until the utilization rate of all the JPEG codec hardware resources does not exceed the corresponding preset threshold value, the heterogeneous resource scheduling module uses the determined priority to call the JPEG codec hardware resources to execute the JPEG codec task.

[0070] Specifically, when the JPEG codec hardware resource is called to perform the JPEG codec task using the determined priority in step S104 of this embodiment, the processing steps for encoding and decoding the picture of the JPEG codec task include: first, the picture of the JPEG codec task is read into the shared memory provided by the shared memory module, and then the picture in the shared memory is called to perform the JPEG codec task using the determined priority by the heterogeneous resource scheduling module and the dynamic batch processing module, including the following steps:

[0071] S201, initializing the heterogeneous resource monitoring module, thereby initializing the codec hardware resource set of the JPEG codec hardware resources, and sorting them from high to low priority to create a high real-time image data queue and a high throughput image data queue;

[0072] S202, the image data in the shared memory is read into different data queues according to the scheduling strategy set by the application, wherein the high real-time image data is placed in the high real-time image data queue, and the process jumps to step S203 to perform high real-time scheduling, while the multi-batch image data is placed in the high throughput image data queue, and the process jumps to step S204 to perform high throughput scheduling;

[0073] S203, the dynamic batch processing module obtains the hardware codec utilization from the heterogeneous resource monitoring module. If the hardware codec utilization does not exceed the set threshold, the heterogeneous resource scheduling module schedules the picture data in the high real-time picture data queue to the hardware codec for encoding and decoding;

[0074] If the hardware codec utilization exceeds the set threshold, the dynamic batch processing module obtains and checks the CPU utilization from the heterogeneous resource monitoring module again. Similarly, if the CPU utilization does not exceed the set threshold, the heterogeneous resource scheduling module schedules the image data to the CPU for encoding and decoding using SIMD instructions;

[0075] If the CPU utilization exceeds the set threshold, the dynamic batch processing module obtains and checks the GPU utilization from the heterogeneous resource monitoring module again. Similarly, if the GPU utilization does not exceed the threshold, the heterogeneous resource scheduling module schedules the image data to the GPU for encoding and decoding;

[0076] If the hardware codec utilization, CPU utilization, and GPU utilization all exceed the threshold, the image data waits in the queue until at least one of the hardware resources utilization of the hardware codec, CPU, and GPU meets the requirement;

[0077] After the encoding and decoding of the picture data in the high real-time picture data queue are completed, jump to step S206;

[0078] S204, such as Figure 5 As shown in the figure, the dynamic batch processing module obtains the number of GPU hardware cores from the heterogeneous resource monitoring module to calculate the total number of GPU computing threads, and then obtains the GPU utilization from the heterogeneous resource monitoring module. The remaining total number of threads is calculated based on the total number of GPU computing threads and utilization. The calculation process is as follows:

[0079] The total number of GPU computing threads g_total_threads is equal to the maximum number of SM (Streaming Multiprocessor) threads * the number of SM cores * FACTOR (set according to the actual value).

[0080] Total number of remaining threads = total number of threads - total number of remaining threads * utilization

[0081] If the total number of remaining threads is greater than 0, the heterogeneous resource scheduling module schedules the batch data in the high-throughput image data queue to the GPU for encoding and decoding.

[0082] If the total number of remaining threads is less than or equal to 0, the waiting time is delayed. If the waiting time does not exceed the maximum waiting time, the above process is repeated again;

[0083] If the waiting time exceeds the maximum waiting time, the high throughput second step scheduling S205 is entered;

[0084] S205, the dynamic batch processing module obtains the CPU utilization from the heterogeneous resource monitoring module. If the CPU utilization does not exceed the set threshold, the heterogeneous resource scheduling module schedules the batch data to the CPU for encoding and decoding using SIMD instructions;

[0085] If the CPU utilization exceeds the set threshold, the dynamic batch processing module will obtain and check the hardware decoder utilization from the heterogeneous resource monitoring module again. Similarly, if the hardware decoder utilization does not exceed the set threshold, the heterogeneous resource scheduling module will schedule the high-throughput image data queue image data to the hardware codec for encoding and decoding in sequence;

[0086] If both the CPU utilization and the hardware decoder utilization exceed the threshold, the batch of image data waits in the queue until the utilization of at least one of the hardware resources among the hardware codec, CPU, and GPU meets the requirement;

[0087] After the batch image data in the high-throughput image data queue are encoded and decoded, jump to step S206;

[0088] S206, when the encoding and decoding tasks in all applications are completed, the resources are released, and the system waits for the next JPEG encoding and decoding task in the system, and jumps to step S201 to schedule a new round of JPEG encoding and decoding tasks.

[0089] Embodiment 2

[0090] This embodiment proposes a JPEG encoding and decoding system based on heterogeneous resource scheduling, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the JPEG encoding and decoding method based on heterogeneous resource scheduling described in the first embodiment.

[0091] This embodiment further proposes a computer-readable storage medium, in which a computer program / instruction is stored. The computer program / instruction is programmed or configured to execute the JPEG encoding and decoding method based on heterogeneous resource scheduling described in Embodiment 1 through a processor.

[0092] This embodiment further proposes a computer program product, including a computer program / instruction, which is programmed or configured to execute the JPEG encoding and decoding method based on heterogeneous resource scheduling described in the first embodiment through a processor.

[0093] In summary, the present invention discloses a JPEG encoding and decoding method based on heterogeneous resource scheduling and a JPEG encoding and decoding heterogeneous system, wherein the heterogeneous system is composed of hardware resources and a JPEG encoding and decoding runtime. The method of the present invention enables a task to be submitted to the JPEG encoding and decoding runtime when an application on the system needs to encode and decode an image. The JPEG encoding and decoding runtime will dynamically combine and batch the image data of the submitted tasks according to the use of heterogeneous resources, and then schedule the tasks to run on different hardware resources according to the scheduling strategy, making full use of the system's hardware resources, significantly improving the throughput of the JPEG encoding and decoding, and having the following advantages:

[0094] (1) Originality of the implementation method. The present invention proposes a JPEG encoding and decoding method and system based on heterogeneous resource scheduling. The JPEG encoding and decoding heterogeneous system consists of hardware resources and JPEG encoding runtime. The JPEG encoding and decoding tasks in the application no longer use a single hardware resource in the system. Instead, the JPEG encoding and decoding runtime performs reasonable scheduling to the hardware for encoding and decoding according to the characteristics of the hardware resources.

[0095] (2) The implementation effect is obvious. The dynamic batch processing of images during JPEG codec runtime and the scheduling through real-time and high-throughput strategies can fully utilize the CPU, GPU and hardware codec resources on the system. The JPEG codec runtime meets the application's requirements for real-time and high-throughput, and significantly improves the speed and throughput of JPEG codec.

[0096] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0097] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A JPEG encoding and decoding method based on heterogeneous resource scheduling, characterized in that: The steps include: S101, obtaining all JPEG encoding and decoding hardware resources in the system; S102, implementing priority division of all JPEG encoding and decoding hardware resources under each given scheduling strategy; S103, obtaining a JPEG codec task, and determining the priority of the JPEG codec task in calling the JPEG codec hardware resources according to the scheduling strategy of the JPEG codec task; S104, using the determined priority to call the JPEG codec hardware resources to perform the JPEG codec task, including: decomposing the JPEG codec task into batch tasks, specifically obtaining the utilization rate of each JPEG codec hardware resource, if the utilization rates of all JPEG codec hardware resources exceed the corresponding preset threshold, then putting the picture data into the next batch of data, if none of them exceed the corresponding preset threshold, then judging whether the batch data of the JPEG codec hardware resources has sufficient capacity, if yes, then putting the picture data into the current batch of data, otherwise, putting the picture data into the next batch of data; For each batch of tasks, the utilization rates of all JPEG codec hardware resources in the system are obtained respectively; it is determined whether the utilization rates of all JPEG codec hardware resources exceed the corresponding preset threshold values; if they do not exceed the corresponding preset threshold values, the tasks in this batch are directly executed according to the priority scheduling of the JPEG codec tasks to call the JPEG codec hardware resources; otherwise, the tasks in this batch are taken as the next batch of tasks, and the JPEG codec hardware resources are called with the determined priority to execute the JPEG codec tasks after a preset period of delay or when the utilization rates of all JPEG codec hardware resources do not exceed the corresponding preset threshold values.

2. The JPEG encoding and decoding method based on heterogeneous resource scheduling according to claim 1, characterized in that: The JPEG codec hardware resources in step S101 include part or all of the CPU, GPU, and hardware codec.

3. The JPEG encoding and decoding method based on heterogeneous resource scheduling according to claim 2, characterized in that: When the CPU executes the JPEG encoding and decoding task, it calls SIMD instructions to perform JPEG encoding and decoding. When the GPU executes the JPEG encoding and decoding task, it uses CUDA or OPENCL to perform multi-channel parallel JPEG encoding and decoding. When the hardware codec executes the JPEG encoding and decoding task, it uses its own hardware codec to perform JPEG encoding and decoding.

4. The JPEG encoding and decoding method based on heterogeneous resource scheduling according to claim 3, characterized in that: The scheduling strategies in step S102 include a high real-time scheduling strategy and a high throughput scheduling strategy. Under the high real-time scheduling strategy, the priority order of the CPU, GPU and hardware codec is: hardware codec>CPU>GPU; under the high throughput scheduling strategy, the priority order of the CPU, GPU and hardware codec is: GPU>CPU>hardware codec.

5. The JPEG encoding and decoding method based on heterogeneous resource scheduling according to claim 4 is characterized in that: In step S103, when determining the priority of the JPEG codec task calling the JPEG codec hardware resources according to the scheduling strategy of the JPEG codec task, if the scheduling strategy of the JPEG codec task is a high real-time scheduling strategy, the priority of the JPEG codec task calling the JPEG codec hardware resources is determined to be the hardware codec, CPU, and GPU; if the scheduling strategy of the JPEG codec task is a high throughput scheduling strategy, the priority of the JPEG codec task calling the JPEG codec hardware resources is determined to be the GPU, CPU, and hardware codec.

6. The JPEG encoding and decoding method based on heterogeneous resource scheduling according to claim 1, characterized in that: When the determined priority is used to call the JPEG codec hardware resource to perform the JPEG codec task in step S104, it includes: firstly reading the picture of the JPEG codec task into the shared memory, and then calling the JPEG codec hardware resource to perform the JPEG codec task using the determined priority of the picture in the shared memory, specifically including: S201, initializing a set of encoding and decoding hardware resources of JPEG encoding and decoding hardware resources and sorting them from high to low priority to create a high real-time picture data queue; S202, putting the high real-time image data in the shared memory into a high real-time image data queue; S203, obtaining the utilization rate of the hardware codec, and if the utilization rate of the hardware codec does not exceed the corresponding threshold, scheduling the high real-time image data queue to the hardware codec for encoding and decoding; If the utilization of the hardware codec exceeds the set threshold, check the CPU utilization. If the CPU utilization does not exceed the corresponding threshold, schedule the high-real-time image data queue to the CPU to use SIMD instructions for encoding and decoding; If the CPU utilization exceeds the corresponding threshold, check the GPU utilization. If the GPU utilization does not exceed the corresponding threshold, schedule the high-real-time image data queue to the GPU for encoding and decoding; If the hardware codec utilization, CPU utilization, and GPU utilization all exceed the threshold, it will wait in the queue until the hardware resource utilization meets its requirements; S206, when the encoding and decoding of the picture data is completed, the corresponding JPEG encoding and decoding hardware resources are released, waiting for the next JPEG encoding and decoding task, and jumping to step S201.

7. The JPEG encoding and decoding method based on heterogeneous resource scheduling according to claim 6, characterized in that: Step S201 further includes: creating a high-throughput image data queue; step S202 further includes: placing multiple batches of image data in the shared memory into the high-throughput image data queue, and jumping to step S204; after step S202, the following also includes: S204, obtaining the number of GPU hardware cores and calculating the total number of GPU computing threads according to the number of GPU hardware cores, checking the GPU utilization, and calculating the remaining total number of threads according to the total number of GPU computing threads and utilization. If the remaining total number of threads is greater than 0, scheduling the high-throughput image data queue to the GPU for encoding and decoding; If the total number of remaining threads is less than or equal to 0, the waiting time is delayed. If the waiting time does not exceed the maximum waiting time, the process of this step is repeated again. If the waiting time exceeds the maximum waiting time, the execution jumps to step S205; S205, obtaining CPU utilization, if the CPU utilization does not exceed the corresponding threshold, scheduling the high throughput image data queue to the CPU for encoding and decoding using SIMD instructions; If the CPU utilization exceeds the corresponding threshold, check the hardware decoder utilization. If the hardware decoder utilization does not exceed the corresponding threshold, schedule the high-throughput image data queue to the hardware codec for encoding and decoding. If both the CPU utilization and the hardware codec utilization exceed the corresponding thresholds, the device waits in the queue until the hardware resource utilization meets the requirements.

8. A JPEG encoding and decoding system based on heterogeneous resource scheduling, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the JPEG encoding and decoding method based on heterogeneous resource scheduling as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored therein, characterized in that: The computer program / instruction is programmed or configured to execute the JPEG encoding and decoding method based on heterogeneous resource scheduling described in any one of claims 1 to 7 through a processor.

Images