Video software and hardware decoding scheduling method and system

Abstract

The invention discloses a video software and hardware decoding scheduling method and system. The method comprises the following steps: acquiring a video file according to a received video playing instruction; acquiring data information of the video file and hardware information of current playing equipment; according to the data information and the hardware information, judging whether a hardware decoder of the current playing equipment supports to decode the video file, and if not, selecting a software decoder to decode the video file; if yes, whether the hardware decoder meeting the performance redundancy requirement exists in the current playing device or not is judged according to the performance use information of the hardware decoder, the decoding consumption estimated value of the video file and the performance redundancy value of the hardware decoder, if yes, the hardware decoder meeting the performance redundancy requirement is selected to decode the video file, and if not, the hardware decoder meeting the performance redundancy requirement is selected to decode the video file. And if so, selecting a software decoder to decode the video file. The technical problem of overlarge load of a single CPU or GPU caused by the fact that the CPU and GPU resources of a computer cannot be fully utilized for video decoding in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of video decoding technology, and in particular to a video software and hardware decoding scheduling method and system. Background Art

[0002] Video decoding can be done in two ways: hardware decoding and software decoding. Hardware decoding refers to the process of decoding compressed audio or video data using dedicated hardware circuits rather than software. Hardware decoding uses graphics card hardware to perform video decoding, which mainly occupies the performance of GPU (Graphics Processing Unit, also known as graphics card). Software decoding refers to the process of decoding compressed audio or video data using software programs rather than dedicated hardware. In software decoding, the CPU executes the decoding algorithm to convert the compressed media data into the original audio or video stream for further processing. In most cases, hardware decoding has better performance than software decoding, with a higher decoding speed in the same time. However, due to the different internal circuit structures of the hardware, it only supports the decoding of videos in some video formats and resolutions. Software decoding uses software algorithms to perform video decoding on the CPU, which has better compatibility than hardware decoding, but usually has a lower decoding speed.

[0003] The existing technical solutions lack a scheduling process in the software and hardware decoding methods. Generally, software decoding or hardware decoding is manually set, or a certain decoding method is selected by default, resulting in all computing resources required for decoding being concentrated on the CPU or a GPU. In addition, hardware decoding on a multi-GPU computer will only use one of the GPUs, which can easily lead to excessive load on a single CPU or GPU and fail to fully utilize the computer's hardware performance. Summary of the invention

[0004] The present application provides a video software and hardware decoding scheduling method and system for improving the technical problem that the prior art cannot fully utilize the CPU and GPU resources of a computer for video decoding, resulting in excessive load on a single CPU or GPU, affecting the video playback effect.

[0005] In view of this, the first aspect of the present application provides a video software and hardware decoding scheduling method, including:

[0006] S1. Receive a video playback instruction and obtain a video file corresponding to the video playback instruction;

[0007] S2, obtaining data information of the video file and hardware information of the current playback device;

[0008] S3, judging whether the hardware decoder of the current playback device supports decoding the video file according to the data information and the hardware information, if so, executing step S4, if not, selecting a software decoder to decode the video file;

[0009] S4. Determine whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder. If so, select the hardware decoder that meets the performance redundancy requirements to decode the video file. If not, select a software decoder to decode the video file.

[0010] Optionally, the data information includes video encoding format and resolution; the hardware information includes hardware decoder model, decoding format and resolution supported by the hardware decoder.

[0011] Optionally, judging whether a hardware decoder of a current playback device supports decoding of the video file according to the data information and the hardware information includes:

[0012] Compare the video encoding format and resolution of the video file with the decoding format and resolution supported by the hardware decoder of the current playback device, and determine whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device;

[0013] If so, determining that the hardware decoder of the current playback device supports decoding the video file;

[0014] If not, it is determined that the hardware decoder of the current playback device does not support decoding of the video file.

[0015] Optionally, before step S4, the method further includes:

[0016] It is determined whether the operating system version of the current playback device meets the preset conditions. If so, step S4 is executed. If not, a software decoder is selected to decode the video file.

[0017] Optionally, before step S4, the method further includes:

[0018] It is determined whether the hardware decoder model of the current playback device belongs to the model in the preset blacklist for matching. If so, a software decoder is selected to decode the video file. If not, step S4 is executed.

[0019] Optionally, judging whether the current playback device has a hardware decoder that meets the performance redundancy requirement according to the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder includes:

[0020] Evaluate the video memory usage or graphics card usage of each model of hardware decoder for decoding the video file according to the resolution of the video file and the model of the hardware decoder, and obtain an estimated video memory usage or graphics card usage of each model of hardware decoder for decoding the video file;

[0021] The estimated performance usage value of each hardware decoder of the current playback device is obtained by summing the video memory usage rate or the graphics card usage rate of each hardware decoder of the current playback device and the estimated video memory usage rate or the graphics card usage rate of the corresponding model of hardware decoder decoding the video file;

[0022] Calculate the performance threshold of each model of hardware decoder according to the performance redundancy value of each model of hardware decoder;

[0023] Determine whether there is a hardware decoder whose estimated performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder;

[0024] If so, it is determined that the current playback device has a hardware decoder that meets the performance redundancy requirements;

[0025] If not, it is determined that the current playback device does not have a hardware decoder that meets the performance redundancy requirements.

[0026] Optionally, the performance threshold of the hardware decoder is a difference between 100% and a performance redundancy value of the hardware decoder;

[0027] The performance redundancy value of the hardware decoder is the sum of the initial performance redundancy value of the hardware decoder and the performance redundancy consumption value of the hardware decoder. The performance redundancy consumption value of the hardware decoder is the performance value consumed by the hardware decoder relative to that before decoding.

[0028] Optionally, the method further includes:

[0029] Score each hardware decoder of the current playback device according to the hardware decoder model of the current playback device and the preset decoder list;

[0030] Determine the scheduling order of each hardware decoder according to the score of each hardware decoder;

[0031] The step of selecting a hardware decoder that meets the performance redundancy requirement to decode the video file comprises:

[0032] When there are multiple hardware decoders that meet the performance redundancy requirement, a hardware decoder is selected from the multiple hardware decoders that meet the performance redundancy requirement to decode the video file in combination with the scheduling order of each hardware decoder.

[0033] Optionally, the method further includes:

[0034] Detect whether the hardware decoder has an exception during the decoding process. If so, reschedule the decoder. If not, record the video memory usage or graphics card usage during the hardware decoder decoding.

[0035] The second aspect of the present application provides a video software and hardware decoding scheduling system, including:

[0036] A receiving unit, used to receive a video playback instruction and obtain a video file corresponding to the video playback instruction;

[0037] An information acquisition unit, used to acquire data information of the video file and hardware information of the current playback device;

[0038] A first judgment unit is used to judge whether the hardware decoder of the current playback device supports decoding the video file according to the data information and the hardware information, and if so, trigger the second judgment unit; if not, select the software decoder to decode the video file;

[0039] The second judgment unit is used to judge whether there is a hardware decoder that meets the performance redundancy requirements in the current playback device based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file and the performance redundancy value of the hardware decoder; if so, the hardware decoder that meets the performance redundancy requirements is selected to decode the video file; if not, the software decoder is selected to decode the video file.

[0040] It can be seen from the above technical solutions that this application has the following advantages:

[0041] The video hardware and software decoding scheduling method provided by the present application determines whether a hardware decoder can be selected by judging whether the hardware decoder of the current playback device supports decoding video files. When the hardware decoder of the current playback device supports decoding video files, it further judges whether the playback device has a hardware decoder with sufficient performance redundancy for video decoding, thereby determining whether to schedule a hardware decoder or a software decoder for video decoding. The present application ensures that the hardware performance of the computer can be fully utilized to meet the needs of more video playback channels by scheduling multiple hardware decoders and combining software decoding and hardware decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1A flowchart of a video software and hardware decoding scheduling method provided in an embodiment of the present application;

[0044] Figure 2 Another flowchart of a video software and hardware decoding scheduling method provided by an embodiment of the present application;

[0045] Figure 3 A structural schematic diagram of a video software and hardware decoding scheduling system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] For easier understanding, please refer to Figure 1 , the embodiment of the present application provides a video software and hardware decoding scheduling method, including:

[0048] Step 110: Receive video playback instructions for each video channel, and obtain the video file corresponding to the video playback instruction.

[0049] When a user initiates multi-channel video playback, multiple video playback instructions will be received. The video playback instructions of each video can be sorted according to the reception time of the video playback instructions of each video to generate a video list, and then the software and hardware decoding of each video can be scheduled in sequence according to the order of the video list; when the video playback instructions of each video are received at the same time, the video playback instructions can be randomly selected and sorted to generate a video list, and then the software and hardware decoding of each video can be scheduled in sequence according to the order in the video list.

[0050] After receiving the video playback instruction, locate the video file specified by the URL (Uniform Resource Locator) carried by the video playback instruction to obtain the video file corresponding to the video playback instruction for subsequent decoder decoding operations. Among them, URL is a standard format for identifying the location of resources on the Internet, usually used to enter in the browser address bar to access web pages. The URL contains information such as the protocol (such as http or https), server domain name / IP address, port number (optional), path and query string, which is used to uniquely identify a resource on the network.

[0051] Step 120: Obtain data information of the video file and hardware information of the current playback device.

[0052] The data information of video files can be obtained from the digital retina platform, including video encoding format, resolution, configuration parameters and other information. The digital retina platform refers to an advanced platform built based on the digital retina system technology. It simulates the function of the human eye retina and can quickly identify and extract key information from massive video data. The digital retina platform combines deep learning, computer vision and big data analysis technologies, and can analyze video streams in real time, perform target detection, behavior analysis and abnormal event warning, and provide algorithm services, video services, storage services and other functions for upper-level applications. It can be widely used in smart cities, security monitoring, traffic management and retail analysis. The digital retina system is a new generation of visual big data perception analysis and processing system that integrates audio and video monitoring, communication, computer network, system integration and other technologies, with real-time feature aggregation, video on-demand retrieval, flexible function definition, and online model update as the application mode. It has the functions of information collection, transmission, control, storage, processing, display, etc., and has the characteristics of unified spatiotemporal information, multi-level data representation, multiple data streams in parallel, and scalable end-edge-cloud collaborative computing architecture.

[0053] The hardware information of the current playback device (such as a computer) can be queried through the operating system interface. The hardware information includes the hardware decoder model, video memory, decoding formats supported by the hardware decoder, and resolution. The decoding format support of the hardware decoder can be determined by parsing the hardware GUID information. After obtaining the hardware information of the current playback device, the hardware information can be saved in the cache module. When a new video is played later, the cache information can be directly read to obtain the hardware information to save time.

[0054] This application directly obtains relevant information of video files based on the digital retina platform. Compared with the traditional method that requires decapsulation and decoding to obtain information, it eliminates the steps of streaming and decoding, and can greatly reduce time. By reading the hardware GUID information through the operating system interface, it can accurately identify whether the hardware level supports the specified format and resolution video files.

[0055] Step 130: determine whether the hardware decoder of the current playback device supports decoding the video file based on the data information and the hardware information. If so, execute step 140; if not, select a software decoder to decode the video file.

[0056] The video encoding format and resolution of the video file are compared with the decoding format and resolution supported by the hardware decoder of the current playback device to determine whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device; if so, it is determined that the hardware decoder of the current playback device supports decoding the video file, and step 140 is executed; if not, it is determined that the hardware decoder of the current playback device does not support decoding the video file, and the software decoder is directly selected to decode the video file, that is, the video file is software decoded.

[0057] Step 140: Determine whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder. If so, select the hardware decoder that meets the performance redundancy requirements to decode the video file. If not, select the software decoder to decode the video file.

[0058] Among them, judging whether the current playback device has a hardware decoder that meets the performance redundancy requirement based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder specifically includes:

[0059] S141, evaluating the video memory usage or graphics card usage of each model of hardware decoder for decoding the video file according to the resolution of the video file and the model of the hardware decoder, and obtaining an estimated value of the video memory usage or graphics card usage of each model of hardware decoder for decoding the video file;

[0060] According to the resolution of the video file and the hardware decoder model, the decoding performance usage record information of the same type of hardware decoders decoding video files of the same resolution and size is obtained to evaluate the video memory usage or graphics card usage of each model of hardware decoder in decoding the current video file, and obtain the estimated video memory usage or graphics card usage of each model of hardware decoder in decoding the video file.

[0061] S142, summing the video memory usage rate or graphics card usage rate of each hardware decoder of the current playback device and the estimated video memory usage rate or graphics card usage rate of the corresponding model of hardware decoder decoding the video file to obtain the estimated performance usage value of each hardware decoder of the current playback device;

[0062] You can use technologies such as DXVADXVA (DirectX Video Acceleration), DXVA2 (DirectX Video Acceleration Version 2), VDPAU (VideoDecode and Presentation API for Unix), and VAAPI (VideoAcceleration API) to calculate the video memory usage or graphics card usage of each hardware decoder of the current playback device, and sum the video memory usage or graphics card usage of each hardware decoder of the current playback device with an estimated video memory usage or graphics card usage of a video file decoded by a hardware decoder of a corresponding model to obtain an estimated performance usage value of each hardware decoder of the current playback device, that is, estimated performance usage value = video memory usage + estimated video memory usage, or, estimated performance usage value = graphics card usage + estimated graphics card usage.

[0063] S143, calculating the performance threshold of each model of hardware decoder according to the performance redundancy value of the hardware decoder of each model of hardware decoder;

[0064] Among them, the performance threshold of the hardware decoder = 100% - the performance redundancy value of the hardware decoder. The performance redundancy value of the hardware decoder is the initial performance redundancy value of the hardware decoder + the performance redundancy consumption value of the hardware decoder. The initial performance redundancy value of the hardware decoder is determined by the performance redundancy base value (in percentage form) of the hardware decoder and the hardware tuning deviation value (in percentage form). The hardware tuning deviation value can be obtained by looking up the table. The hardware tuning deviation table is a table that corresponds to different hardware tuning deviation values ​​according to the hardware model, system version, and hardware driver version. Combined with the hardware description provided by the manufacturer, the higher the hardware model, the newer the system version and the hardware driver version, the smaller the hardware tuning deviation; the performance redundancy base value of the hardware decoder can be 10%, 30%, and other custom values. The performance redundancy base value of the hardware decoder is determined according to the different playback decoding strategies set by the application. For example, when the playback decoding strategy is playback priority, the performance redundancy base value can be set to 10%. At this time, the performance redundancy value of the hardware decoder is (10% + hardware tuning deviation value). When the playback decoding strategy is global priority, the performance redundancy base value can be set to 30%. At this time, the performance redundancy value of the hardware decoder is (30% + hardware tuning deviation value). The performance redundancy consumption value of the hardware decoder is the performance value that the hardware decoder has consumed relative to the performance value before decoding. The performance redundancy consumption value of the hardware decoder is obtained by subtracting the performance consumption base value (which may be 0) from the performance consumption value of the hardware decoder obtained through the hardware driver interface. The performance consumption value (which may be 0) of the hardware decoder before the first video is played is obtained as the performance consumption base value, that is, the performance redundancy consumption value of the hardware decoder = the performance consumption value of the hardware decoder - the performance consumption base value of the hardware decoder.

[0065] When receiving a video playback instruction for N videos, for the first video, the performance threshold of each model of hardware decoder is calculated according to the initial performance redundancy value of each model of hardware decoder, that is, the performance redundancy value of each hardware decoder at this time is the initial performance redundancy value (initial value), and the performance threshold of the hardware decoder at this time = 100% - the initial performance redundancy value of the hardware decoder; wherein N is an integer greater than 0; assuming that the initial performance redundancy value of a hardware decoder is 10%, the performance consumption base value of the hardware decoder before the first video is played is 0, and when the first video is received, the performance threshold of the hardware decoder at this time is 90%;

[0066] For the i-th (i=2,3,...,N) video, the performance threshold of each model of hardware decoder is calculated based on the initial performance redundancy value of each model of hardware decoder and the performance redundancy consumption value of the hardware decoder selected for the first i-1 videos after decoding. That is, in a multi-channel video playback scenario, the performance redundancy value of the hardware decoder that has been selected for decoding is the sum of the initial performance redundancy value and the performance redundancy consumption value. For example, suppose there are two videos, and the playback device has two hardware decoders A and B. The initial performance redundancy values ​​of hardware decoders A and B are both 10%, and the performance consumption base values ​​before the first video is played are both 0. At this time, the performance thresholds of hardware decoders A and B are both 90%. The scheduling result of the first video is to select hardware decoder A for video decoding. When the software and hardware decoders are scheduled for the second video, when calculating the performance threshold of hardware decoder B, since hardware decoder B is not selected to decode the first video, that is, hardware decoder B has no performance consumption, that is, the performance redundancy consumption value of the hardware decoder is 0. At this time, the performance threshold of hardware decoder B remains unchanged and is still 90%. When calculating the performance threshold of hardware decoder A, it is necessary to obtain the performance consumption value of hardware decoder A decoding the first video. Assuming that the performance consumption value of hardware decoder A decoding the first video is 10%, since the performance consumption base value is 0, the performance redundancy consumption value of hardware decoder A is 10%. The performance threshold of hardware decoder A = 100%-10%-10%=80%. For the i-th video, the performance redundancy consumption value of the hardware decoder is the performance value that the hardware decoder has consumed relative to the time when it is not decoded.

[0067] The present application takes into account that the prior art only selects the decoding method from the perspective of single-channel video playback. When facing a multi-channel video playback scenario, a single decoding method is still used for each video, which easily leads to excessive load on a single CPU or GPU, and cannot fully utilize the hardware performance of the computer, affecting the overall playback effect of the multi-channel video. In order to improve this problem, the present application adaptively adjusts the performance redundancy value of the hardware decoder when facing a multi-channel video playback scenario, and then adjusts the performance threshold of the hardware decoder. After the hardware decoder has decoded a certain video, the performance redundancy consumption value of the hardware decoder after decoding is calculated to update the performance redundancy value of the hardware decoder, and then adjusts the performance threshold of the hardware decoder, thereby realizing the performance threshold adjustment of the hardware decoder before and after decoding the video, thereby avoiding the problem of repeated selection of a hardware decoder for video decoding when playing multiple videos, resulting in excessive load on a certain GPU.

[0068] S144. Determine whether there is a hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder; if so, determine that the current playback device has a hardware decoder that meets the performance redundancy requirements; if not, determine that the current playback device does not have a hardware decoder that meets the performance redundancy requirements.

[0069] Determine whether there is a hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the hardware decoder of the corresponding model. If there is a hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the hardware decoder of the corresponding model, then it is determined that the current playback device has a hardware decoder that meets the performance redundancy requirements, and the hardware decoder that meets the performance redundancy requirements is selected to decode the video file; if there is no hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the hardware decoder of the corresponding model, then it is determined that the current playback device does not have a hardware decoder that meets the performance redundancy requirements, and the software decoder is selected to decode the video file.

[0070] After the video file is successfully decoded, the decoded video file is rendered to achieve playback of the video file.

[0071] The video hardware and software decoding scheduling method provided by the present application determines whether a hardware decoder can be selected by judging whether the hardware decoder of the current playback device supports decoding video files. When the hardware decoder of the current playback device supports decoding video files, it further judges whether the playback device has a hardware decoder with sufficient performance redundancy for video decoding, thereby determining whether to schedule a hardware decoder or a software decoder for video decoding. The present application ensures that the hardware performance of the computer can be fully utilized to meet the needs of more video playback channels by scheduling multiple hardware decoders and combining software decoding and hardware decoding.

[0072] The above is an embodiment of a video software and hardware decoding scheduling method provided by the present application. The following is another embodiment of a video software and hardware decoding scheduling method provided by the present application.

[0073] Please refer to Figure 2 , a video software and hardware decoding scheduling method provided by an embodiment of the present application includes:

[0074] Step 210: Receive video playback instructions for each video channel, and obtain the video file corresponding to the video playback instruction.

[0075] When a user initiates multi-channel video playback, multiple video playback instructions will be received. The video playback instructions of each video can be sorted according to the reception time of the video playback instructions of each video to generate a video list, and then the software and hardware decoding of each video can be scheduled in sequence according to the order of the video list; when the video playback instructions of each video are received at the same time, the video playback instructions can be randomly selected and sorted to generate a video list, and then the software and hardware decoding of each video can be scheduled in sequence according to the order in the video list.

[0076] After receiving the video playback instruction, locate the video file specified by the URL (Uniform Resource Locator) carried by the video playback instruction to obtain the video file corresponding to the video playback instruction for subsequent decoder decoding operations. Among them, URL is a standard format for identifying the location of resources on the Internet, usually used to enter in the browser address bar to access web pages. The URL contains information such as the protocol (such as http or https), server domain name / IP address, port number (optional), path and query string, which is used to uniquely identify a resource on the network.

[0077] Step 220: Obtain data information of the video file and hardware information of the current playback device.

[0078] The data information of the video file can be obtained from the digital retina platform. The data information includes video encoding format, resolution, configuration parameters and other information. The hardware information of the current playback device (such as a computer) can be queried through the operating system interface. The hardware information includes the hardware decoder model, video memory, decoding formats supported by the hardware decoder, and resolution. Among them, the decoding format support of the hardware decoder can be determined by parsing the hardware GUID information. After obtaining the hardware information of the current playback device, the hardware information can be saved in the cache module. When a new video is played later, the cache information can be directly read to obtain the hardware information to save time.

[0079] This application directly obtains relevant information of video files based on the digital retina platform. Compared with the traditional method that requires decapsulation and decoding to obtain information, it eliminates the steps of streaming and decoding, and can greatly reduce time. By reading the hardware GUID information through the operating system interface, it can accurately identify whether the hardware level supports the specified format and resolution video files.

[0080] Step 230 , determine whether the hardware decoder model of the current playback device belongs to the model in the preset blacklist. If so, select the software decoder to decode the video file. If not, execute step 240 .

[0081] The hardware decoder model of the current playback device is matched with the models in the preset blacklist. If the hardware decoder model of the current playback device matches a model in the preset blacklist, the software decoder is selected to decode the video file, and the next scheduling judgment process is not entered; if the hardware decoder model of the current playback device does not match any model in the preset blacklist, execute step 240.

[0082] Among them, the preset blacklist can store hardware decoder models with poor decoding effect or high decoding failure rate. This application uses the hardware decoding blacklist mechanism for judgment, which can quickly screen out hardware decoders that do not meet the conditions and reduce the time consumption of subsequent links.

[0083] Step 240, determine whether the operating system version of the current playback device meets the preset conditions, if so, execute step 250, if not, select a software decoder to decode the video file.

[0084] The operating system version of the current playback device can be obtained through the computing set system interface to determine whether the operating system version of the current playback device meets the preset conditions. For example, the Windows 7 system cannot use DXVA2 and D3D11 technologies, and the Linux system cannot use DXVA, DXVA2, D3D11, D3D9 and other technologies, and can only use VDPAU and VAAPI technologies. At this time, the system and driver will be further detected to determine whether the current playback device supports the use conditions that meet the required technology. If it does not meet the preset conditions, the video file to be played is directly decoded using software, and the next scheduling judgment process is not entered. If it meets the preset conditions, step 250 is executed.

[0085] Step 250: determine whether the hardware decoder of the current playback device supports decoding the video file based on the data information and the hardware information. If so, execute step 260; if not, select a software decoder to decode the video file.

[0086] The video encoding format and resolution of the video file are compared with the decoding format and resolution supported by the hardware decoder of the current playback device to determine whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device; if so, it is determined that the hardware decoder of the current playback device supports decoding the video file, and step 260 is executed; if not, it is determined that the hardware decoder of the current playback device does not support decoding the video file, and the software decoder is directly selected to decode the video file, that is, the video file is software decoded.

[0087] It should be noted that the order of step 230, step 240 and step 250 can be swapped.

[0088] Step 260: Determine whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder. If so, select the hardware decoder that meets the performance redundancy requirements to decode the video file. If not, select the software decoder to decode the video file.

[0089] According to the performance usage information of the hardware decoder, the estimated value of the decoding consumption of the video file and the performance redundancy value of the hardware decoder, it is judged whether the current playback device has a hardware decoder that meets the performance redundancy requirements. The specific judgment process can refer to the aforementioned embodiment and will not be repeated here. When the number of decodings of the hardware decoder that meets the performance redundancy requirements is 1, the hardware decoder is directly selected to decode the video file. When the number of decodings of the hardware decoder that meets the performance redundancy requirements is multiple, the hardware decoder that can be selected to decode the video file from the multiple hardware decoders that meet the performance redundancy requirements can be combined with the scheduling order of each hardware decoder to decode the video file, and the hardware decoder with a high scheduling order is preferably selected to decode the video file. For example, the current playback device has two hardware decoders A and B that meet the performance redundancy requirements. When the scheduling order of hardware decoder A is higher than that of hardware decoder B, hardware decoder A is preferably selected to decode the video file; when the scheduling order of hardware decoder A is the same as the scheduling order of hardware decoder B, a hardware decoder can be randomly selected from hardware decoders A and B.

[0090] Furthermore, the hardware decoders of the current playback device can be scored according to the hardware decoder model of the current playback device and the preset decoder list, and the scheduling order of each hardware decoder can be determined according to the score of each hardware decoder. The performance score of each model of hardware decoder can be set in advance according to the decoding performance of each model of hardware decoder to generate a preset decoder list.

[0091] In one embodiment, step 260 further includes:

[0092] Step 270, detect whether an exception occurs in the hardware decoder during the decoding process, if so, re-schedule the decoder, if not, record the video memory usage or graphics card usage occupied by the hardware decoder during decoding.

[0093] When performing hardware decoding on a video file, the operating system interface and the driver interface can be called to record the video memory usage rate or the graphics card usage rate occupied by the video decoding. At the same time, it is also possible to detect in real time whether an abnormality occurs in the hardware decoder during the decoding process, including failure to create a decoder, decoding failure, low decoding frame rate and other abnormalities. If an abnormality is detected, the decoder can be re-scheduled, that is, return to the step corresponding to obtaining the hardware information to select a new hardware decoder. If there is no suitable hardware decoder, the software decoder is used; if no abnormality is detected, that is, decoding is successful, the video memory usage rate or the graphics card usage rate occupied by the hardware decoder during decoding is recorded.

[0094] This application ensures that the computer's hardware performance can be fully utilized to meet the needs of more video playback through the scheduling of multiple hardware decoders and the combination of software decoding and hardware decoding; through statistical estimation and monitoring of the overall utilization rate of the decoder and the utilization rate of video memory, it can effectively ensure that the hardware decoder has sufficient performance redundancy, thereby ensuring the playback effect of multiple videos; this application directly obtains relevant information of video files based on the digital retina platform. Compared with the traditional method, which requires decapsulation and decoding to obtain information, it eliminates the links of streaming and decoding, and can greatly reduce time.

[0095] The following is a specific application example of a video software and hardware decoding scheduling method provided by this application.

[0096] Step 1: The user uses a computer to broadcast multiple videos. The software and hardware decoding scheduling system receives the video playback instruction, locates the video file specified by the URL, and obtains the video file corresponding to the video playback instruction;

[0097] Step 2: The scheduling system obtains the corresponding video file from the digital retina platform in HEVC format with a video resolution of 1920x1080;

[0098] Step 3. Through the operating system interface, it is found that the user's computer has two hardware decoders, namely, Intel Iris Xe Graphics (hereinafter referred to as hardware decoder A) and GeForce RTX 4060 Laptop GPU (hereinafter referred to as hardware decoder B). These two hardware decoders are not on the hardware decoding blacklist of the video with the encoding format and resolution.

[0099] Step 4: Through the operating system interface, it is obtained that the user's computer is Windows 11, and combined with the driver interface information, it is determined that the computer can support DXVA2 and D3D11 technologies and meet the hardware decoding requirements;

[0100] Step 5: Read the GUID information of the hardware decoder through the operating system interface to determine the encoding format and resolution supported by the hardware decoding. It is determined that both hardware decoders support hardware decoding of video files in HEVC format with a resolution of 1920x1080.

[0101] Step 6: The current overall usage rate and video memory occupancy rate of hardware decoder A and hardware decoder B are obtained by calling the system interface and the hardware decoder driver interface. After judging that both hardware decoder A and hardware decoder B meet the performance redundancy requirements, but hardware decoder B has better decoding performance, that is, the scheduling order of hardware decoder B is higher than that of hardware decoder A, so hardware decoder B is preferentially called to decode the video file;

[0102] Step 7: The called hardware decoder B decodes the video file normally and completes the video playback operation;

[0103] Step 8. The user continues to play the next video and repeats steps 1 to 6. During the decoding process, the corresponding graphics card usage growth and video memory growth during the decoding process are counted as reference evaluation conditions for subsequent evaluation. When hardware decoder B detects that the performance redundancy is insufficient, hardware decoder A is used for hardware decoding and step 7 is executed. When both hardware decoder A and hardware decoder B have no redundant performance decoding, the software decoder is used to decode the video file.

[0104] Due to the change in bit rate, there will be a certain degree of usage and memory changes in the hardware decoding process. By recording the increase in the usage rate and memory of the hardware decoder during the video file decoding process, it can effectively ensure that the hardware decoder has sufficient performance redundancy to ensure the playback effect of multiple videos.

[0105] Please refer to Figure 2 The embodiment of the present application also provides a video software and hardware decoding scheduling system, including:

[0106] A receiving unit, used to receive a video playback instruction and obtain a video file corresponding to the video playback instruction;

[0107] An information acquisition unit, used to acquire data information of the video file and hardware information of the current playback device;

[0108] The first judgment unit is used to judge whether the hardware decoder of the current playback device supports decoding the video file according to the data information and the hardware information, and if so, trigger the second judgment unit; if not, select the software decoder to decode the video file;

[0109] The second judgment unit is used to judge whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file and the performance redundancy value of the hardware decoder. If so, the hardware decoder that meets the performance redundancy requirements is selected to decode the video file; if not, the software decoder is selected to decode the video file.

[0110] As a further improvement, the first judgment unit is specifically used to:

[0111] Compare the video encoding format and resolution of the video file with the decoding format and resolution supported by the hardware decoder of the current playback device to determine whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device;

[0112] If so, it is determined that the hardware decoder of the current playback device supports decoding the video file, and the second determination unit is triggered;

[0113] If not, it is determined that the hardware decoder of the current playback device does not support decoding the video file, and a software decoder is selected to decode the video file;

[0114] As a further improvement, the device further comprises:

[0115] The third judgment unit is used to judge whether the operating system version of the current playback device meets the preset conditions. If so, the second judgment unit is triggered. If not, a software decoder is selected to decode the video file.

[0116] As a further improvement, the device further comprises:

[0117] The fourth judgment unit is used to judge whether the hardware decoder model of the current playback device belongs to the model in the preset blacklist. If so, the software decoder is selected to decode the video file. If not, the second judgment unit is triggered.

[0118] As a further improvement, the device further comprises:

[0119] The scheduling order generating unit is used to score each hardware decoder of the current playback device according to the hardware decoder model of the current playback device and the preset decoder list; and determine the scheduling order of each hardware decoder according to the score of each hardware decoder.

[0120] As a further improvement, the device further comprises:

[0121] The detection unit is used to detect whether an exception occurs in the decoding process of the hardware decoder. If so, the decoder is rescheduled. If not, the video memory usage or graphics card usage of the hardware decoder during decoding is recorded.

[0122] This application ensures that the computer's hardware performance can be fully utilized to meet the needs of more video playback through the scheduling of multiple hardware decoders and the combination of software decoding and hardware decoding. By statistically estimating and monitoring the overall utilization rate and video memory utilization rate of the decoder, it can effectively ensure that the hardware decoder has sufficient performance redundancy, thereby ensuring the playback effect of multiple videos.

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0125] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0129] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0130] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video software and hardware decoding scheduling method, characterized in that: include: S1. Receive a video playback instruction and obtain a video file corresponding to the video playback instruction; S2, obtaining data information of the video file and hardware information of the current playback device; S3, judging whether the hardware decoder of the current playback device supports decoding the video file according to the data information and the hardware information, if so, executing step S4, if not, selecting a software decoder to decode the video file; S4. Determine whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder. If so, select the hardware decoder that meets the performance redundancy requirements to decode the video file. If not, select a software decoder to decode the video file.

2. The video software and hardware decoding scheduling method according to claim 1, characterized in that: The data information includes video encoding format and resolution; the hardware information includes hardware decoder model, decoding format and resolution supported by the hardware decoder.

3. The video software and hardware decoding scheduling method according to claim 2, characterized in that: The determining, according to the data information and the hardware information, whether the hardware decoder of the current playback device supports decoding the video file includes: Compare the video encoding format and resolution of the video file with the decoding format and resolution supported by the hardware decoder of the current playback device, and determine whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device; If so, determining that the hardware decoder of the current playback device supports decoding the video file; If not, it is determined that the hardware decoder of the current playback device does not support decoding of the video file.

4. The video software and hardware decoding scheduling method according to claim 1, characterized in that: Before step S4, the method further includes: Determine whether the operating system version of the current playback device meets the preset conditions. If so, execute step S4. If not, select a software decoder to decode the video file.

5. The video software and hardware decoding scheduling method according to claim 1, characterized in that: Before step S4, the method further includes: It is determined whether the hardware decoder model of the current playback device belongs to the model in the preset blacklist. If so, a software decoder is selected to decode the video file. If not, step S4 is executed.

6. The video software and hardware decoding scheduling method according to claim 1, characterized in that: The determining whether the current playback device has a hardware decoder that meets the performance redundancy requirement according to the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder includes: Evaluate the video memory usage or graphics card usage of each model of hardware decoder for decoding the video file according to the resolution of the video file and the model of the hardware decoder, and obtain an estimated video memory usage or graphics card usage of each model of hardware decoder for decoding the video file; The estimated performance usage value of each hardware decoder of the current playback device is obtained by summing the video memory usage rate or the graphics card usage rate of each hardware decoder of the current playback device and the estimated video memory usage rate or the graphics card usage rate of the corresponding model of hardware decoder decoding the video file; Calculate the performance threshold of each model of hardware decoder according to the performance redundancy value of each model of hardware decoder; Determine whether there is a hardware decoder whose estimated performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder; If so, it is determined that the current playback device has a hardware decoder that meets the performance redundancy requirements; If not, it is determined that the current playback device does not have a hardware decoder that meets the performance redundancy requirements.

7. The video software and hardware decoding scheduling method according to claim 6, characterized in that: The performance threshold of the hardware decoder is the difference between 100% and the performance margin value of the hardware decoder; The performance redundancy value of the hardware decoder is the sum of the initial performance redundancy value of the hardware decoder and the performance redundancy consumption value of the hardware decoder. The performance redundancy consumption value of the hardware decoder is the performance value consumed by the hardware decoder relative to that before decoding.

8. The video software and hardware decoding scheduling method according to claim 6, characterized in that: The method further comprises: Score each hardware decoder of the current playback device according to the hardware decoder model of the current playback device and the preset decoder list; Determine the scheduling order of each hardware decoder according to the score of each hardware decoder; The step of selecting a hardware decoder that meets the performance redundancy requirement to decode the video file comprises: When there are multiple hardware decoders that meet the performance redundancy requirement, a hardware decoder is selected from the multiple hardware decoders that meet the performance redundancy requirement to decode the video file in combination with the scheduling order of each hardware decoder.

9. The video software and hardware decoding scheduling method according to claim 1, characterized in that: The method further comprises: Detect whether the hardware decoder has an exception during the decoding process. If so, reschedule the decoder. If not, record the video memory usage or graphics card usage during the hardware decoder decoding.

10. A video software and hardware decoding scheduling system, characterized in that: include: A receiving unit, used to receive a video playback instruction and obtain a video file corresponding to the video playback instruction; An information acquisition unit, used to acquire data information of the video file and hardware information of the current playback device; A first judgment unit is used to judge whether the hardware decoder of the current playback device supports decoding the video file according to the data information and the hardware information, and if so, trigger the second judgment unit; if not, select the software decoder to decode the video file; The second judgment unit is used to judge whether there is a hardware decoder that meets the performance redundancy requirements in the current playback device based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file and the performance redundancy value of the hardware decoder; if so, the hardware decoder that meets the performance redundancy requirements is selected to decode the video file; if not, the software decoder is selected to decode the video file.

Images