A video hardware and software decoding scheduling method and system

Abstract

The application discloses a video software and hardware decoding scheduling method and system. The method comprises the following steps: obtaining a video file according to a received video playing instruction; obtaining data information of the video file and hardware information of a current playing device; judging whether a hardware decoder of the current playing device supports decoding the video file according to the data information and the hardware information; if not, selecting a software decoder to decode the video file; if yes, judging whether there is a hardware decoder meeting a performance redundancy requirement in the current playing device according to performance using information of the hardware decoder, a decoding consumption estimation value of the video file and a performance redundancy value of the hardware decoder; if yes, selecting the hardware decoder meeting the performance redundancy requirement to decode the video file; and if not, selecting the software decoder to decode the video file. The application improves the technical problem that the prior art cannot fully utilize CPU and GPU resources of a computer for video decoding, resulting in excessive load of a single CPU or GPU.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video decoding, and particularly relates to a video software and hardware decoding scheduling method and system. BACKGROUND

[0002] Video decoding can adopt two ways of hardware decoding and software decoding, hardware decoding refers to a process of decoding compressed audio or video data by using special hardware circuit instead of software, hardware decoding performs video decoding through video card hardware, and mainly occupies GPU (Graphics Processing Unit, graphics processor, also known as a video card) performance. Software decoding refers to a process of decoding compressed audio or video data by using software program instead of special hardware. In software decoding, a CPU executes a decoding algorithm to convert compressed media data into original audio or video stream for further processing. In most cases, hardware decoding has better performance than software decoding, and has higher decoding speed in the same time. However, due to the difference in internal circuit structure of hardware, hardware decoding only supports decoding of videos in some video formats and resolutions. Software decoding performs video decoding on a CPU through a software algorithm, and has better compatibility compared with hardware decoding, but generally has lower decoding speed.

[0003] In the prior art, the software and hardware decoding mode lacks a scheduling process, and generally manually sets software decoding or hardware decoding, or selects a certain decoding mode by default, which causes all computing resources required for decoding to be concentrated in a CPU or a certain GPU, and hardware decoding only uses one GPU in a multi-GPU computer, which easily causes excessive load of a single CPU or GPU and cannot fully utilize the hardware performance of the computer. SUMMARY

[0004] The present application provides a video software and hardware decoding scheduling method and system, which is used to solve the technical problem that the prior art cannot fully utilize the CPU and GPU resources of a computer for video decoding, causes excessive load of a single CPU or GPU, and affects video playing effect.

[0005] Therefore, the present application provides a video software and hardware decoding scheduling method, which comprises the following steps:

[0006] S1, receiving a video playing instruction, and acquiring a video file corresponding to the video playing instruction;

[0007] S2, acquiring data information of the video file and hardware information of a current playing device;

[0008] S3, judging whether the hardware decoder of the current playback device supports decoding the video file based on the data information and the hardware information, and if so, executing step S4; if not, selecting a software decoder to decode the video file;

[0009] S4. Determine whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder. If so, select the hardware decoder that meets the performance redundancy requirements to decode the video file. If not, select the software decoder to decode the video file.

[0010] Optionally, the data information includes video encoding format and resolution; the hardware information includes hardware decoder model, decoding format and resolution supported by the hardware decoder.

[0011] Optionally, determining whether a hardware decoder of a current playback device supports decoding the video file according to the data information and the hardware information includes:

[0012] Comparing the video encoding format and resolution of the video file with the decoding format and resolution supported by the hardware decoder of the current playback device, and determining whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device;

[0013] If so, determining that the hardware decoder of the current playback device supports decoding the video file;

[0014] If not, it is determined that the hardware decoder of the current playback device does not support decoding the video file.

[0015] Optionally, before step S4, the following steps may be included:

[0016] Determine whether the operating system version of the current playback device meets the preset conditions. If so, execute step S4. If not, select a software decoder to decode the video file.

[0017] Optionally, before step S4, the method further includes:

[0018] Determine whether the hardware decoder model of the current playback device belongs to the models in the preset blacklist for matching. If so, select the software decoder to decode the video file. If not, execute step S4.

[0019] Optionally, determining whether the current playback device has a hardware decoder that meets performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder includes:

[0020] Evaluate the video memory usage or graphics card usage of each model of hardware decoder for decoding the video file based on the resolution of the video file and the model of the hardware decoder, and obtain an estimated video memory usage or graphics card usage of each model of hardware decoder for decoding the video file;

[0021] Summing the video memory usage or graphics card usage of each hardware decoder of the current playback device with the estimated video memory usage or graphics card usage of the corresponding hardware decoder for decoding the video file, to obtain an estimated performance usage value of each hardware decoder of the current playback device;

[0022] Calculate the performance threshold of each model of hardware decoder according to the performance redundancy value of each model of hardware decoder;

[0023] Determine whether there is a hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder;

[0024] If so, it is determined that the current playback device has a hardware decoder that meets the performance redundancy requirements;

[0025] If not, it is determined that the current playback device does not have a hardware decoder that meets the performance redundancy requirements.

[0026] Optionally, the performance threshold of the hardware decoder is 100% minus the performance redundancy value of the hardware decoder;

[0027] The performance redundancy value of the hardware decoder is the sum of the initial performance redundancy value of the hardware decoder and the performance redundancy consumption value of the hardware decoder. The performance redundancy consumption value of the hardware decoder is the performance value consumed by the hardware decoder relative to that before decoding.

[0028] Optionally, the method further includes:

[0029] Score each hardware decoder of the current playback device based on the hardware decoder model of the current playback device and the preset decoder list;

[0030] Determine the scheduling order of each hardware decoder according to the score of each hardware decoder;

[0031] The selecting a hardware decoder that meets the performance redundancy requirement to decode the video file includes:

[0032] When there are multiple hardware decoders that meet the performance redundancy requirement, a hardware decoder is selected from the multiple hardware decoders that meet the performance redundancy requirement to decode the video file in combination with the scheduling order of each hardware decoder.

[0033] Optionally, the method further includes:

[0034] Detect whether the hardware decoder has an exception during the decoding process. If so, reschedule the decoder. If not, record the video memory usage or graphics card usage during the hardware decoder decoding.

[0035] A second aspect of the present application provides a video software and hardware decoding scheduling system, including:

[0036] A receiving unit, configured to receive a video playback instruction and obtain a video file corresponding to the video playback instruction;

[0037] An information acquisition unit, configured to acquire data information of the video file and hardware information of the current playback device;

[0038] a first determining unit, configured to determine, based on the data information and the hardware information, whether the hardware decoder of the current playback device supports decoding the video file; if so, triggering the second determining unit; if not, selecting the software decoder to decode the video file;

[0039] The second judgment unit is used to judge whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file and the performance redundancy value of the hardware decoder. If so, the hardware decoder that meets the performance redundancy requirements is selected to decode the video file; if not, the software decoder is selected to decode the video file.

[0040] It can be seen from the above technical solutions that this application has the following advantages:

[0041] The video hardware and software decoding scheduling method provided in this application determines whether a hardware decoder can be selected by judging whether the hardware decoder of the current playback device supports decoding video files. When the hardware decoder of the current playback device supports decoding video files, it further judges whether the playback device has a hardware decoder with sufficient performance redundancy for video decoding, thereby determining whether to schedule a hardware decoder or a software decoder for video decoding. This application ensures that the computer's hardware performance can be fully utilized to meet more video playback needs through the scheduling of multiple hardware decoders and the combination of software decoding and hardware decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1A flowchart of a video software and hardware decoding scheduling method provided in an embodiment of the present application;

[0044] Figure 2 Another flowchart of a video software and hardware decoding scheduling method provided in an embodiment of the present application;

[0045] Figure 3 A structural diagram of a video software and hardware decoding scheduling system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0047] For easier understanding, please refer to Figure 1 , an embodiment of the present application provides a video software and hardware decoding scheduling method, including:

[0048] Step 110: Receive video playback instructions for each video channel, and obtain the video file corresponding to the video playback instruction.

[0049] When a user initiates multi-channel video playback, multiple video playback instructions will be received. The video playback instructions of each video can be sorted according to the reception time of the video playback instructions of each video to generate a video list, and then the software and hardware decoding of each video can be scheduled in sequence according to the order of the video list; when the video playback instructions of each video are received at the same time, the video playback instructions can be randomly selected and sorted to generate a video list, and then the software and hardware decoding of each video can be scheduled in sequence according to the order in the video list.

[0050] After receiving a video playback command, the video file specified by the Uniform Resource Locator (URL) carried in the video playback command is located to obtain the video file corresponding to the video playback command for subsequent decoder operations. A URL is a standard format for identifying resource locations on the internet, commonly entered in a browser address bar to access a web page. A URL contains information such as the protocol (such as http or https), the server domain name / IP address, an optional port number, a path, and a query string, and is used to uniquely identify a resource on the internet.

[0051] Step 120: Obtain data information of the video file and hardware information of the current playback device.

[0052] The Digital Retina Platform can retrieve video file data information, including video encoding format, resolution, and configuration parameters. The Digital Retina Platform is an advanced platform built based on digital retina system technology. It simulates the functions of the human retina and can quickly identify and extract key information from massive amounts of video data. Combining deep learning, computer vision, and big data analytics technologies, the Digital Retina Platform analyzes video streams in real time, performing object detection, behavior analysis, and abnormal event warnings. It also provides algorithm services, video services, and storage services to upper-layer applications, enabling widespread application in smart cities, security monitoring, traffic management, and retail analytics. The Digital Retina System is a next-generation visual big data perception, analysis, and processing system that integrates technologies such as audiovisual surveillance, communications, computer networks, and system integration. Its application model utilizes real-time feature aggregation, on-demand video retrieval, flexible function definition, and online model updates. It features information acquisition, transmission, control, storage, processing, and display capabilities. It features unified spatiotemporal information, multi-level data representation, multiple parallel data streams, and a scalable edge-edge-cloud collaborative computing architecture.

[0053] The operating system interface allows you to query the hardware information of the current playback device (such as a computer). This information includes the hardware decoder model, video memory, supported decoding formats, and resolutions. The hardware decoder's supported decoding formats can be determined by parsing the hardware GUID. After obtaining the hardware information of the current playback device, you can save it in a cache module. When playing a new video, you can directly read the cached information to obtain the hardware information, saving time.

[0054] This application directly obtains relevant information of video files based on the digital retina platform. Compared with the traditional method that requires decapsulation and decoding to obtain information, it eliminates the streaming and decoding steps and can greatly reduce time. By reading the hardware GUID information through the operating system interface, it can accurately identify whether the hardware level supports the specified format and resolution video file.

[0055] Step 130 : Determine whether the hardware decoder of the current playback device supports decoding the video file based on the data information and the hardware information. If so, execute step 140 ; ​​if not, select a software decoder to decode the video file.

[0056] The video encoding format and resolution of the video file are compared with the decoding format and resolution supported by the hardware decoder of the current playback device to determine whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device; if so, it is determined that the hardware decoder of the current playback device supports decoding the video file, and step 140 is executed; if not, it is determined that the hardware decoder of the current playback device does not support decoding the video file, and the software decoder is directly selected to decode the video file, that is, the video file is software-decoded.

[0057] Step 140: Determine whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder. If so, select the hardware decoder that meets the performance redundancy requirements to decode the video file. If not, select the software decoder to decode the video file.

[0058] The method of determining whether the current playback device has a hardware decoder that meets the performance redundancy requirements is based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder. Specifically, the method includes:

[0059] S141. Evaluate the video memory usage or graphics card usage of each model of hardware decoder for decoding the video file based on the resolution of the video file and the model of the hardware decoder, and obtain an estimated video memory usage or graphics card usage of each model of hardware decoder for decoding the video file;

[0060] Based on the resolution of the video file and the hardware decoder model, obtain the decoding performance usage record information of the same type of hardware decoders decoding video files of the same resolution and size, so as to evaluate the video memory usage or graphics card usage of each model of hardware decoder for decoding the current video file, and obtain the estimated video memory usage or graphics card usage of each model of hardware decoder for decoding the video file.

[0061] S142: summing the video memory usage or graphics card usage of each hardware decoder of the current playback device with the estimated video memory usage or graphics card usage of the corresponding hardware decoder model for decoding the video file to obtain an estimated performance usage value of each hardware decoder of the current playback device;

[0062] You can use technologies such as DXVA (DirectX Video Acceleration), DXVA2 (DirectX Video Acceleration Version 2), VDPAU (Video Decode and Presentation API for Unix), and VAAPI (Video Acceleration API) to calculate the video memory usage or graphics card usage of each hardware decoder on the current playback device. The estimated performance usage of each hardware decoder on the current playback device is calculated by summing the video memory usage or graphics card usage of each hardware decoder on the current playback device with the estimated video memory usage or graphics card usage of the corresponding hardware decoder model for decoding video files. That is, estimated performance usage = video memory usage + estimated video memory usage, or estimated performance usage = graphics card usage + estimated graphics card usage.

[0063] S143, calculating the performance threshold of each model of hardware decoder according to the performance redundancy value of the hardware decoder of each model of hardware decoder;

[0064] The hardware decoder's performance threshold = 100% - the hardware decoder's performance margin. The hardware decoder's performance margin is calculated as the hardware decoder's initial performance margin plus the hardware decoder's performance margin consumption. The hardware decoder's initial performance margin is determined by the hardware decoder's performance margin base value (in percentage) and the hardware tuning deviation value (in percentage). The hardware tuning deviation value can be obtained by looking up a table. The hardware tuning deviation table lists hardware tuning deviation values ​​based on the hardware model, system version, and hardware driver version. Based on the hardware specifications provided by the manufacturer, higher-end hardware models and newer system and hardware driver versions have lower hardware tuning deviations. The hardware decoder's performance margin base value can be 10%, 30%, or other custom values. This value is determined by the application's playback and decoding policy. For example, when the playback and decoding policy is playback priority, the performance margin base value can be set to 10%. In this case, the hardware decoder's performance margin is (10% + hardware tuning deviation). When the playback and decoding policy is global priority, the performance margin base value can be set to 30%. In this case, the hardware decoder's performance margin is (30% + hardware tuning deviation). The performance redundancy consumption value of the hardware decoder is the performance value consumed by the hardware decoder relative to the performance value before decoding. The performance redundancy consumption value of the hardware decoder is obtained by subtracting the performance consumption base value (which may be 0) from the performance consumption value of the hardware decoder obtained through the hardware driver interface. The performance consumption value of the hardware decoder before the first video is played (which may be 0) is obtained as the performance consumption base value, that is, the performance redundancy consumption value of the hardware decoder = the performance consumption value of the hardware decoder - the performance consumption base value of the hardware decoder.

[0065] When receiving a video playback instruction for N video channels, for the first video channel, the performance threshold of each hardware decoder model is calculated based on the initial performance redundancy value of each hardware decoder model. That is, the performance redundancy value of each hardware decoder at this time is the initial performance redundancy value (initial value). At this time, the performance threshold of the hardware decoder = 100% - the initial performance redundancy value of the hardware decoder. Where N is an integer greater than 0. Assume that the initial performance redundancy value of a hardware decoder is 10%, and the performance consumption base value of the hardware decoder before the first video channel is played is 0. When the first video channel is received, the performance threshold of the hardware decoder is 90%;

[0066] For the i-th (i = 2, 3, …, N) video, the performance threshold of each type of hardware decoder is calculated according to the initial performance redundancy value of each type of hardware decoder and the performance redundancy consumption value of the hardware decoder selected for the previous i-1 videos after decoding. That is, in the multi-video playing scene, the performance redundancy value of the hardware decoder that has been selected for decoding is the sum of the initial performance redundancy value and the performance redundancy consumption value. For example, assuming that there are two videos, the playing device has two hardware decoders A and B, the initial performance redundancy values of the hardware decoders A and B are both 10%, and the performance consumption base values of the hardware decoders A and B before playing the first video are both 0. At this time, the performance thresholds of the hardware decoders A and B are both 90%. The scheduling result of the first video is to select the hardware decoder A to decode the video. When scheduling the soft and hardware decoders for the second video, the performance threshold of the hardware decoder B is calculated. Since the hardware decoder B is not selected to decode the first video, the hardware decoder B has no performance consumption, that is, the performance redundancy consumption value of the hardware decoder is 0. At this time, the performance threshold of the hardware decoder B remains unchanged and is still 90%. When calculating the performance threshold of the hardware decoder A, the performance consumption value of the hardware decoder A for decoding the first video needs to be obtained. Assuming that the performance consumption value of the hardware decoder A for decoding the first video is 10%, since the performance consumption base value is 0, the performance redundancy consumption value of the hardware decoder A is 10%. The performance threshold of the hardware decoder A = 100% - 10% - 10% = 80%. For the i-th video, the performance redundancy consumption value of the hardware decoder is the performance value that has been consumed by the hardware decoder relative to the time when the hardware decoder is not decoded.

[0067] The present application considers that the prior art only selects the decoding mode from the perspective of single video playing. When facing the multi-video playing scene, a single decoding mode is still used for each video, which easily leads to excessive load of a single CPU or GPU, cannot fully utilize the hardware performance of the computer, and affects the overall playing effect of the multi-video. In order to improve this problem, the present application adaptively adjusts the performance redundancy value of the hardware decoder when facing the multi-video playing scene, and then adjusts the performance threshold of the hardware decoder. After the hardware decoder decodes a certain video, the performance redundancy consumption value of the hardware decoder after decoding is calculated to update the performance redundancy value of the hardware decoder, and then the performance threshold of the hardware decoder is adjusted, so as to realize the adjustment of the performance threshold of the hardware decoder before and after decoding the video, thereby avoiding the problem that a certain hardware decoder is repeatedly selected to decode the video when playing the multi-video, leading to excessive load of a certain GPU.

[0068] S144. Determine whether there is a hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder; if so, determine that the current playback device has a hardware decoder that meets the performance redundancy requirements; if not, determine that the current playback device does not have a hardware decoder that meets the performance redundancy requirements.

[0069] Determine whether there is a hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder. If there is a hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder, then it is determined that the current playback device has a hardware decoder that meets the performance redundancy requirements, and in this case, the hardware decoder that meets the performance redundancy requirements is selected to decode the video file; if there is no hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder, then it is determined that the current playback device does not have a hardware decoder that meets the performance redundancy requirements, and in this case, the software decoder is selected to decode the video file.

[0070] After the video file is successfully decoded, the decoded video file is rendered to play the video file.

[0071] The video hardware and software decoding scheduling method provided in this application determines whether a hardware decoder can be selected by judging whether the hardware decoder of the current playback device supports decoding video files. When the hardware decoder of the current playback device supports decoding video files, it further judges whether the playback device has a hardware decoder with sufficient performance redundancy for video decoding, thereby determining whether to schedule a hardware decoder or a software decoder for video decoding. This application ensures that the computer's hardware performance can be fully utilized to meet more video playback needs through the scheduling of multiple hardware decoders and the combination of software decoding and hardware decoding.

[0072] The above is an embodiment of a video software and hardware decoding scheduling method provided by the present application. The following is another embodiment of a video software and hardware decoding scheduling method provided by the present application.

[0073] Please refer to Figure 2 , an embodiment of the present application provides a video software and hardware decoding scheduling method, comprising:

[0074] Step 210: Receive video playback instructions for each video channel, and obtain the video file corresponding to the video playback instruction.

[0075] When a user initiates multi-channel video playback, multiple video playback instructions will be received. The video playback instructions of each video can be sorted according to the reception time of the video playback instructions of each video to generate a video list, and then the software and hardware decoding of each video can be scheduled in sequence according to the order of the video list; when the video playback instructions of each video are received at the same time, the video playback instructions can be randomly selected and sorted to generate a video list, and then the software and hardware decoding of each video can be scheduled in sequence according to the order in the video list.

[0076] After receiving a video playback command, the video file specified by the Uniform Resource Locator (URL) carried in the video playback command is located to obtain the video file corresponding to the video playback command for subsequent decoder operations. A URL is a standard format for identifying resource locations on the internet, commonly entered in a browser address bar to access a web page. A URL contains information such as the protocol (such as http or https), the server domain name / IP address, an optional port number, a path, and a query string, and is used to uniquely identify a resource on the internet.

[0077] Step 220: Acquire data information of the video file and hardware information of the current playback device.

[0078] The data information of the video file can be obtained from the digital retina platform. The data information includes information such as video encoding format, resolution, configuration parameters, etc. The hardware information of the current playback device (such as a computer) can be queried through the operating system interface. The hardware information includes the hardware decoder model, video memory, decoding formats supported by the hardware decoder, and resolution. Specifically, the decoding format support of the hardware decoder can be determined by parsing the hardware GUID information. After obtaining the hardware information of the current playback device, the hardware information can be saved in the cache module. When playing a new video later, the cache information can be directly read to obtain the hardware information, saving time.

[0079] This application directly obtains relevant information of video files based on the digital retina platform. Compared with the traditional method that requires decapsulation and decoding to obtain information, it eliminates the streaming and decoding steps and can greatly reduce time. By reading the hardware GUID information through the operating system interface, it can accurately identify whether the hardware level supports the specified format and resolution video file.

[0080] Step 230 , determining whether the hardware decoder model of the current playback device belongs to a model in a preset blacklist; if so, selecting a software decoder to decode the video file; if not, executing step 240 .

[0081] Match the hardware decoder model of the current playback device with the models in the preset blacklist. If the hardware decoder model of the current playback device matches a model in the preset blacklist, select the software decoder to decode the video file and do not enter the next scheduling judgment process; if the hardware decoder model of the current playback device does not match any model in the preset blacklist, execute step 240.

[0082] Among them, the preset blacklist can store hardware decoder models with poor decoding effect or high decoding failure rate. This application uses the hardware decoding blacklist mechanism for judgment, which can quickly screen out hardware decoders that do not meet the conditions and reduce the time consumption of subsequent links.

[0083] Step 240 , determining whether the operating system version of the current playback device meets the preset conditions, if so, executing step 250 , if not, selecting a software decoder to decode the video file.

[0084] The operating system version of the current playback device can be obtained through the computing system interface to determine whether the operating system version of the current playback device meets the preset conditions. For example, Windows 7 systems cannot use DXVA2 and D3D11 technologies, and Linux systems cannot use DXVA, DXVA2, D3D11, D3D9, and other technologies, and can only use VDPAU and VAAPI technologies. At this point, the system and driver are further checked to determine whether the current playback device supports the required technology usage conditions. If it does not meet the preset conditions, the video file to be played is directly decoded using software, and the next scheduling judgment process is not entered. If it meets the preset conditions, step 250 is executed.

[0085] Step 250: Determine whether the hardware decoder of the current playback device supports decoding the video file based on the data information and hardware information. If so, execute step 260; if not, select a software decoder to decode the video file.

[0086] Compare the video encoding format and resolution of the video file with the decoding format and resolution supported by the hardware decoder of the current playback device to determine whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device; if so, determine that the hardware decoder of the current playback device supports decoding the video file, and execute step 260; if not, determine that the hardware decoder of the current playback device does not support decoding the video file, and directly select the software decoder to decode the video file, that is, perform software decoding on the video file.

[0087] It should be noted that the order of step 230, step 240 and step 250 can be swapped.

[0088] Step 260: Determine whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder. If so, select the hardware decoder that meets the performance redundancy requirements to decode the video file. If not, select the software decoder to decode the video file.

[0089] Based on the performance usage information of the hardware decoder, the estimated value of the decoding consumption of the video file and the performance redundancy value of the hardware decoder, it is judged whether the current playback device has a hardware decoder that meets the performance redundancy requirements. The specific judgment process can refer to the above embodiment and will not be repeated here. When the number of decodings of the hardware decoder that meets the performance redundancy requirements is 1, the hardware decoder is directly selected to decode the video file. When the number of decodings of the hardware decoder that meets the performance redundancy requirements is multiple, a hardware decoder can be selected from the multiple hardware decoders that meet the performance redundancy requirements in combination with the scheduling order of each hardware decoder to decode the video file, and the hardware decoder with a high scheduling order can be preferably selected to decode the video file. For example, the current playback device has two hardware decoders A and B that meet the performance redundancy requirements. When the scheduling order of hardware decoder A is higher than that of hardware decoder B, hardware decoder A is preferably selected to decode the video file; when the scheduling order of hardware decoder A is the same as the scheduling order of hardware decoder B, a hardware decoder can be randomly selected from hardware decoders A and B.

[0090] Furthermore, each hardware decoder of the current playback device can be scored based on the hardware decoder model and the preset decoder list, and the scheduling order of each hardware decoder can be determined based on the score of each hardware decoder. Performance scores can be set for each hardware decoder model based on the decoding performance of each hardware decoder model in advance to generate a preset decoder list.

[0091] In one embodiment, step 260 further includes:

[0092] Step 270: Detect whether an exception occurs in the hardware decoder during the decoding process. If so, reschedule the decoder. If not, record the video memory usage or graphics card usage occupied by the hardware decoder during decoding.

[0093] When performing hardware decoding on a video file, the operating system interface and the driver interface can be called to record the video memory usage or graphics card usage occupied by the video decoding. At the same time, it is also possible to detect in real time whether any abnormality occurs in the hardware decoder during the decoding process, including failure to create a decoder, decoding failure, low decoding frame rate and other abnormalities. If an abnormality is detected, the decoder can be re-scheduled, that is, return to the step corresponding to obtaining hardware information to select a new hardware decoder. If there is no suitable hardware decoder, the software decoder is used; if no abnormality is detected, that is, decoding is successful, the video memory usage or graphics card usage occupied by the hardware decoder during decoding is recorded.

[0094] This application ensures that the computer's hardware performance can be fully utilized to meet the needs of more video playback channels through the scheduling of multiple hardware decoders and the combination of software decoding and hardware decoding; through statistical estimation and monitoring of the overall utilization rate of the decoder and the video memory utilization rate, it can effectively ensure that the hardware decoder has sufficient performance redundancy, thereby ensuring the playback effect of multiple videos; this application directly obtains relevant information of video files based on the digital retina platform. Compared with the traditional method, which requires decapsulation and decoding to obtain information, it eliminates the links of pulling streams and decoding, and can greatly reduce time.

[0095] The following is a specific application example of a video software and hardware decoding scheduling method provided by this application.

[0096] Step 1: The user uses a computer to play multiple videos. The software and hardware decoding scheduling system receives the video playback instruction, locates the video file specified by the URL, and obtains the video file corresponding to the video playback instruction;

[0097] Step 2: The scheduling system obtains the corresponding video file from the digital retina platform in HEVC format with a video resolution of 1920x1080;

[0098] Step 3. The user's computer has two hardware decoders, obtained through the operating system interface. The models are: Intel Iris Xe Graphics (hereinafter referred to as hardware decoder A) and GeForce RTX 4060 Laptop GPU (hereinafter referred to as hardware decoder B). Both hardware decoders are not on the hardware decoding blacklist for the video of the encoding format and resolution.

[0099] Step 4: Through the operating system interface, it is obtained that the user's computer is Windows 11. Combined with the driver interface information, it is determined that the computer can support DXVA2 and D3D11 technologies and meet the hardware decoding requirements;

[0100] Step 5: Read the GUID information of the hardware decoder through the operating system interface to determine the encoding format and resolution supported by the hardware decoding. It is determined that both hardware decoders support hardware decoding of video files in HEVC format and resolution of 1920x1080.

[0101] Step 6: The current overall usage rate and video memory occupancy rate of hardware decoder A and hardware decoder B are obtained by calling the system interface and the hardware decoder driver interface. After judging that both hardware decoder A and hardware decoder B meet the performance redundancy requirements, but hardware decoder B has better decoding performance, that is, the scheduling order of hardware decoder B is higher than that of hardware decoder A, so hardware decoder B is called first to decode the video file;

[0102] Step 7: The called hardware decoder B decodes the video file normally and completes the video playback operation;

[0103] Step 8. The user continues to play the next video and repeats steps 1 to 6. During the decoding process, the corresponding graphics card usage growth and video memory growth are counted as reference evaluation conditions for subsequent evaluation. When hardware decoder B detects insufficient performance redundancy, hardware decoder A is used for hardware decoding and step 7 is executed. When both hardware decoder A and hardware decoder B have no redundant decoding performance, the software decoder is used to decode the video file.

[0104] Due to changes in bit rate, there will be a certain degree of changes in usage and video memory during the hardware decoding process. By recording the increase in usage and video memory of the hardware decoder during the video file decoding process, it can effectively ensure that the hardware decoder has sufficient performance redundancy to ensure the playback effect of multiple videos.

[0105] Please refer to Figure 2 , the embodiment of the present application also provides a video software and hardware decoding scheduling system, including:

[0106] A receiving unit, configured to receive a video playback instruction and obtain a video file corresponding to the video playback instruction;

[0107] An information acquisition unit, used to acquire data information of the video file and hardware information of the current playback device;

[0108] The first judgment unit is used to judge whether the hardware decoder of the current playback device supports decoding the video file according to the data information and the hardware information, and if so, trigger the second judgment unit; if not, select the software decoder to decode the video file;

[0109] The second judgment unit is used to judge whether the current playback device has a hardware decoder that meets the performance redundancy requirements based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file and the performance redundancy value of the hardware decoder. If so, the hardware decoder that meets the performance redundancy requirements is selected to decode the video file; if not, the software decoder is selected to decode the video file.

[0110] As a further improvement, the first judgment unit is specifically configured to:

[0111] Comparing the video encoding format and resolution of the video file with the decoding format and resolution supported by the hardware decoder of the current playback device to determine whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device;

[0112] If so, it is determined that the hardware decoder of the current playback device supports decoding the video file, triggering the second judgment unit;

[0113] If not, it is determined that the hardware decoder of the current playback device does not support decoding the video file, and the software decoder is selected to decode the video file;

[0114] As a further improvement, the device further comprises:

[0115] The third judgment unit is used to judge whether the operating system version of the current playback device meets the preset conditions. If so, the second judgment unit is triggered; if not, a software decoder is selected to decode the video file.

[0116] As a further improvement, the device further comprises:

[0117] The fourth judgment unit is used to judge whether the hardware decoder model of the current playback device belongs to the models in the preset blacklist. If so, the software decoder is selected to decode the video file. If not, the second judgment unit is triggered.

[0118] As a further improvement, the device further comprises:

[0119] The scheduling order generating unit is used to score each hardware decoder of the current playback device according to the hardware decoder model of the current playback device and the preset decoder list; and determine the scheduling order of each hardware decoder according to the score of each hardware decoder.

[0120] As a further improvement, the device further comprises:

[0121] The detection unit is used to detect whether an exception occurs in the decoding process of the hardware decoder. If so, the decoder is rescheduled. If not, the video memory usage or graphics card usage of the hardware decoder during decoding is recorded.

[0122] This application ensures that the computer's hardware performance can be fully utilized to meet the needs of more video playback channels through the scheduling of multiple hardware decoders and the combination of software decoding and hardware decoding. By statistically estimating and monitoring the overall utilization rate and video memory utilization rate of the decoder, it can effectively ensure that the hardware decoder has sufficient performance redundancy, thereby ensuring the playback effect of multiple videos.

[0123] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In the specification of this application and the above-mentioned drawings, the terms "first," "second," "third," "fourth," etc. (if any) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0125] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0128] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0129] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name: Read-Only Memory, English abbreviation: ROM), random access memory (full name: Random Access Memory, English abbreviation: RAM), disk or optical disk, and other media that can store program code.

[0130] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video software and hardware decoding scheduling method, characterized in that: include: S1. Receive video playback instructions for each video channel and obtain the video file corresponding to the video playback instruction; S2. Obtain data information of the video file and hardware information of the current playback device; S3, judging whether the hardware decoder of the current playback device supports decoding the video file based on the data information and the hardware information, and if so, executing step S4; if not, selecting a software decoder to decode the video file; S4. Determine whether the current playback device has a hardware decoder that meets the performance redundancy requirement based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder. If so, select the hardware decoder that meets the performance redundancy requirement to decode the video file. If not, select a software decoder to decode the video file. The determining whether the current playback device has a hardware decoder that meets the performance redundancy requirement based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder includes: Evaluate the video memory usage or graphics card usage of each model of hardware decoder for decoding the video file based on the resolution of the video file and the model of the hardware decoder, and obtain an estimated video memory usage or graphics card usage of each model of hardware decoder for decoding the video file; Summing the video memory usage or graphics card usage of each hardware decoder of the current playback device with the estimated video memory usage or graphics card usage of the corresponding hardware decoder for decoding the video file, to obtain an estimated performance usage value of each hardware decoder of the current playback device; The performance threshold of each hardware decoder model is calculated based on the performance redundancy value of each hardware decoder model. The performance threshold of the hardware decoder is the difference between 100% and the performance redundancy value of the hardware decoder. The performance redundancy value of the hardware decoder is the sum of the initial performance redundancy value of the hardware decoder and the performance redundancy consumption value of the hardware decoder. The performance redundancy consumption value of the hardware decoder is the performance value consumed by the hardware decoder relative to that before decoding. Determine whether there is a hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder; If so, it is determined that the current playback device has a hardware decoder that meets the performance redundancy requirements; If not, it is determined that the current playback device does not have a hardware decoder that meets the performance redundancy requirements; The selecting a hardware decoder that meets the performance redundancy requirement to decode the video file includes: When there are multiple hardware decoders that meet the performance redundancy requirement, selecting a hardware decoder to decode the video file from the multiple hardware decoders that meet the performance redundancy requirement in combination with the scheduling order of the hardware decoders; The process of determining the scheduling order of each hardware decoder includes: Scoring each hardware decoder of the current playback device according to the hardware decoder model of the current playback device and the preset decoder list; determining the scheduling order of each hardware decoder according to the score of each hardware decoder.

2. The video software and hardware decoding scheduling method according to claim 1, characterized in that: The data information includes the video encoding format and resolution; the hardware information includes the hardware decoder model, the decoding format and resolution supported by the hardware decoder.

3. The video software and hardware decoding scheduling method according to claim 2, characterized in that: The determining, based on the data information and the hardware information, whether the hardware decoder of the current playback device supports decoding the video file includes: Comparing the video encoding format and resolution of the video file with the decoding format and resolution supported by the hardware decoder of the current playback device, and determining whether the video encoding format and resolution of the video file are the decoding format and resolution supported by the hardware decoder of the current playback device; If so, determining that the hardware decoder of the current playback device supports decoding the video file; If not, it is determined that the hardware decoder of the current playback device does not support decoding the video file.

4. The video software and hardware decoding scheduling method according to claim 1, characterized in that: Before step S4, the method further includes: Determine whether the operating system version of the current playback device meets the preset conditions. If so, execute step S4. If not, select a software decoder to decode the video file.

5. The video software and hardware decoding scheduling method according to claim 1, characterized in that: Before step S4, the method further includes: Determine whether the hardware decoder model of the current playback device belongs to the models in the preset blacklist. If so, select the software decoder to decode the video file. If not, execute step S4.

6. The video software and hardware decoding scheduling method according to claim 1, characterized in that: The method further comprises: Detect whether the hardware decoder has an exception during the decoding process. If so, reschedule the decoder. If not, record the video memory usage or graphics card usage during the hardware decoder decoding.

7. A video software and hardware decoding scheduling system, characterized in that: include: A receiving unit, configured to receive a video playback instruction for each video channel and obtain a video file corresponding to the video playback instruction; An information acquisition unit, configured to acquire data information of the video file and hardware information of the current playback device; a first determining unit, configured to determine, based on the data information and the hardware information, whether the hardware decoder of the current playback device supports decoding the video file; if so, triggering the second determining unit; if not, selecting the software decoder to decode the video file; a second determining unit, configured to determine whether a hardware decoder that meets the performance redundancy requirement exists on the current playback device based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder; if so, selecting the hardware decoder that meets the performance redundancy requirement to decode the video file; and if not, selecting a software decoder to decode the video file; The determining whether the current playback device has a hardware decoder that meets the performance redundancy requirement based on the performance usage information of the hardware decoder, the estimated decoding consumption of the video file, and the performance redundancy value of the hardware decoder includes: Evaluate the video memory usage or graphics card usage of each model of hardware decoder for decoding the video file based on the resolution of the video file and the model of the hardware decoder, and obtain an estimated video memory usage or graphics card usage of each model of hardware decoder for decoding the video file; Summing the video memory usage or graphics card usage of each hardware decoder of the current playback device with the estimated video memory usage or graphics card usage of the corresponding hardware decoder for decoding the video file, to obtain an estimated performance usage value of each hardware decoder of the current playback device; The performance threshold of each hardware decoder model is calculated based on the performance redundancy value of each hardware decoder model. The performance threshold of the hardware decoder is the difference between 100% and the performance redundancy value of the hardware decoder. The performance redundancy value of the hardware decoder is the sum of the initial performance redundancy value of the hardware decoder and the performance redundancy consumption value of the hardware decoder. The performance redundancy consumption value of the hardware decoder is the performance value consumed by the hardware decoder relative to that before decoding. Determine whether there is a hardware decoder whose expected performance usage value is less than or equal to the performance threshold of the corresponding model hardware decoder; If so, it is determined that the current playback device has a hardware decoder that meets the performance redundancy requirements; If not, it is determined that the current playback device does not have a hardware decoder that meets the performance redundancy requirements; The selecting a hardware decoder that meets the performance redundancy requirement to decode the video file includes: When there are multiple hardware decoders that meet the performance redundancy requirement, selecting a hardware decoder to decode the video file from the multiple hardware decoders that meet the performance redundancy requirement in combination with the scheduling order of the hardware decoders; The process of determining the scheduling order of each hardware decoder includes: Scoring each hardware decoder of the current playback device according to the hardware decoder model of the current playback device and the preset decoder list; determining the scheduling order of each hardware decoder according to the score of each hardware decoder.

Images