A multi-channel dynamic video stream decoding method and terminal equipment

Abstract

The present invention discloses a multi-channel dynamic video stream decoding method and terminal device, which belongs to the field of video stream decoding technology, and includes the following steps: defining three scenarios and five priorities; executing initialization code to obtain the maximum decoding frame rate of the terminal device; dividing the terminal into two types: high-configuration device and low-configuration device; obtaining the video stream and decoding and displaying it through the terminal device; the CPU of the terminal device dynamically schedules the decoding frequency, and sends it to the image processing module of the terminal device for decoding, to obtain multi-channel decoded video data, and then the CPU is responsible for dynamically scheduling the display frequency, and sending it to the display screen of the terminal device for display. Through the above method, the present invention dynamically adjusts the priority strategy according to the timed task, and dynamically adjusts the decoding frequency and the display frequency according to the priority, which greatly reduces the load of the entire device, so that the high-priority video stream can be played without lag.

Description

Technical Field

[0001] The present invention relates to the technical field of video stream decoding, and in particular to a multi-channel dynamic video stream decoding method and terminal equipment. Background Art

[0002] With the development of technology, more and more terminal devices support multi-channel decoding and display. However, due to the performance limitations of the terminal devices, multi-channel decoding causes playback freezes and a poor experience.

[0003] Patent CN116506618A discloses a method for optimizing video decoding based on dynamic load adaptation, which selects different decoding methods according to the mapping relationship between decoding time and CPU / GPU occupancy rate to solve the problem of smooth video playback under high load. However, this solution is mainly aimed at decoding a single video, and requires knowing the GPU occupancy rate, which is difficult for some terminal devices.

[0004] Patent CN110381322A discloses a video stream decoding method, device, terminal device and storage medium, which uses GPU to replace CPU to complete multi-channel parallel decoding, thereby improving video decoding efficiency and GPU utilization. This solution is aimed at multi-channel video decoding, but it only replaces CPU with GPU. If the GPU capability of the terminal device is weak, multi-channel decoding will also be stuck.

[0005] Based on this, the present invention designs a multi-channel dynamic video stream decoding method and terminal equipment to solve the above problems. Summary of the invention

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides a multi-channel dynamic video stream decoding method and terminal equipment.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A method for decoding multi-channel dynamic video streams, comprising the following steps:

[0009] Step 1: define three scenarios: video stream display, thumbnail display, no decoding and no thumbnail display. The three scenarios are specifically defined as five priorities.

[0010] Step 2: Execute the initialization code to obtain the maximum decoding frame rate of the terminal device;

[0011] Step 3: Set terminal configuration attributes: According to the maximum decoding frame rate, the terminals are divided into high-configuration devices and low-configuration devices;

[0012] Step 4: Obtain the video stream and decode and display it through the terminal device;

[0013] Step 5: The CPU of the terminal device dynamically schedules the decoding frequency and sends it to the image processing module of the terminal device for decoding to obtain multi-channel decoded video data. The CPU is then responsible for dynamically scheduling the display frequency and sending it to the display screen of the terminal device for display;

[0014] The dynamic scheduling of decoding frequency and dynamic scheduling of display frequency is achieved by adjusting the priority according to the percentage of the maximum number of decodings or the maximum decoding frame rate occupied by the current decoding of the terminal device.

[0015] Further, specifically, the three scenarios are:

[0016] Scenario 1: The received video stream is displayed on the display screen;

[0017] Scenario 2: The received video stream is in the list to be displayed on the screen, and the I frame is obtained to display the thumbnail;

[0018] Scenario 3: The received video stream is in the list to be displayed on the screen. The video stream is obtained, but it is not decoded and the thumbnail is not displayed.

[0019] To go further, the five priorities are:

[0020] First priority: the video stream being displayed on the display;

[0021] Second priority: Video streams that are in the list to be displayed and have been connected for no more than 1 day are decoded normally, and each I frame is sent and displayed in the list to be displayed;

[0022] The third priority: for video streams that are in the list to be displayed and have not been displayed for more than one day, only the I frames are decoded and displayed in the list to be displayed;

[0023] Fourth priority: Video streams that are in the list to be displayed and have not been displayed for more than 7 days will be decoded and displayed every 5 seconds.

[0024] Fifth priority: For video sources in the list to be displayed on the screen, only the video stream is obtained without decoding or displaying.

[0025] Furthermore, in step 2, the maximum decoding frame rate of the terminal device is obtained by loading a 60-frame video when the terminal device is started for the first time, calling the relevant methods of Android's MediaCodec, not setting the decoding frequency, decoding according to the maximum performance of the terminal device, and measuring the maximum decoding frame rate of the terminal device when idle.

[0026] Furthermore, the maximum decoding frame rate is less than 120 frames, that is, less than four-way simultaneous decoding, and is set as a low-configuration device; the low-configuration device scenario only needs to configure scene 1 and scene 3, and the priority only needs to configure the first priority and the fifth priority;

[0027] The maximum decoding frame rate is ≥120 frames, set to high configuration; high configuration devices are configured with three scenarios and five priorities.

[0028] Furthermore, in step 4, the network camera video stream is obtained through the RTSP protocol, the mobile phone and pad are projected to the terminal device, the terminal device obtains the video stream of the mobile phone and pad, and the network live video stream is obtained through the RTMP streaming protocol.

[0029] Furthermore, the priority is adjusted according to the percentage of the maximum number of decodings or the maximum decoding frame rate of the terminal device occupied by the current decoding, and the steps are as follows:

[0030] 5.1. Determine whether it is a high-configuration device;

[0031] A. If not, it is a low-configuration device; then determine whether it is displayed on the screen; if it is displayed on the screen, determine whether all video sources on the screen exceed the maximum number of decoding. If not, add the first priority and decode and display normally; if it exceeds, add the fifth priority, prompt that it cannot be displayed on the screen, only obtain the video stream, and do not decode and display; if it is not displayed on the screen, directly set it to the fifth priority, only obtain the video stream, and do not decode;

[0032] B. If yes, it is a high-configuration device; then determine whether it is displayed on the screen; if it is displayed on the screen, determine whether all video sources on the screen exceed 70% of the maximum decoding frame rate. If not, add the first priority and decode and display normally; if it exceeds, add the second priority, prompt that it cannot be displayed on the screen, decode normally, send I frame to display, and refresh the survival time of the video stream; if it is not displayed on the screen, directly set it to the fifth priority, only obtain the video stream without decoding;

[0033] 5.2. When a new video source is added to the corresponding priority for decoding, if the current decoding frame rate exceeds 90% of the maximum decoding frame rate, the priority of the video source is reduced until it is reduced to the lowest priority. When it is at the fifth priority, only the video stream is obtained without decoding;

[0034] 5.3. For high-configuration devices, check whether the sources in the list exceed the set threshold according to the scheduled task. If exceeded, the priority is automatically lowered.

[0035] Furthermore, it is detected whether the survival time of the second priority video stream in the video stream list exceeds 1 day. If so, the second priority video stream is added to the third priority queue, and only the I frame is decoded and displayed.

[0036] Furthermore, it is detected whether the survival time of the third priority video stream in the video stream list exceeds 7 days. If so, the video stream in the third priority queue is added to the fourth priority queue, and the I frame is decoded and displayed every 5 seconds.

[0037] The present invention also provides a terminal device for a multi-channel dynamic video stream decoding method, comprising a processor and a memory;

[0038] The memory is used to store program codes for executing a method for decoding multiple dynamic video streams, and transmit the program codes to the processor;

[0039] The processor is used to execute a multi-channel dynamic video stream decoding method according to instructions in the program code.

[0040] Beneficial Effects

[0041] The present invention comprehensively considers the decoding performance of the terminal device, dynamically adjusts the priority strategy according to the scheduled tasks, and dynamically adjusts the decoding frequency and the display frequency according to the priority, thereby greatly reducing the load of the entire device, so that high-priority video streams can be played without any lag. It can be widely used in scenarios where terminal devices process the decoding and display of multiple video streams, which is conducive to obtaining a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 The present invention is a flowchart of a method for decoding multi-channel dynamic video streams. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The present invention will be further described below in conjunction with the embodiments.

[0046] Example 1

[0047] Please refer to the instruction manual Figure 1 , a multi-channel dynamic video stream decoding method, comprising the following steps:

[0048] Step 1: Define three scenarios and five priorities;

[0049] Specifically, there are three scenarios:

[0050] Scenario 1: The received video stream is displayed on the display screen;

[0051] Scenario 2: The received video stream is in the list to be displayed on the screen, and the I frame is obtained to display the thumbnail;

[0052] Scenario 3: The received video stream is in the list to be displayed on the screen. The video stream is obtained through airplay, miracast and other protocols, but is not decoded and thumbnails are not displayed.

[0053] Specifically, the five priorities are:

[0054] First priority: the video stream being displayed on the display;

[0055] Second priority: Video streams that are in the list to be displayed and have been connected for no more than 1 day are decoded normally, and each I frame is sent and displayed in the list to be displayed;

[0056] The third priority: for video streams that are in the list to be displayed and have not been displayed for more than one day, only the I frames are decoded and displayed in the list to be displayed;

[0057] Fourth priority: Video streams that are in the list to be displayed and have not been displayed for more than 7 days will be decoded and displayed every 5 seconds.

[0058] Fifth priority: For video sources in the list to be displayed on the screen, only the video stream is obtained without decoding or displaying.

[0059] Step 2: Execute the initialization code to obtain the maximum decoding frame rate of the terminal device;

[0060] Specifically, the maximum decoding frame rate of the terminal device is obtained by loading a 60-frame video when the terminal device is first started, calling the relevant methods of Android's MediaCodec, not setting the decoding frequency, decoding according to the maximum performance of the terminal device, and measuring the maximum decoding frame rate of the terminal device when idle;

[0061] Step 3: Set terminal configuration attributes: According to the maximum decoding frame rate, the terminals are divided into high-configuration devices and low-configuration devices;

[0062] The maximum decoding frame rate is less than 120 frames, that is, less than four-channel simultaneous decoding, and is set as a low-configuration device. For low-configuration devices, only scene 1 and scene 3 need to be configured, and the priority only needs to be configured as the first priority and the fifth priority.

[0063] The maximum decoding frame rate is ≥120 frames, set to high configuration; high configuration devices are configured with three scenarios and five priorities;

[0064] Related protocol implementation: Implement RTSP protocol client, Airplay and Miracast protocol server, RTMP streaming protocol, etc. in the Android framework;

[0065] Step 4: Obtain video streams from different sources, such as network cameras, screen projection devices, etc.; the video streams are decoded and displayed by the terminal device;

[0066] Specifically, the video stream of the network camera is obtained through the RTSP protocol, and the mobile phone, pad, etc. are projected to the terminal device. The terminal device obtains the video stream of the mobile phone and pad, etc., and obtains the live video stream through the RTMP streaming protocol;

[0067] Step 5: The CPU of the terminal device dynamically schedules the decoding frequency and sends it to the image processing module of the terminal device for decoding to obtain multi-channel decoded video data. The CPU is then responsible for dynamically scheduling the display frequency and sending it to the display screen of the terminal device for display;

[0068] The dynamic scheduling of decoding frequency and dynamic scheduling of display frequency is achieved by adjusting the priority according to the percentage of the maximum number of decodings or the maximum decoding frame rate occupied by the current decoding of the terminal device, and the steps include:

[0069] 5.1. Determine whether it is a high-configuration device;

[0070] A. If not, it is a low-configuration device; then determine whether it is displayed on the screen; if it is displayed on the screen (sent to the display screen), determine whether all video sources on the screen exceed the maximum number of decoding. If not, add the first priority and decode and display normally (scenario one); if it exceeds, add the fifth priority, prompt that it cannot be displayed on the screen, only obtain the video stream, and do not decode and display (scenario three). If it is not displayed on the screen, directly set it to the fifth priority, only obtain the video stream, and do not decode (scenario three).

[0071] B. If yes, it is a high-configuration device; then determine whether it is displayed on the screen; if it is displayed on the screen, determine whether all video sources on the screen exceed 70% of the maximum decoding frame rate. If not, add the first priority and decode and display normally (scenario one); if it exceeds, add the second priority, prompt that it cannot be displayed on the screen, decode normally, send I frames to display, and refresh the survival time of the video stream (scenario two). If it is not displayed on the screen, directly set it to the fifth priority, only obtain the video stream, and do not decode (scenario three).

[0072] 5.2. When a new video source is added to the corresponding priority for decoding, if the current decoding frame rate exceeds 90% of the maximum decoding frame rate, the priority of the video source is reduced until it is reduced to the lowest priority. When it is at the fifth priority, only the video stream is obtained without decoding;

[0073] 5.3. For high-configuration devices, check whether the sources in the list exceed the set threshold according to the scheduled task. If exceeded, the priority is automatically lowered.

[0074] Specifically, detect whether the survival time of the second priority video stream in the video stream list exceeds 1 day. If so, the second priority video stream is added to the third priority queue, and only I frames are decoded and displayed; and detect whether the survival time of the third priority video stream in the video stream list exceeds 7 days. If so, the video stream in the third priority queue is added to the fourth priority queue, and I frames are decoded and displayed every 5 seconds;

[0075] A terminal device for a multi-channel dynamic video stream decoding method, comprising a processor and a memory;

[0076] The memory is used to store program codes for executing a method for decoding multiple dynamic video streams, and transmit the program codes to the processor;

[0077] The processor is used to execute a multi-channel dynamic video stream decoding method according to instructions in the program code.

[0078] The present invention comprehensively considers the decoding performance of the terminal device, dynamically adjusts the priority strategy according to the scheduled tasks, and dynamically adjusts the decoding frequency and the display frequency according to the priority, thereby greatly reducing the load of the entire device, so that high-priority video streams can be played without any lag. It can be widely used in scenarios where terminal devices process the decoding and display of multiple video streams, which is conducive to obtaining a better user experience.

[0079] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for decoding multi-channel dynamic video streams, characterized in that: The following steps are involved: Step 1: define three scenarios: video stream display, thumbnail display, no decoding and no thumbnail display. The three scenarios are specifically defined as five priorities. Step 2: Execute the initialization code to obtain the maximum decoding frame rate of the terminal device; Step 3: Set terminal configuration attributes: According to the maximum decoding frame rate, the terminals are divided into high-configuration devices and low-configuration devices; Step 4: Obtain the video stream and decode and display it through the terminal device; Step 5: The CPU of the terminal device dynamically schedules the decoding frequency and sends it to the image processing module of the terminal device for decoding to obtain multi-channel decoded video data. The CPU is then responsible for dynamically scheduling the display frequency and sending it to the display screen of the terminal device for display; The dynamic scheduling of decoding frequency and dynamic scheduling of display frequency is achieved by adjusting the priority according to the percentage of the maximum number of decodings or the maximum decoding frame rate occupied by the current decoding of the terminal device; Specifically, the five priorities are: First priority: the video stream being displayed on the display; Second priority: Video streams that are in the list to be displayed and have been connected for no more than 1 day are decoded normally, and each I frame is sent and displayed in the list to be displayed; The third priority: for video streams that are in the list to be displayed and have not been displayed for more than one day, only the I frames are decoded and displayed in the list to be displayed; Fourth priority: Video streams that are in the list to be displayed and have not been displayed for more than 7 days will be decoded and displayed every 5 seconds. The fifth priority: For video sources in the list to be displayed, only the video stream is obtained without decoding or displaying. The maximum decoding frame rate is less than 120 frames, that is, less than four-channel simultaneous decoding, and is set to a low-configuration device. In the scenario of a low-configuration device, you only need to configure the video stream display, no decoding, and no thumbnail display, and the priority only needs to be configured as the first priority and the fifth priority. The maximum decoding frame rate is ≥120 frames, and the device is set to a high-configuration device. A high-configuration device is configured with three scenarios and five priorities. Priority adjustment is performed based on the percentage of the current decoding that occupies the maximum number of decodings or the maximum decoding frame rate of the terminal device. The steps are as follows: 5.

1. Determine whether it is a high-configuration device; A. If not, it is a low-configuration device; then determine whether it is displayed on the screen; if it is displayed on the screen, determine whether all video sources on the screen exceed the maximum number of decoding. If not, add the first priority and decode and display normally; if it exceeds, add the fifth priority, prompt that it cannot be displayed on the screen, only obtain the video stream, and do not decode and display; if it is not displayed on the screen, directly set it to the fifth priority, only obtain the video stream, and do not decode; B. If yes, it is a high-configuration device; then determine whether it is displayed on the screen; if it is displayed on the screen, determine whether all video sources on the screen exceed 70% of the maximum decoding frame rate. If not, add the first priority and decode and display normally; if it exceeds, add the second priority, prompt that it cannot be displayed on the screen, decode normally, send I frame to display, and refresh the survival time of the video stream; if it is not displayed on the screen, directly set it to the fifth priority, only obtain the video stream without decoding; 5.

2. When a new video source is added to the corresponding priority for decoding, if the current decoding frame rate exceeds 90% of the maximum decoding frame rate, the priority of the video source is reduced until it is reduced to the lowest priority. When it is at the fifth priority, only the video stream is obtained without decoding; 5.

3. For high-configuration devices, check whether the video sources in the list exceed the set threshold according to the scheduled task. If exceeded, the priority is automatically lowered; Detect whether the survival time of the second priority video stream in the video stream list exceeds 1 day. If so, the second priority video stream is added to the third priority queue and only decodes the I frame for display; Check whether the survival time of the third priority video stream in the video stream list exceeds 7 days. If so, the video stream in the third priority queue is added to the fourth priority queue and decoded I frames every 5 seconds for display.

2. The multi-channel dynamic video stream decoding method according to claim 1, characterized in that: Specifically, there are three scenarios: Scenario 1: The received video stream is displayed on the display screen; Scenario 2: The received video stream is in the list to be displayed on the screen, and the I frame is obtained to display the thumbnail; Scenario 3: The received video stream is in the list to be displayed on the screen. The video stream is obtained, but it is not decoded and the thumbnail is not displayed.

3. The multi-channel dynamic video stream decoding method according to claim 2, characterized in that: In step 2, the maximum decoding frame rate of the terminal device is obtained by loading a 60-frame video when the terminal device is started for the first time, calling the relevant methods of Android's MediaCodec, not setting the decoding frequency, decoding according to the maximum performance of the terminal device, and measuring the maximum decoding frame rate of the terminal device when idle.

4. The multi-channel dynamic video stream decoding method according to claim 3, characterized in that: In step 4, the network camera video stream is obtained through the RTSP protocol; or the mobile phone or pad is projected to the terminal device, the terminal device obtains the video stream of the mobile phone and pad, and obtains the live broadcast video stream through the RTMP streaming protocol.

5. A terminal device using the multi-channel dynamic video stream decoding method according to claim 4, characterized in that: including a processor and a memory; The memory is used to store program codes for executing a method for decoding multiple dynamic video streams, and transmit the program codes to the processor; The processor is used to execute the multi-channel dynamic video stream decoding method as claimed in claim 4 according to the instructions in the program code.

Images