A video code stream processing and rendering system and method

Abstract

The present invention provides a video code stream processing and rendering system and method, the system includes a video streaming server and client application software; the client application software includes a UI thread, a video data processing thread and a state management thread. The method is: for each code stream, the UI component whose picture is to be played is first registered with the corresponding state manager, the state manager records the instances and quantity of these UI components, and when the first UI component is registered, it will automatically initiate a code stream request to the server and continue until all UI components are unregistered; the state manager receives data and intercepts image frames and then decodes them by opening a separate video data processing thread; each time the state manager receives a new image frame from the video data processing thread, it will notify each registered UI component to update the rendering picture. The present invention can significantly reduce the client application software performance overhead and video streaming server load overhead of multi-channel code stream multi-screen rendering.

Description

Technical Field

[0001] The present invention relates to the technical field of video data processing, and in particular to a video code stream processing and rendering system and method. Background Art

[0002] For a video stream composed of multiple images (such as MJPEG), the client obtains the video stream pushed by the server through a network connection (such as HTTP connection), then decodes the image frame by frame and updates and renders it to the UI component. The general process is as follows: Figure 2 As shown:

[0003] The video data processing thread is responsible for establishing a network connection and initiating data requests to the video streaming server. The video streaming server will continuously push video images frame by frame to this client socket. The video data processing thread will parse and decode the received video data, separate the image data frames, and then update and render them on the UI component.

[0004] If there are multiple UI components in the client APP that need to display the same video source, the general implementation process is as follows: Figure 3 As shown:

[0005] At this time, the APP control program will create an independent video data processing thread for each UI component that displays the video screen. This thread will independently initiate a network request to the video streaming server, parse and decode the image frame, and then update and render it to the corresponding UI component. At this time, for the video streaming server, it is equivalent to having n clients requesting resources from it.

[0006] The above prior art has two obvious disadvantages:

[0007] (1) Occupying client resources: n UI components actually display the same video source, but they create n threads, which greatly occupies CPU and memory resources. In serious cases, it may cause the client APP to freeze or even crash;

[0008] (2) Occupying video streaming server resources: One client APP initiates n network requests to the video streaming server, causing the server to repeatedly consume streaming resources. In serious cases, this may cause the server to be overloaded and the bandwidth to be overused.

[0009] In view of this, Chinese patent CN202111151089.8 discloses a method for real-time processing of multiple video streams, same-screen playback and related systems. Among them, the real-time processing method of multiple video streams includes: receiving the real-time video streams sent by each video server through the associated stream receiving channel in parallel, splitting the received real-time video stream into multiple data frames, adding the data frames to the corresponding cache queue, binding the stream receiving channel to the allocated CPU memory, and the stream receiving channel corresponds to the cache queue one by one; for each cache queue, sending data frames to the corresponding hardware decoding cache area at a set interval; hardware decoding of the data frames in each decoding cache area in parallel, and delivering the obtained decoded data frames to the corresponding rendering cache area; parallel off-screen rendering of the decoded data frames in each rendering cache area to obtain the rendered data frames for playback. Although this patent also performs real-time processing of multiple streams, its focus is on the processing related to the data frame cache, but it cannot achieve optimization when a certain stream needs to be rendered in multiple places in the UI, and the rendering effect is limited.

[0010] To sum up, it is necessary to further improve and innovate the existing technology. Summary of the invention

[0011] In response to the technical problems existing in the above-mentioned background technology, the present invention proposes a video code stream processing and rendering system and method, which can significantly reduce the resource waste (including CPU resources, memory resources and bandwidth resources) of client application software and video streaming servers and improve their performance and load.

[0012] To solve the above technical problems, the present invention provides a video code stream processing and rendering system, comprising a video streaming server and a client application software connected and communicating with the video streaming server; the video streaming server is a service program deployed in the cloud, used to push video data streams to the client application software; the client application software comprises a UI thread and a video data processing thread; the running target of the UI thread is one or more UI components; the client application software also comprises a state management thread connected between the UI thread and the video data processing thread; the running target of the state management thread is a state manager;

[0013] The state management thread is used to run the code logic related to the state manager to ensure that the state manager is in an independent and asynchronous operating environment;

[0014] The state manager is used to request data from the video streaming server and maintain the obtained frame stream data and state information of each stage; the state manager also holds the monitoring callback of the UI component, and whenever there is a state change, the monitoring callback of the UI component is called to notify the UI thread to update the UI component;

[0015] The video data processing thread has one for collecting data pushed from the video streaming server, then decoding the received code stream data to obtain each frame of image data and pushing the image data to the state manager.

[0016] The video code stream processing and rendering system, wherein: when the client application software needs to display the same video source for more than one UI component, each time a UI component instance is created, the UI component instance must first be registered with the state manager and the image frame data state in the state manager must be monitored;

[0017] The video code stream processing and rendering system, wherein: when the UI component is destroyed, the UI component instance is unregistered from the state manager and the monitoring of the image frame data state is canceled.

[0018] The video code stream processing and rendering system, wherein: the state manager checks how many UI component instances are currently registered each time a UI component is registered or unregistered during state management.

[0019] The video code stream processing and rendering system, wherein: in state management, when the number of registered UI components changes from 0 to 1, the state manager will initiate a network request to the video streaming server and create a video data processing thread for receiving, parsing and decoding video frame image data. When the video data processing thread parses each image, it will push it to the state manager, and then the state manager will notify each registered UI component to update the rendering screen.

[0020] The video code stream processing and rendering system, wherein: in the state management, the state manager increases the number of registered UI components from non-zero. At this time, the video data processing thread has been started and no processing is performed.

[0021] The video code stream processing and rendering system, wherein: in the state management, when the number of registered UI components changes from non-zero to zero, the state manager will close the connection with the video streaming server to save resource consumption, and at the same time destroy the video data processing thread to release related resources.

[0022] A video code stream processing and rendering method, based on the video code stream processing and rendering system, is that for each code stream, the UI component that wants to play its picture first registers with the corresponding state manager, the state manager records the instances and quantity of these UI components, and when the first UI component is registered, it automatically initiates a code stream request to the server, and continues until all UI components are unregistered; the state manager receives data and intercepts image frames and then decodes them by opening a separate video data processing thread; whenever the state manager receives a new frame of image from the video data processing thread, it notifies each registered UI component to update the rendering picture

[0023] The video code stream processing and rendering method specifically comprises the following steps:

[0024] (1) When the client application software needs to display the same video source on multiple UI components, first create a corresponding number of UI components in the UI thread;

[0025] (2) When the UI component is created, it will actively register itself with the state manager to monitor the image frame and playback status from the state manager;

[0026] (3) When the first UI component is registered with the state manager, the state manager will initiate a network request to the video streaming server to obtain the video stream and create a video data processing thread to receive and process the stream data;

[0027] (4) After receiving the request command sent by the state manager, the video streaming server continuously pushes the video stream and the video stream data eventually flows into the video data processing thread;

[0028] (5) The video data processing thread caches and decodes the received data, pushes each decoded frame to the state manager, and clears the cache to continue collecting data for the next frame;

[0029] (6) Whenever the state manager obtains a frame of image from the video data processing thread, it updates its own state and then notifies each UI component registered with it;

[0030] (7) When the UI component receives notification from the state manager, it will update and display the latest image frame;

[0031] (8) In the above step (3), if the network request fails, the state manager will mark the state as connection failure and notify each UI component. At this time, the UI component will display an abnormal state.

[0032] The video code stream processing and rendering method, wherein: the client application software can dynamically increase the number of UI components; when a UI component is added, all UI components need to be registered with the state manager, and only when the first UI component is registered with the state manager will the process of step (3) be triggered.

[0033] The video code stream processing and rendering method, wherein: the client application software can dynamically reduce the number of UI components; when a UI component is reduced, all UI components need to be deregistered from the state manager, and when the last UI component is deregistered, the state manager will disconnect from the video streaming server, destroy the video data processing thread, and reset all its own states.

[0034] By adopting the above technical solution, the present invention has the following beneficial effects:

[0035] The video code stream processing and rendering system and method of the present invention are reasonably conceived and have good rendering effect. They can significantly reduce the resource waste (including CPU resources, memory resources and bandwidth resources) of client application software and video streaming servers and improve their performance and load. The video code stream processing and rendering system of the present invention can display multiple video images at the same time, and each video image can be repeatedly displayed on multiple UI components.

[0036] The specific advantages of the present invention are embodied in the following points:

[0037] (1) Since the number of video data processing threads is reduced to only one, the CPU overhead of n-1 threads (mainly the CPU overhead of thread management, network requests, and image decoding) and memory usage (mainly image data cache) are saved, thus greatly improving the performance of the client application software;

[0038] (2) Since only one network request needs to be made to the video streaming server, it reduces n-1 loads on the streaming server and n-1 repeated traffic on the network bandwidth.

[0039] (3) The “registration-notification” mechanism between UI components and the state manager ensures the synchronization of the images displayed by each UI component;

[0040] (4) Since the state manager creates or destroys the video data processing thread only when the number of registered UI components changes between 0 and 1, the thread management overhead is greatly reduced;

[0041] (5) For users of client application software, the interactivity will become smoother (because the client consumes less resources), and the real-time quality of the video will also be improved (because the bandwidth pressure and server load pressure are reduced). BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 The schematic diagram of the structure of the video code stream processing and rendering system of the present invention;

[0044] Figure 2 It is the structural principle diagram of the existing video code stream processing and rendering system;

[0045] Figure 3 This is the existing video code stream processing and rendering flow chart. DETAILED DESCRIPTION

[0046] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The present invention is further explained below in conjunction with specific implementation modes.

[0048] Example 1

[0049] like Figure 1 As shown, the video code stream processing and rendering system provided in this embodiment includes a video streaming server and a client application software (ie, a client APP).

[0050] The video streaming server is a service program deployed in the cloud and is used to push video data streams to client application software.

[0051] The client application software is connected and communicated with the video streaming server; wherein the client application software includes a UI thread, a state management thread and a video data processing thread;

[0052] The running target of the UI thread is one or more UI components, and its main function is to achieve asynchrony, concurrency, and improve processor utilization. The client application software needs to display the screen content of the same video source in multiple UI components.

[0053] One end of the state management thread is connected to communicate with the UI thread and transfers data, and the other end of the state management thread is connected to communicate with the video data processing thread and transfers data.

[0054] The running target of the state management thread is the state manager; the state management thread is used to run the code logic related to the state manager to ensure that the state manager is in an independent, asynchronous running environment. Among them, the relevant code logic of the state manager refers to the specific code program running in the state management thread, including initiating HTTP requests to the streaming server, creating and destroying video data processing threads, maintaining UI component instances, maintaining image frames sent from the video data processing thread, maintaining the playback status (such as playing, passive, network abnormality, etc.), notifying the UI thread to update the UI component, etc.

[0055] The state manager is used to request data from the video streaming server and maintain the obtained frame stream data and status information of each stage; the state manager also holds the instance or listening callback of the UI component (actively registered with the state manager in the respective UI thread). Whenever there is a change in data or status, the listening callback of the UI component is called to notify the UI thread to update the UI component.

[0056] The state manager is connected to the video streaming server for communication and is used to initiate an HTTP network request to the video streaming server to obtain the video code stream.

[0057] The video data processing thread has one and is used to collect the code stream data pushed from the video streaming server, then decode the received code stream data to obtain each frame of image data and push the image data to the state manager.

[0058] The video code stream processing and rendering method of the present invention is based on the video code stream processing and rendering system of the present invention. For each code stream, the UI component that wants to play its picture is first registered with the corresponding state manager. The state manager records the instances and quantities of these UI components. When the first UI component is registered, it will automatically initiate a code stream request to the server and continue until all UI components are unregistered; the state manager receives data and captures image frames and then decodes them by opening a separate video data processing thread; whenever the state manager receives a new image frame from the video data processing thread, it will notify each registered UI component to update the rendering picture.

[0059] Example 2

[0060] The video code stream processing and rendering method of the present invention specifically comprises the following steps:

[0061] (1) When the client application software needs to display the same video source on multiple UI components, it will first create a corresponding number of UI components in the UI thread (the UI component specifically refers to the component used to display the video screen, such as the video player component);

[0062] (2) When the UI component is created, it will actively register itself with the state manager to monitor the image frame and playback status from the state manager;

[0063] (3) When the first UI component is registered with the state manager, the state manager will initiate a network request to the video streaming server to obtain the video stream and create a video data processing thread to receive and process the stream data;

[0064] (4) After receiving the request command from the state manager, the video streaming server will continuously push the video stream, and the video stream data will eventually flow into the video data processing thread;

[0065] (5) The video data processing thread caches and decodes the received data. Each decoded frame is pushed to the state manager, and the cache is cleared to continue collecting data for the next frame.

[0066] (6) Whenever the state manager obtains a frame of image from the video data processing thread, it updates its own state and then notifies each UI component registered with it;

[0067] (7) When the UI component receives notification from the state manager, it will update and display the latest image frame;

[0068] (8) In the above step (3), if the network request fails, the state manager will mark the state as connection failure and notify each UI component. At this time, the UI component will display the abnormal state in an appropriate manner.

[0069] Among them, the client application software can dynamically increase or decrease the number of UI components; when adding a UI component, the UI component needs to be registered with the state manager, and only when the first UI component is registered with the state manager will the process of step (3) be triggered; whenever a UI component is reduced, the UI component needs to be unregistered from the state manager. When the last UI component is unregistered, the state manager will disconnect from the video streaming server, destroy the video data processing thread, and reset all its own states.

[0070] Example 3

[0071] The working scenarios of the video code stream processing and rendering system of the present invention in actual application are as follows:

[0072] When the client application software needs to display the same video source for multiple UI components, each time a UI component instance is created, it is necessary to first register the UI component instance with the state manager and listen to the image frame data status in the state manager; when the UI component is destroyed, the UI component instance will be unregistered from the state manager and the listening of the image frame data status will be cancelled.

[0073] In the state management, every time a UI component is registered or unregistered, it will check how many UI component instances are currently registered; if the number of registered UI components changes from 0 to 1, the state manager will initiate a network request to the video streaming server and create a video data processing thread for receiving, parsing and decoding video frame image data. The video data processing thread will push each image it parses to the state manager, and then the state manager will notify each registered UI component to update the rendering screen; if the number of registered UI components increases from non-zero, no processing will be performed at this time because the video data processing thread has been started; if the number of registered UI components changes from non-zero to 0, the state manager will close the connection with the video streaming server to save resource consumption, and destroy the video data processing thread to release related resources.

[0074] The present invention can significantly reduce resource waste (including CPU resources, memory resources and bandwidth resources) of client application software and video streaming servers, improve their performance and load, and is suitable for promotion and application.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video code stream processing and rendering system, comprising a video streaming server and a client application software connected and communicating with the video streaming server; the video streaming server is a service program deployed in the cloud and is used to push video data streams to the client application software; The client application software includes a UI thread and a video data processing thread; the running target of the UI thread is one or more UI components; the characteristics are: The client application software also includes a state management thread connected between the UI thread and the video data processing thread; the running target of the state management thread is the state manager; The state management thread is used to run the code logic related to the state manager to ensure that the state manager is in an independent and asynchronous operating environment; The state manager is used to request data from the video streaming server and maintain the obtained frame stream data and state information of each stage; the state manager also holds the monitoring callback of the UI component, and whenever there is a state change, the monitoring callback of the UI component is called to notify the UI thread to update the UI component; The video data processing thread has one, which is used to collect the data pushed from the video streaming server, then decode the received code stream data to obtain each frame of image data and push the image data to the state manager; When the client application software needs to display the same video source for more than one UI component, each time a UI component instance is created, the UI component instance must first be registered with the state manager and the image frame data state in the state manager must be monitored; Whenever the state manager obtains a frame of image from the video data processing thread, the state manager updates its own state and then notifies each UI component registered with it so that each registered UI component updates the rendering screen.

2. The video code stream processing and rendering system according to claim 1, wherein: When the UI component is destroyed, the UI component instance is deregistered from the state manager and the monitoring of the image frame data state is canceled; In state management, the state manager checks how many UI component instances are currently registered each time a UI component is registered or unregistered.

3. The video code stream processing and rendering system according to claim 1, wherein: In state management, when the number of registered UI components changes from 0 to 1, the state manager will initiate a network request to the video streaming server and create a video data processing thread for receiving, parsing and decoding video frame image data. When the video data processing thread parses each image, it will push it to the state manager, and then the state manager will notify each registered UI component to update the rendering screen.

4. The video code stream processing and rendering system according to claim 1, wherein: In the state management, when the number of registered UI components increases from non-zero, the video data processing thread has been started and no processing is performed.

5. The video code stream processing and rendering system according to claim 1, wherein: In state management, when the number of registered UI components changes from non-zero to zero, the state manager closes the connection with the video streaming server to save resource consumption, and destroys the video data processing thread to release related resources.

6. A video code stream processing and rendering method, based on the video code stream processing and rendering system according to any one of claims 1 to 5, characterized in that: For each bitstream, the UI component that wants to play its image must first register with the corresponding state manager. The state manager records the instances and quantities of these UI components. When the first UI component is registered, it will automatically initiate a bitstream request to the server and continue until all UI components are unregistered. The state manager receives data and captures image frames and then decodes them by opening a separate video data processing thread. Whenever the state manager receives a new image frame from the video data processing thread, it will notify each registered UI component to update the rendered image.

7. The video code stream processing and rendering method according to claim 6, characterized in that: The specific steps include: (1) When the client application software needs to display the same video source on multiple UI components, first create a corresponding number of UI components in the UI thread; (2) When the UI component is created, it will actively register itself with the state manager to monitor the image frame and playback status from the state manager; (3) When the first UI component is registered with the state manager, the state manager will initiate a network request to the video streaming server to obtain the video stream and create a video data processing thread to receive and process the stream data; (4) After receiving the request command sent by the state manager, the video streaming server continuously pushes the video stream and the video stream data eventually flows into the video data processing thread; (5) The video data processing thread caches and decodes the received data, pushes each decoded frame to the state manager, and clears the cache to continue collecting data for the next frame; (6) Whenever the state manager obtains a frame of image from the video data processing thread, it updates its own state and then notifies each UI component registered with it; (7) When the UI component receives notification from the state manager, it will update and display the latest image frame; (8) In the above step (3), if the network request fails, the state manager will mark the state as connection failure and notify each UI component. At this time, the UI component will display an abnormal state.

8. The video code stream processing and rendering method according to claim 7, characterized in that: The client application software can dynamically increase the number of UI components; when a UI component is added, all UI components need to be registered with the state manager, and only when the first UI component is registered with the state manager will the process of step (3) be triggered.

9. The video code stream processing and rendering method according to claim 7, characterized in that: The client application software can dynamically reduce the number of UI components; when a UI component is reduced, all UI components need to be deregistered from the state manager. When the last UI component is deregistered, the state manager will disconnect from the video streaming server, destroy the video data processing thread, and reset all its own states.

Images