Video stream data processing method, device, computer equipment, storage medium and program product

Abstract

The present application relates to a method, device, computer equipment, storage medium and program product for processing video stream data. The method comprises: upon receiving a stream mode start signal sent by a cloud host, initializing a rendering engine and a hardware decoder, receiving video stream data sent by the cloud host, calling a hardware decoder to decode the video stream data, obtaining decoded data, rendering the decoded data to an external texture structure through a rendering engine, and rendering the external texture structure to a visible area of ​​a cloud computer client through a rendering engine, and upon receiving a stop signal sent by the cloud host, stopping the process of calling a hardware decoder to decode the video stream data, and releasing decoding resources through the hardware decoder. The use of this method can ensure real-time processing of video stream data, improve decoding efficiency, and reduce resource consumption.

Description

Technical Field

[0001] The present application relates to the field of digital intelligence platform technology, and in particular to a video stream data processing method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] In the computer client cloud desktop screen display, it is mainly realized through the Clink data security protocol. The cloud host server intercepts the virtual machine desktop in real time and forwards it to the corresponding client. After the client receives the video stream data, it will pass it to the GStreamer library to complete the final video decoding and screen rendering display.

[0003] However, since the decoder creation and decoding of the GStreamer library are implemented by reflectively calling the MediaCodec interface of the Java layer through the JNI interface, this reflective calling method will have additional performance overhead, thus affecting the decoding efficiency and overall performance. Summary of the invention

[0004] Based on this, it is necessary to provide a video stream data processing method, device, computer equipment, storage medium and program product that can improve decoding efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for processing video stream data, comprising:

[0006] Upon receiving a streaming mode start signal sent by the cloud host, the rendering engine and the hardware decoder are initialized;

[0007] Receive the video stream data sent by the cloud host, call the hardware decoder to decode the video stream data, and obtain the decoded data;

[0008] Rendering the decoded data to an external texture structure through a rendering engine, and rendering the external texture structure to a visible area of ​​a cloud computer client through the rendering engine;

[0009] When a stop signal is received from the cloud host, the process of calling the hardware decoder to decode the video stream data is stopped, and decoding resources are released through the hardware decoder; the decoding resources include memory resources, processor resources, and graphics rendering resources.

[0010] In one embodiment, the process of initializing the rendering engine includes:

[0011] Get the image buffer associated with the texture view;

[0012] The image buffer is loaded into the native layer through the first interface, and the image buffer is converted into a native window object in the native layer; the native window object is used to provide a graphics drawing window for the rendering engine.

[0013] In one embodiment, the process of initializing the hardware decoder includes:

[0014] Create a decoding thread and a data queue; the decoding thread is used to receive the video stream data sent by the cloud host and insert the video stream data into the data queue;

[0015] In the decoding thread, the second interface is called to create a hardware decoder, and optimization parameters are configured for the hardware decoder; the optimization parameters include delay decoding parameters and color optimization parameters.

[0016] In one embodiment, the step of calling a hardware decoder to process and decode the video stream data to obtain decoded data includes:

[0017] Call the hardware decoder in the decoding thread and read the video stream data from the data queue;

[0018] The video stream data is decoded by a hardware decoder to obtain decoded data.

[0019] In one embodiment, the step of rendering the decoded data to an external texture structure by a rendering engine includes:

[0020] Obtain video frame data according to the decoded data;

[0021] The rendering engine is called to execute drawing commands and render the video frame data into an external texture structure; the external texture structure is used to store the video frame data.

[0022] In one embodiment, the step of rendering the external texture structure to the visible area of ​​the cloud computer client by a rendering engine includes:

[0023] Through the rendering engine, the external texture structure is associated with the native window object;

[0024] The video frame data in the external texture structure is rendered to the visible area of ​​the cloud computer client through the native window object.

[0025] In a second aspect, the present application further provides a video stream data processing device, comprising:

[0026] An initialization module, used to initialize the rendering engine and the hardware decoder when receiving a streaming mode start signal sent by the cloud host;

[0027] The decoding module is used to receive the video stream data sent by the cloud host, call the hardware decoder to decode the video stream data, and obtain decoded data;

[0028] A rendering module, used to render the decoded data to an external texture structure through a rendering engine, and render the external texture structure to a visible area of ​​a cloud computer client through the rendering engine;

[0029] The release module is used to stop calling the hardware decoder to decode the video stream data when receiving a stop signal sent by the cloud host, and release the decoding resources through the hardware decoder; the decoding resources include memory resources, processor resources and graphics rendering resources.

[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method steps of any one of the first aspects are implemented.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements any one of the method steps in the first aspect when the computer program is executed by a processor.

[0032] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any one of the method steps in the first aspect when executed by a processor.

[0033] The above-mentioned video stream data processing method, device, computer equipment, storage medium and program product can ensure that the video stream data can be processed immediately upon receipt by initializing the rendering engine and the hardware decoder, thereby reducing the startup delay. Through the efficient decoding capability of the hardware decoder, it can ensure the real-time processing of the video stream data, improve the decoding efficiency, reduce resource consumption, and thus improve the overall performance. At the same time, by rendering the decoded data directly to the external texture structure and then rendering it to the visible area, unnecessary data conversion and copying are reduced, the utilization of resources is further optimized, and the operating efficiency of the system is improved. In addition, by releasing resources in a timely manner when a stop signal is received, the effective utilization of system resources is ensured, and sufficient resources are provided for other applications or tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0035] FIG1 is an application environment diagram of a method for processing video stream data in one embodiment;

[0036] FIG2 is a schematic diagram of a flow chart of a method for processing video stream data in one embodiment;

[0037] FIG3 is a structural block diagram of a video stream data processing system in one embodiment;

[0038] FIG4 is a timing diagram of a method for processing video stream data in one embodiment;

[0039] FIG5 is a structural block diagram of a video stream data processing device in one embodiment;

[0040] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] With the continuous development of cloud computing technology, cloud computers, as a new computing model, have gradually been widely used. Cloud computers concentrate computing resources, storage resources, and applications on cloud servers. Users can access and use these resources anytime and anywhere through client devices (such as PCs, mobile phones, tablets, etc.), thereby realizing the sharing and efficient use of computing resources. In the client screen of cloud computers, the display of cloud desktop screen is a key technical link. In order to realize the real-time display of cloud desktop screen, cloud computer clients usually need to intercept the virtual machine desktop in real time and forward it to the corresponding client through a specific data transmission protocol.

[0043] In this process, the cloud computer client uses the self-developed Clink data security protocol, and the ClinkServer service of the cloud host completes the real-time interception and forwarding of the virtual machine desktop. After receiving the video stream data, the client needs to decode and render it in order to display the cloud desktop image on the screen. In this process, the GStreamer library, as a free cross-platform multimedia framework, is widely used in the processing of multimedia data. The GStreamer library supports a variety of media processing components, including audio playback, audio and video playback, recording, streaming media and audio editing, etc., with the advantages of cross-platform, high performance and low latency. Therefore, the cloud computer client has built a set of audio and video codec frameworks based on the GStreamer library for different platforms to complete the real-time decoding and rendering of cloud desktop audio and video data on the client.

[0044] However, despite the powerful functions of the GStreamer library, there are some problems on the Android platform. First, the creation and decoding of the decoder of the GStreamer library on the Android platform is implemented through the Java layer MediaCodec interface called by the JNI interface reflection. This reflection call method has additional performance overhead compared to directly using the native interface, which affects the decoding efficiency. Secondly, the decoded data is not written directly to the video memory, but is first written to the memory and then copied from the memory back to the video memory for rendering. This data copying process not only adds additional overhead, but also may cause problems with data format conversion, further affecting the rendering efficiency and picture quality. In addition, the size of the GStreamer library is about 20M, and the Android client only needs to use the audio and video encoding and decoding functions of the library. Other unused functions increase the client package size, which is not conducive to the lightweight application. Finally, the compilation requirements and business logic implementation of GStreamer are relatively complex, and the maintenance cost is high. In the case of a wide variety of Android devices, if there is a device compatibility problem, it is not easy to solve. At the same time, GStreamer cannot give full play to the characteristics of the device decoder, such as configuring low-latency parameters, color standards, and other customized optimizations, which is also not conducive to the widespread application of terminal-core integration scenarios.

[0045] In response to the above problems, the embodiment of the present application proposes a solution based on the native interface NDK MediaCodec and OpenGLES, which migrates the decoding and rendering work from GStreamer to the Android Native layer. It uses the hardware decoder to decode the video, and then writes the decoded data directly into the external texture (OES texture) created by OpenGLE S. Finally, OpenGL ES completes the final rendering and display. It not only gives full play to the performance of the device hardware, improves the decoding efficiency and reduces resource consumption, but also avoids unnecessary data conversion and copying, further optimizes resource utilization, helps to reduce the delay and freeze of video playback, and provides a smoother viewing and operating experience. At the same time, it can make the whole process autonomous and controllable, and the optimization of decoding performance can reach the chip level, laying a good foundation for the wide application of terminal-core integration scenarios.

[0046] Before describing the embodiments of the present application, the concepts that may be involved in the embodiments of the present application are described below:

[0047] ClinkServer: Clink protocol server, used to capture and distribute virtual machine desktop video streams.

[0048] ClinkClient: Clink protocol client, used to receive video streams distributed by ClinkServer.

[0049] GStreamer: An open source multimedia framework used to create various types of media processing components, such as audio and video playback, recording, conversion, and editing. It has the advantages of cross-platform, high performance, and low latency.

[0050] OpenGL ES: Provides a hardware-accelerated 3D graphics programming interface that enables developers to create high-performance graphics applications on these devices.

[0051] OES texture: The full name is: GL_TEXTURE_EXTERNAL_OES, which is a special texture type defined in OpenGLES. It is mainly used to process external image or video data.

[0052] NDK MediaCodec: NDK is part of Android SDK, which allows developers to use C or C++ code in Android applications. NDK supports calling native libraries to perform performance-sensitive tasks, thereby improving application performance. MediaCodec is part of NDK, which provides a set of native interfaces to use the device's hardware codec to decode video or audio data.

[0053] H.264 and H.265: are two widely used codec standards for video compression and transmission.

[0054] TextureView: A subclass of View, which is usually used with OpenGL ES to display dynamic graphics, such as video streams, camera previews, or game renderings. It is especially suitable for scenes that require high-performance graphics display. It contains an image buffer and the core component of the screen display, Surface, which can be used to display or capture image data.

[0055] ANative Window: An ANative Window represents a screen buffer, through which image data can be operated and displayed. It can be regarded as the Native version of Surface provided by NDK.

[0056] JNI interface: The full name is: Java Native Interface, which is a set of native interfaces provided by the Android SDK for interacting with local code (such as C or C++) to achieve application performance optimization, hardware access and other purposes.

[0057] The video stream data processing method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 is used to initialize the rendering engine and the hardware decoder when receiving the streaming mode start signal sent by the server 104, receive the video stream data sent by the server 104, call the hardware decoder to decode the video stream data, obtain the decoded data, render the decoded data to the external texture structure through the rendering engine, and render the external texture structure to the visible area of ​​the terminal 102 through the rendering engine, and stop calling the hardware decoder to decode the video stream data when receiving the stop signal sent by the server 104, and release the decoding resources through the hardware decoder. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0058] In an exemplary embodiment, as shown in FIG2 , a video stream data processing method is provided, which is described by taking the method applied to the terminal 102 in FIG1 as an example, and includes the following steps 202 to 206. Among them:

[0059] S202: Upon receiving a streaming mode start signal sent by the cloud host, the rendering engine and the hardware decoder are initialized.

[0060] Optionally, when the cloud computer client starts and receives a streaming mode start signal from the cloud host, the rendering engine and hardware decoder are initialized first, wherein the rendering engine is a software component used to convert graphic data into visible pixels on the screen. In the Android operating system, the rendering engine is usually part of the Native layer and implements efficient graphics rendering by calling the underlying graphics API. The hardware decoder is a device or software module that performs video decoding tasks through specialized hardware components (such as GPU, specialized decoding chips, etc.). When video data is input into the hardware decoder, the decoder uses the internal hardware acceleration unit to decode the video data. The decoded video data can be directly transmitted to the display device for playback without further processing by the CPU.

[0061] S204: Receive the video stream data sent by the cloud host, call the hardware decoder to decode the video stream data, and obtain decoded data.

[0062] Optionally, the cloud computer client receives video stream data from the cloud host and calls a hardware decoder to decode the data. The decoding process converts the compressed video data into original video frames, and the decoded video frames are rendered to the visible area of ​​the cloud computer client. Compared with the software decoder that decodes and processes video data through the CPU, the hardware decoder decodes through a dedicated hardware component. Since the CPU load is reduced, the power consumption of the cloud computer client can be reduced. In addition, the hardware decoder can usually support multiple video encoding formats and resolutions, which can improve the compatibility of the cloud computer client.

[0063] S206: Rendering the decoded data to an external texture structure through a rendering engine, and rendering the external texture structure to a visible area of ​​the cloud computer client through the rendering engine.

[0064] Optionally, when the hardware decoder successfully decodes a frame of video data, it passes the decoded image data to the rendering engine. The rendering engine receives the decoded image data and renders it into an external texture structure (such as OES texture in OpenGLES). When the image data is rendered into the external texture structure, the rendering engine can further render it into the visible area of ​​the cloud computer client. The rendering engine processes texture mapping, transformation, lighting and shadow effects to generate the final visual output.

[0065] S208: When a stop signal is received from the cloud host, the process of calling the hardware decoder to decode the video stream data is stopped, and decoding resources are released through the hardware decoder; the decoding resources include memory resources, processor resources, and graphics rendering resources.

[0066] Optionally, when the cloud computer client receives a stop signal sent by the cloud host, it stops calling the hardware decoder to decode the video stream data. In addition, the hardware decoder releases all resources related to the decoding process, including memory resources, processor resources, and graphics rendering resources. The release of these resources helps to ensure the efficient use of system resources and avoid potential memory leaks or performance degradation.

[0067] In the above-mentioned video stream data processing method, by initializing the rendering engine and the hardware decoder, it can be ensured that the video stream data can be processed immediately when it is received, reducing the startup delay. Through the efficient decoding capability of the hardware decoder, it can ensure the real-time processing of the video stream data, improve the decoding efficiency, reduce resource consumption, and thus improve the overall performance. At the same time, by rendering the decoded data directly to the external texture structure and then rendering it to the visible area, unnecessary data conversion and copying are reduced, the utilization of resources is further optimized, and the operating efficiency of the system is improved. In addition, by releasing resources in time when a stop signal is received, the effective utilization of system resources is ensured, and sufficient resources are provided for other applications or tasks.

[0068] In an exemplary embodiment, the process of initializing the rendering engine includes: obtaining an image buffer associated with a texture view; loading the image buffer into a native layer through a first interface, and converting the image buffer into a native window object in the native layer; the native window object is used to provide a graphics drawing window for the rendering engine.

[0069] Optionally, in Android application development, TextureView is a view component used to display texture content. This view is associated with an image buffer, which is where graphics data is stored. Obtaining this image buffer is the first step in initializing the rendering engine. All subsequent graphics data will be processed and displayed based on this buffer.

[0070] Optionally, the first interface refers to an interface for cross-layer communication, such as a JNI interface. Through this interface, the image buffer obtained in the Java layer is passed to the native layer (i.e., the Native layer). The native layer usually refers to the underlying code part written in languages ​​such as C or C++, which can interact more directly with the underlying resources and hardware of the system. In the native layer, the received image buffer is converted into a native window object (such as ANative Window). The native window object is an important concept used by the native layer in the Android system to handle window-related operations. It represents a window that can be used for graphics drawing, and provides a specific graphics drawing place that is closely integrated with the underlying system for the subsequent rendering engine. This conversion enables the rendering engine to perform graphics drawing operations in a way that is more in line with the underlying system specifications, avoiding some restrictions that may be imposed in the high-level language layer.

[0071] Specifically, in Android development, TextureView is a commonly used component that allows developers to embed a texture in the UI. This texture can be used as an OpenGL ES rendering target. When using TextureView to display video, it provides an image buffer associated with the OpenGL ES texture. This buffer is where the video frame can be drawn directly after being decoded. After the cloud computer client is started, the Surface of the Java layer TextureView is passed to the Native layer through the JNI interface and converted into an ANativeWindow object so that OpenGL ES can use it, thereby completing the initialization of the OpenGL ES engine and preparing for rendering the decoded video data.

[0072] In this embodiment, by directly processing video decoding and rendering at the native layer, the underlying resources of the device can be accessed more directly, which can improve the performance of the overall application. In addition, richer underlying graphics drawing functions are provided through native window objects, which can perform more sophisticated window operations and graphics processing, thereby improving video rendering effects.

[0073] In an exemplary embodiment, the process of initializing a hardware decoder includes: creating a decoding thread and a data queue; the decoding thread is used to receive video stream data sent by the cloud host and insert the video stream data into the data queue; calling the second interface in the decoding thread to create a hardware decoder, and configuring optimization parameters for the hardware decoder; the optimization parameters include delay decoding parameters and color optimization parameters.

[0074] Optionally, when processing video stream data, a dedicated decoding thread is created to isolate the decoding work from other operations and achieve asynchronous processing. This thread is mainly responsible for receiving the video stream data sent from the cloud host, thereby avoiding interference between the video decoding work and other tasks in the application (such as user interface updates, other background tasks, etc.), and ensuring the independence and stability of the decoding process. The data queue is a container for temporarily storing video stream data. Since the video stream is continuously sent from the cloud host, and the decoding thread may not be able to process each data fragment immediately, a queue is needed to store the data in the order received. When the cloud host sends video stream data, the decoding thread will insert the received video stream data into the data queue, so that subsequent decoding work can be processed in the original order of the video stream, avoiding video playback abnormalities caused by data confusion.

[0075] Optionally, the second interface refers to the MediaCodec interface in the NDK, through which a hardware decoder can be created. A hardware decoder is a component that uses the device's hardware (such as a GPU or a dedicated video decoding chip) to accelerate the video decoding process. After creating the hardware decoder, you need to configure optimization parameters for it. Among them, the delayed decoding parameter is mainly used to control the delay of video decoding. For cloud computer application scenarios with high real-time requirements, by setting this parameter reasonably, the delay in video playback can be reduced. The color optimization parameter is used to improve the color quality of the video screen to ensure that the video color is more vivid and accurate.

[0076] Specifically, when the cloud host ClinkServer starts the streaming mode and sends a startup signaling to the cloud computer client ClinkClient, the ClinkClient module will process the signaling and pass it to the Native layer decoding module. At the Native layer, the decoding module will first create a decoding thread and a data queue. In the decoding thread, the decoding module will call the MediaCodec API provided by NDK to create a hardware accelerated decoder. The hardware decoder is specially designed for video streams in encoding formats such as H.264 or H.265, and can efficiently decode these video stream data.

[0077] In this embodiment, by creating a dedicated decoding thread and data queue, asynchronous processing of video decoding is achieved, the concurrent processing capability of the entire system is improved, and by using a hardware accelerated decoder, the decoding efficiency of video stream data can be improved, thereby improving resource utilization.

[0078] In an exemplary embodiment, calling a hardware decoder to process and decode the video stream data to obtain decoded data includes: calling a hardware decoder in a decoding thread to read the video stream data from a data queue; and decoding the video stream data by the hardware decoder to obtain decoded data.

[0079] Optionally, the decoding thread is the core execution unit of the entire decoding process, and the data queue stores the video stream data waiting to be decoded sent from the cloud host. The decoding thread reads data from this queue in sequence and calls the hardware decoder to decode the video stream data to obtain decoded data. For common video encoding formats (such as H.264, H.265, etc.), the hardware decoder can quickly identify key information in the video stream, and restore the compressed video data to the original image data (decoded data) based on this information and the rules in the encoding standard.

[0080] Specifically, the ClinkClient module, as the receiving end of the cloud computer client, will continuously receive video stream data from the cloud host ClinkServer and insert this data into the decoding queue of the decoding module. In the decoding thread, the decoding module will call the hardware decoder that has been created and configured before. After receiving the video stream data, the hardware decoder will call the MediaCodec interface to decode the video stream data. The decoded video frame or image data will be passed to the subsequent rendering module or display module for further processing and display.

[0081] In this embodiment, by calling the hardware decoder to decode the video stream data in real time in the decoding thread, the smoothness and real-time performance of the video decoding can be ensured. The hardware decoder can increase the decoding speed, reduce CPU resource consumption, and thus improve resource utilization.

[0082] In an exemplary embodiment, the decoded data is rendered to an external texture structure through a rendering engine, including: obtaining video frame data according to the decoded data; calling the rendering engine to execute a drawing command to render the video frame data to an external texture structure; the external texture structure is used to store the video frame data.

[0083] Optionally, after the decoding of the video stream data is completed by the hardware decoder, the decoded data obtained is in the form of raw data containing the video picture information. After that, the rendering engine is called to execute the drawing command to render the video frame data into the external texture structure. Among them, the external texture structure is a special storage area for storing the video frame data decoded and rendered from the video stream. This structure can be regarded as an intermediate storage medium. On the one hand, it receives the video frame data from the rendering engine, and on the other hand, it can conveniently provide these data to the subsequent display link to ensure that the video picture can be displayed correctly.

[0084] Specifically, the hardware decoder uses MediaCodec to decode video data. MediaCodec is an efficient video and audio codec API that can decode video with the support of hardware acceleration. For each successfully decoded frame, MediaCodec does not directly output the decoded frame data as raw data, but directly renders it into an OES texture created by OpenGLES. OES texture is a resource managed by OpenGL ES for storing image data. When MediaCodec renders the decoded frame into the OES texture, OpenGL ES draws the data in the texture onto the final display target.

[0085] In this embodiment, by rendering the decoded data to an external texture structure, unnecessary data conversion and copying can be avoided, so that real-time rendering of data can be achieved, resource utilization can be further optimized, and thus overall processing efficiency can be improved.

[0086] In an exemplary embodiment, an external texture structure is rendered to a visible area of ​​a cloud computer client through a rendering engine, including: associating the external texture structure to a native window object through the rendering engine; and rendering video frame data in the external texture structure to the visible area of ​​the cloud computer client through the native window object.

[0087] Optionally, the rendering engine connects the external texture structure storing the video frame data with the native window object. The external texture structure can be regarded as the "data source" containing the video screen content, and the native window object is the "display window" that ultimately displays these contents. The rendering engine enables the data in the external texture structure to "find" the corresponding display window through specific association operations. The native window object is a window-related object for graphics drawing defined in the system native layer. It has the characteristics of being closely integrated with the underlying operating system and can directly utilize the system's graphics display mechanism. By associating the external texture structure with the native window object, the video frame data is actually introduced into this window environment that can directly interact with the system display, preparing for subsequent video display. Once the association is established, the native window object will use the underlying graphics display function of the system to render the video frame data in the external texture structure to the visible area of ​​the cloud computer client.

[0088] Specifically, after the decoded data is rendered into the OES texture, the rendering thread (ie, OpenGL ES) is notified to start working. After the rendering thread starts working, it first needs to associate the OES texture with the native window object and render the OES texture onto the Surface of TextureView, thereby displaying the video screen in the visible area of ​​the cloud computer client.

[0089] In this embodiment, rendering through native window objects can make full use of the system's underlying graphics display optimization, which can improve rendering efficiency. By directly associating the external texture structure and rendering through the native window object, the intermediate data conversion and transmission links are reduced, avoiding possible performance losses caused by operations such as data copying and format conversion.

[0090] In an exemplary embodiment, as shown in FIG3 , a video stream data processing system is provided, including: a cloud host 100 and a cloud computer client 200. The cloud host 100 includes a ClinkServer module 110 for intercepting and distributing virtual machine desktop video streams. The cloud computer client 200 includes a ClinkClient module 210, a Native layer 220, and a Java layer 230. The ClinkClient module 210 is used to receive the video stream distributed by the ClinkServer module 110. The Native layer 220 includes an NDK MediaCodec module 221 and an OpenGL ES module 222, wherein the NDK MediaCodec module 221 is used to create a hardware decoder, and the OpenGL ES module 222 is used to process graphics rendering. The Java layer 230 includes a TextureView module 231 for displaying dynamic graphics through a Surface component 232,

[0091] Based on this, a video stream data processing method is provided, and the timing diagram of the method is shown in FIG4 , which includes the following steps:

[0092] (1) After the cloud computer client is started, the Surface component 232 of the TextureView module 231 in the Java layer 230 is passed to the Native layer 220 through the JNI interface and converted into an ANativeWindow object to complete the initialization of the OpenGL ES module 222.

[0093] (2) After the ClinkServer module 110 starts the streaming mode, it will send a startup signaling to the ClinkClient module 210. After receiving the signaling, the ClientClient module 210 will transparently pass it to the NDK MediaCodec module 221 of the Native layer 220.

[0094] (3) In the decoding thread, the interface of the NDK MediaCodec module 221 is called to create a decoder and configure various optimization parameters, such as low-latency decoding, color optimization, etc., and then the decoding thread is placed in a blocked state to wait for real-time decoding.

[0095] (4) The ClinkServer module 110 continuously forwards the intercepted virtual machine desktop video stream data to the ClinkClient module 210. After receiving the stream data, the ClinkClient module 210 inserts it into the decoding queue of the decoding module, and then wakes up the decoding thread to start real-time decoding.

[0096] (5) After the NDK MediaCodec module 221 successfully decodes each frame, it directly renders the decoded data into the OES texture created by the OpenGLES module 222, and notifies the rendering thread to render the OES texture onto the Surface component 232 of the TextureView module 231 for display.

[0097] (6) When the ClinkServer module 110 stops the streaming mode, it sends a stop signaling to the ClinkClient module 210. After receiving the signaling, the ClinkClient module 210 transparently transmits it to the NDK MediaCodec module 221 of the Native layer 220. The module stops decoding and releases all resources related to decoding and rendering.

[0098] In this embodiment, by migrating the decoding and rendering work to the AndroidNative layer and using the hardware decoder for video decoding, the hardware performance can be fully utilized, the decoding efficiency can be improved, and the resource consumption can be reduced, thereby improving the overall performance. By directly writing the decoded data into the external texture created by OpenGLES, unnecessary data conversion and copying can be avoided, thereby further optimizing the utilization of resources and improving the operating efficiency of the system.

[0099] In an exemplary embodiment, in order to verify the feasibility and effectiveness of the video stream data processing method provided in the embodiment of the present application, an Android client APK is compiled based on the GStreamer method and the video stream data processing method respectively, and the test results are shown in Table 1 and Table 2.

[0100] Table 1 APK package size comparison

[0101]

[0102] Table 2 Performance test comparison

[0103]

[0104] As can be seen from Table 1, firstly, by comparing the sizes of the two APKs, it can be observed that the APK package size compiled by the video stream data processing method provided in the embodiment of the present application is significantly smaller than the APK package size compiled by the GStreamer method. Then, the two APKs are installed on the same Android thin terminal in sequence. In order to eliminate the terminal differences, multiple terminals of different models and configurations can be tested. By performing the same operation on the cloud desktop, it can be seen from Table 2 that when decoding 1080P video using the video stream data processing method provided in the embodiment of the present application, the frame rate is stabilized at 60fps, and the CPU usage is reduced by 15%~30%, and the color deviation is significantly improved.

[0105] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiment of the present application also provides a video stream data processing device for implementing the video stream data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the one or more video stream data processing device embodiments provided below can refer to the limitations of the video stream data processing method above, and will not be repeated here.

[0107] In an exemplary embodiment, as shown in FIG5 , a video stream data processing device is provided, comprising: an initialization module 10, a decoding module 20, a rendering module 30 and a release module 40, wherein:

[0108] The initialization module 10 is used to initialize the rendering engine and the hardware decoder when receiving a streaming mode start signal sent by the cloud host.

[0109] The decoding module 20 is used to receive the video stream data sent by the cloud host, and call the hardware decoder to decode the video stream data to obtain decoded data.

[0110] The rendering module 30 is used to render the decoded data to an external texture structure through a rendering engine, and render the external texture structure to a visible area of ​​the cloud computer client through the rendering engine.

[0111] The release module 40 is used to stop calling the hardware decoder to decode the video stream data when receiving a stop signal sent by the cloud host, and release decoding resources through the hardware decoder; the decoding resources include memory resources, processor resources and graphics rendering resources.

[0112] In an exemplary embodiment, the initialization module 10 is also used to obtain an image buffer associated with a texture view; load the image buffer to a native layer through a first interface, and convert the image buffer into a native window object in the native layer; the native window object is used to provide a graphics drawing window for a rendering engine.

[0113] In an exemplary embodiment, the initialization module 10 is also used to create a decoding thread and a data queue; the decoding thread is used to receive video stream data sent by the cloud host and insert the video stream data into the data queue; the second interface is called in the decoding thread to create a hardware decoder, and optimization parameters are configured for the hardware decoder; the optimization parameters include delay decoding parameters and color optimization parameters.

[0114] In an exemplary embodiment, the decoding module 20 is further configured to call a hardware decoder in a decoding thread to read video stream data from a data queue; and decode the video stream data through the hardware decoder to obtain decoded data.

[0115] In an exemplary embodiment, the rendering module 30 is further used to obtain video frame data according to the decoded data; call the rendering engine to execute drawing commands to render the video frame data into an external texture structure; and the external texture structure is used to store the video frame data.

[0116] In an exemplary embodiment, the rendering module 30 is also used to associate the external texture structure with the native window object through the rendering engine; and render the video frame data in the external texture structure to the visible area of ​​the cloud computer client through the native window object.

[0117] Each module in the above video stream data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0118] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a video stream data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0119] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: upon receiving a streaming mode start signal sent by a cloud host, initializing a rendering engine and a hardware decoder; receiving video stream data sent by the cloud host, calling a hardware decoder to decode the video stream data to obtain decoded data; rendering the decoded data to an external texture structure through a rendering engine, and rendering the external texture structure to a visible area of ​​a cloud computer client through the rendering engine; upon receiving a stop signal sent by the cloud host, stopping the process of calling the hardware decoder to decode the video stream data, and releasing decoding resources through the hardware decoder; decoding resources include memory resources, processor resources, and graphics rendering resources.

[0121] In one embodiment, the process of initializing a rendering engine involved in executing a computer program by a processor includes: obtaining an image buffer associated with a texture view; loading the image buffer into a native layer through a first interface, and converting the image buffer into a native window object in the native layer; the native window object is used to provide a graphics drawing window for the rendering engine.

[0122] In one embodiment, the process of initializing a hardware decoder involved in executing a computer program by a processor includes: creating a decoding thread and a data queue; the decoding thread is used to receive video stream data sent by a cloud host and insert the video stream data into the data queue; calling a second interface in the decoding thread to create a hardware decoder, and configuring optimization parameters for the hardware decoder; the optimization parameters include delay decoding parameters and color optimization parameters.

[0123] In one embodiment, the processor executes a computer program involving calling a hardware decoder to decode a video stream data process to obtain decoded data, including: calling a hardware decoder in a decoding thread to read video stream data from a data queue; decoding the video stream data through the hardware decoder to obtain decoded data.

[0124] In one embodiment, the processor executes a computer program involving rendering decoded data to an external texture structure through a rendering engine, including: obtaining video frame data based on the decoded data; calling the rendering engine to execute a drawing command to render the video frame data to an external texture structure; the external texture structure is used to store the video frame data.

[0125] In one embodiment, when a processor executes a computer program, the process of rendering an external texture structure to a visible area of ​​a cloud computer client through a rendering engine includes: associating the external texture structure to a native window object through the rendering engine; and rendering video frame data in the external texture structure to the visible area of ​​the cloud computer client through the native window object.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: upon receiving a streaming mode start signal sent by a cloud host, a rendering engine and a hardware decoder are initialized; video stream data sent by the cloud host are received, and a hardware decoder is called to decode the video stream data to obtain decoded data; the decoded data is rendered to an external texture structure by the rendering engine, and the external texture structure is rendered to a visible area of ​​a cloud computer client by the rendering engine; upon receiving a stop signal sent by the cloud host, the process of calling the hardware decoder to decode the video stream data is stopped, and decoding resources are released by the hardware decoder; decoding resources include memory resources, processor resources, and graphics rendering resources.

[0127] In one embodiment, the process of initializing a rendering engine involved in executing a computer program by a processor includes: obtaining an image buffer associated with a texture view; loading the image buffer into a native layer through a first interface, and converting the image buffer into a native window object in the native layer; the native window object is used to provide a graphics drawing window for the rendering engine.

[0128] In one embodiment, the process of initializing a hardware decoder involved in executing a computer program by a processor includes: creating a decoding thread and a data queue; the decoding thread is used to receive video stream data sent by a cloud host and insert the video stream data into the data queue; calling a second interface in the decoding thread to create a hardware decoder, and configuring optimization parameters for the hardware decoder; the optimization parameters include delay decoding parameters and color optimization parameters.

[0129] In one embodiment, when a computer program is executed by a processor, the process of calling a hardware decoder to decode a video stream data process to obtain decoded data includes: calling a hardware decoder in a decoding thread to read video stream data from a data queue; decoding the video stream data through the hardware decoder to obtain decoded data.

[0130] In one embodiment, when a computer program is executed by a processor, rendering decoded data to an external texture structure through a rendering engine includes: obtaining video frame data based on the decoded data; calling the rendering engine to execute a drawing command to render the video frame data to an external texture structure; and the external texture structure is used to store the video frame data.

[0131] In one embodiment, when a computer program is executed by a processor, the computer program involves rendering an external texture structure to a visible area of ​​a cloud computer client through a rendering engine, including: associating the external texture structure to a native window object through the rendering engine; and rendering the video frame data in the external texture structure to the visible area of ​​the cloud computer client through the native window object.

[0132] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: upon receiving a streaming mode start signal sent by a cloud host, initializing a rendering engine and a hardware decoder; receiving video stream data sent by the cloud host, calling a hardware decoder to decode the video stream data to obtain decoded data; rendering the decoded data to an external texture structure through a rendering engine, and rendering the external texture structure to a visible area of ​​a cloud computer client through the rendering engine; upon receiving a stop signal sent by the cloud host, stopping the process of calling the hardware decoder to decode the video stream data, and releasing decoding resources through the hardware decoder; decoding resources include memory resources, processor resources, and graphics rendering resources.

[0133] In one embodiment, the process of initializing a rendering engine involved in executing a computer program by a processor includes: obtaining an image buffer associated with a texture view; loading the image buffer into a native layer through a first interface, and converting the image buffer into a native window object in the native layer; the native window object is used to provide a graphics drawing window for the rendering engine.

[0134] In one embodiment, the process of initializing a hardware decoder involved in executing a computer program by a processor includes: creating a decoding thread and a data queue; the decoding thread is used to receive video stream data sent by a cloud host and insert the video stream data into the data queue; calling a second interface in the decoding thread to create a hardware decoder, and configuring optimization parameters for the hardware decoder; the optimization parameters include delay decoding parameters and color optimization parameters.

[0135] In one embodiment, when a computer program is executed by a processor, the process of calling a hardware decoder to decode a video stream data process to obtain decoded data includes: calling a hardware decoder in a decoding thread to read video stream data from a data queue; decoding the video stream data through the hardware decoder to obtain decoded data.

[0136] In one embodiment, when a computer program is executed by a processor, rendering decoded data to an external texture structure through a rendering engine includes: obtaining video frame data based on the decoded data; calling the rendering engine to execute a drawing command to render the video frame data to an external texture structure; and the external texture structure is used to store the video frame data.

[0137] In one embodiment, when a computer program is executed by a processor, the computer program involves rendering an external texture structure to a visible area of ​​a cloud computer client through a rendering engine, including: associating the external texture structure to a native window object through the rendering engine; and rendering the video frame data in the external texture structure to the visible area of ​​the cloud computer client through the native window object.

[0138] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0140] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video stream data processing method, characterized in that: Applied to a cloud computer client; the method comprises: Upon receiving a streaming mode start signal sent by the cloud host, obtaining an image buffer associated with the texture view; Loading the image buffer to a native layer through a first interface, and converting the image buffer into a native window object in the native layer; the native window object is used to provide a graphics drawing window for a rendering engine; Creating a decoding thread and a data queue; the decoding thread is used to receive the video stream data sent by the cloud host and insert the video stream data into the data queue; In the decoding thread, the second interface is called to create a hardware decoder, and optimization parameters are configured for the hardware decoder; the optimization parameters include delay decoding parameters and color optimization parameters; Receiving the video stream data sent by the cloud host, calling the hardware decoder to decode the video stream data to obtain decoded data; Rendering the decoded data to an external texture structure through the rendering engine, and rendering the external texture structure to a visible area of ​​the cloud computer client through the rendering engine; When a stop signal is received from the cloud host, the process of calling the hardware decoder to decode the video stream data is stopped, and decoding resources are released through the hardware decoder; the decoding resources include memory resources, processor resources and graphics rendering resources.

2. The method according to claim 1, characterized in that The calling of the hardware decoder to process and decode the video stream data to obtain decoded data includes: Calling the hardware decoder in the decoding thread to read video stream data from the data queue; The video stream data is decoded by the hardware decoder to obtain decoded data.

3. The method according to claim 1, characterized in that The rendering engine rendering the decoded data to an external texture structure includes: Acquire video frame data according to the decoded data; The rendering engine is called to execute a drawing command to render the video frame data into an external texture structure; the external texture structure is used to store the video frame data.

4. The method according to claim 1, characterized in that The rendering of the external texture structure to the visible area of ​​the cloud computer client by the rendering engine includes: Associating the external texture structure with the native window object via the rendering engine; The video frame data in the external texture structure is rendered to the visible area of ​​the cloud computer client through the native window object.

5. A video stream data processing device, characterized in that: The device comprises: an initialization module, for obtaining an image buffer associated with a texture view upon receiving a streaming mode start signal sent by a cloud host; Loading the image buffer to a native layer through a first interface, and converting the image buffer into a native window object in the native layer; the native window object is used to provide a graphics drawing window for a rendering engine; Creating a decoding thread and a data queue; the decoding thread is used to receive the video stream data sent by the cloud host and insert the video stream data into the data queue; In the decoding thread, the second interface is called to create a hardware decoder, and optimization parameters are configured for the hardware decoder; the optimization parameters include delay decoding parameters and color optimization parameters; A decoding module, used for receiving the video stream data sent by the cloud host, calling the hardware decoder to decode the video stream data to obtain decoded data; A rendering module, used to render the decoded data to an external texture structure through the rendering engine, and render the external texture structure to a visible area of ​​the cloud computer client through the rendering engine; A release module is used to stop the process of calling the hardware decoder to decode the video stream data when a stop signal is received from the cloud host, and release decoding resources through the hardware decoder; the decoding resources include memory resources, processor resources and graphics rendering resources.

6. The device according to claim 5, characterized in that The decoding module is also used to call the hardware decoder in the decoding thread to read the video stream data from the data queue; and decode the video stream data through the hardware decoder to obtain decoded data.

7. The device according to claim 5, characterized in that The rendering module is also used to obtain video frame data according to the decoded data; call the rendering engine to execute a drawing command to render the video frame data into an external texture structure; and the external texture structure is used to store the video frame data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images