Display device and multi-channel video display method

Abstract

The invention provides a display device and a multi-channel video display method. When the display device plays the first video, in response to the play starting instruction of the second video, whether the hardware display capability of the hardware processor supports multi-path display or not is recognized firstly, and when the hardware display capability does not support multi-path display, the software-assisted rendering mode can be started to continue to process the video stream of the first video through the hardware processor. And the video stream of the second video is processed in a software manner through the image rendering engine, so that the video streams of the multiple videos can be processed at the same time, and the video pictures of the two videos can be displayed at the same time, and the display effect of a multi-window mode can still be realized under the condition that the hardware display capability of the display equipment does not support multi-path display.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a method for displaying multiple video streams. Background Technology

[0002] Display devices refer to terminal devices capable of outputting specific display images, such as smart TVs, mobile terminals, smart advertising screens, and projectors. Taking smart TVs as an example, smart TVs are television products based on Internet application technologies, possessing open operating systems and chips, and having open application platforms. They enable two-way human-computer interaction and integrate multiple functions such as audio-visual, entertainment, and data to meet diverse and personalized user needs.

[0003] When displaying video, a display device needs to perform decoding and rendering processes on the video stream, which are typically handled by the device's hardware. If the display device's hardware supports multi-channel display, meaning it can process multiple video streams through its hardware processor, it can achieve a multi-window mode that displays multiple videos simultaneously. However, if the display device's hardware only supports single-channel display, meaning it can only process one video stream at a time through its hardware processor, it can only display one video at a time and cannot support multi-window mode.

[0004] Therefore, display devices whose hardware display capabilities do not support multi-channel display will not be able to meet users' needs for watching videos in multi-window mode. Summary of the Invention

[0005] This application provides a display device and a method for displaying multiple video streams, which can solve the problem that display devices whose hardware display capabilities do not support multi-channel display cannot display multiple video streams simultaneously.

[0006] In a first aspect, this application provides a display device, comprising:

[0007] monitor;

[0008] The controller is configured as follows:

[0009] In response to the start playback command of the first video, the video stream of the first video is processed by the hardware processor to obtain the video frame of the first video;

[0010] Control the display to show the video frame of the first video on the video layer;

[0011] In response to the start command of the second video, the system reads the currently set first multi-window mode and the hardware display capabilities of the hardware processor.

[0012] If the hardware display capability does not support multi-channel display, start the software-assisted rendering mode;

[0013] In the software-assisted rendering mode, the hardware processor processes the video stream of the first video to obtain the first video frame of the first video in the first multi-window mode; and the image rendering engine is invoked to process the video stream of the second video to obtain the second video frame of the second video in the first multi-window mode.

[0014] The display is controlled to display the first video image on the video layer and the second video image on the graphics layer, wherein the graphics layer is located above the video layer.

[0015] The above technical solution has the following beneficial effects or advantages:

[0016] When the display device is playing the first video, in response to the start command of the second video, it first identifies whether the hardware display capability of the hardware processor supports multi-channel display. If the hardware display capability does not support multi-channel display, it can start the software-assisted rendering mode to continue processing the video stream of the first video through the hardware processor, and process the video stream of the second video through the image rendering engine in a software manner. In this way, multiple video streams can be processed simultaneously, and the video images of two videos can be displayed at the same time, so that the multi-window display effect can still be achieved even when the hardware display capability of the display device does not support multi-channel display.

[0017] In some embodiments of this application, in the software-assisted rendering mode, the controller processes the video stream of the first video through the hardware processor to obtain the first video frame of the first video in the first multi-window mode, specifically configured as follows:

[0018] The first split-screen middleware corresponding to the hardware processor calculates the first size and first position of the first window used to display the first video according to the first multi-window mode;

[0019] The hardware processor creates the first window based on the first size and the first position.

[0020] Based on the first window, the video stream of the first video is processed to obtain the first video frame of the first video in the first multi-window mode, wherein the first video frame is matched with the first window.

[0021] The above technical solution has the following beneficial effects or advantages:

[0022] The display device is configured with a corresponding first split-screen middleware for the hardware processor. In the first multi-window mode, the first split-screen middleware can accurately calculate the size and position of the first window, so that the display effect of the first video can match the first multi-window mode.

[0023] In some embodiments of this application, in the software-assisted rendering mode, the controller invokes an image rendering engine to process the video stream of the second video through the image rendering engine, thereby obtaining the second video frame of the second video in the first multi-window mode, specifically configured as follows:

[0024] The second split-screen middleware corresponding to the image rendering engine calculates the second size and second position of the second window used to display the second video according to the first multi-window mode;

[0025] The second window is created by the image rendering engine based on the second size and the second position;

[0026] Based on the second window, the video stream of the second video is processed to obtain the second video frame of the second video in the first multi-window mode, wherein the second video frame is matched with the second window.

[0027] The above technical solution has the following beneficial effects or advantages:

[0028] The display device is configured with a corresponding second split-screen middleware for the image rendering engine. In the first multi-window mode, the second split-screen middleware can accurately calculate the size and position of the second window, so that the display effect of the second video can match the first multi-window mode.

[0029] In some embodiments of this application, in the software-assisted rendering mode, after the controller invokes the image rendering engine to process the video stream of the second video and obtains the second video frame of the second video in the first multi-window mode, it is further configured to:

[0030] The image rendering engine determines whether the resolution of the second video frame is higher than a resolution threshold.

[0031] If the resolution of the second video frame is higher than the resolution threshold, the second video frame is subjected to image quality compression processing according to the resolution threshold.

[0032] The above technical solution has the following beneficial effects or advantages:

[0033] The display device processes the second video stream through its image rendering engine. After obtaining the second video frame, it compares the resolution of the second video frame with a resolution threshold to determine whether the display device supports the resolution of the second video. If the resolution of the second video frame exceeds the resolution threshold, meaning the display device does not support the resolution of the second video, the second video frame undergoes image quality compression to adapt to the resolution supported by the display device, reducing system bandwidth usage and preventing other abnormal situations.

[0034] In some embodiments of this application, the controller, in response to a start-up command for the second video, reads the hardware display capabilities of the hardware processor and is specifically configured as follows:

[0035] In response to the start command of the second video, the hardware display capabilities of the hardware processor are read from the local configuration file; or,

[0036] In response to the start-up command of the second video, a configuration request is sent to the server;

[0037] The server receives feedback from the hardware processor regarding its hardware display capabilities based on the configuration request.

[0038] The above technical solution has the following beneficial effects or advantages:

[0039] In response to the start command of the second video, the display device can request the hardware display capabilities of the hardware processor from the local configuration file or from the server. This ensures that the hardware display capabilities are accurately determined, so that if the hardware display capabilities support multi-channel display, the hardware display capabilities can be fully utilized to improve the quality and efficiency of multi-channel display. It also ensures that if the hardware display capabilities do not support multi-channel display, the software-assisted rendering mode can be started in a timely manner to assist in the implementation of multi-window mode.

[0040] In some embodiments of this application, in the software-assisted rendering mode, the controller invokes an image rendering engine to process the video stream of the second video through the image rendering engine, thereby obtaining the second video frame of the second video in the first multi-window mode, specifically configured as follows:

[0041] Read the software rendering capabilities of the display device;

[0042] Based on the software rendering capabilities, determine at least one image rendering engine supported by the display device;

[0043] Call one of the at least one image rendering engines to process the video stream of the second video through the image rendering engine, and obtain the second video frame of the second video in the first multi-window mode.

[0044] The above technical solution has the following beneficial effects or advantages:

[0045] After the display device starts the software-assisted rendering mode, it first reads the software rendering capabilities of the display device to accurately determine the image rendering engines that the display device can support. Then, it can call the image rendering engines supported by the display device to process the video stream of the second video, ensuring the effective display of the second video.

[0046] In some embodiments of this application, the controller is further configured to:

[0047] If the hardware display capability supports multi-channel display, according to the first multi-window mode, the hardware processor processes the video stream of the first video to obtain the first video frame of the first video in the first multi-window mode; and the hardware processor processes the video stream of the second video to obtain the second video frame of the second video in the first multi-window mode.

[0048] The first video frame and the second video frame are combined to obtain a combined video frame, which includes the first video frame and the second video frame.

[0049] The display is controlled to show the synthesized video image on the video layer.

[0050] The above technical solution has the following beneficial effects or advantages:

[0051] When the display device's hardware display capabilities support multi-channel display, it uses a hardware processor to simultaneously process the video streams of the first and second videos. This fully utilizes the hardware display capabilities and improves the quality and efficiency of multi-channel display. After obtaining the video frames from the two videos through the hardware processor, the display device can combine the two video frames into one video stream for display, achieving a multi-window display effect.

[0052] In some embodiments of this application, after controlling the display to display the composited video image on the video layer, the controller is further configured to:

[0053] In response to the start-up command of the third video, if the total number of videos to be displayed exceeds the number of channels supported by the hardware display capability, the software-assisted rendering mode is activated, and the currently set first multi-window mode is read.

[0054] In the software-assisted rendering mode, the hardware processor processes the video stream of the first video to obtain the first video frame of the first video in the first multi-window mode; the hardware processor processes the video stream of the second video to obtain the second video frame of the second video in the first multi-window mode; and the image rendering engine is invoked to process the video stream of the third video to obtain the third video frame of the third video in the first multi-window mode.

[0055] The first video frame and the second video frame are combined to obtain a combined video frame, which includes the first video frame and the second video frame.

[0056] The display is controlled to show the composited video image on the video layer and the third video image on the graphics layer.

[0057] The above technical solution has the following beneficial effects or advantages:

[0058] When a display device's hardware supports multi-channel display, it can process multiple video streams simultaneously through its hardware processor to achieve simultaneous display of multiple videos. In response to a third video playback command, if the number of videos to be displayed exceeds the number of streams the hardware processor can handle simultaneously (i.e., exceeding the hardware display capacity), the display device can activate a software-assisted rendering mode to expand its multi-channel display capabilities and enable the simultaneous display of even more videos.

[0059] Secondly, embodiments of this application also provide a method for displaying multiple video streams, applied to a display device, the method comprising:

[0060] In response to the start playback command of the first video, the video stream of the first video is processed by the hardware processor to obtain the video frame of the first video;

[0061] Display the video frame of the first video in the video layer;

[0062] In response to the start command of the second video, the system reads the currently set first multi-window mode and the hardware display capabilities of the hardware processor.

[0063] If the hardware display capability does not support multi-channel display, start the software-assisted rendering mode;

[0064] In the software-assisted rendering mode, the hardware processor processes the video stream of the first video to obtain the first video frame of the first video in the first multi-window mode; and the image rendering engine is invoked to process the video stream of the second video to obtain the second video frame of the second video in the first multi-window mode.

[0065] The first video frame is displayed in the video layer, and the second video frame is displayed in the graphics layer, wherein the graphics layer is located above the video layer.

[0066] The above technical solution has the following beneficial effects or advantages:

[0067] When the display device is playing the first video, in response to the start command of the second video, it first identifies whether the hardware display capability of the hardware processor supports multi-channel display. If the hardware display capability does not support multi-channel display, it can start the software-assisted rendering mode to continue processing the video stream of the first video through the hardware processor, and process the video stream of the second video through the image rendering engine in a software manner. In this way, multiple video streams can be processed simultaneously, and the video images of two videos can be displayed at the same time, so that the multi-window display effect can still be achieved even when the hardware display capability of the display device does not support multi-channel display.

[0068] In some embodiments of this application, the method further includes:

[0069] If the hardware display capability supports multi-channel display, according to the first multi-window mode, the hardware processor processes the video stream of the first video to obtain the first video frame of the first video in the first multi-window mode; and the hardware processor processes the video stream of the second video to obtain the second video frame of the second video in the first multi-window mode.

[0070] The first video frame and the second video frame are combined to obtain a combined video frame, which includes the first video frame and the second video frame.

[0071] The synthesized video image is displayed on the video layer.

[0072] The above technical solution has the following beneficial effects or advantages:

[0073] When the display device's hardware display capabilities support multi-channel display, it uses a hardware processor to simultaneously process the video streams of the first and second videos. This fully utilizes the hardware display capabilities and improves the quality and efficiency of multi-channel display. After obtaining the video frames from the two videos through the hardware processor, the display device can combine the two video frames into one video stream for display, achieving a multi-window display effect. Attached Figure Description

[0074] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a schematic diagram illustrating the operation scenario between the display device 200 and the control device 100 in an embodiment of this application;

[0076] Figure 2 This is a hardware configuration block diagram of the display device 200 in this embodiment of the application;

[0077] Figure 3 This is a diagram showing the operating system configuration of the display device 200 in this embodiment of the application;

[0078] Figure 4 This is a flowchart illustrating the display device 200 displaying multiple video streams in an embodiment of this application.

[0079] Figure 5 This is a timing diagram of the display device 200 displaying multiple video streams in an embodiment of this application;

[0080] Figures 6A-6B A schematic diagram illustrating the interface changes of the display device 200 displaying multiple video streams provided in this embodiment of the application;

[0081] Figure 7 This is a flowchart illustrating how the display device 200 reads the hardware display capability in an embodiment of this application.

[0082] Figure 8 This is a flowchart illustrating how the display device 200 calls the image rendering engine in an embodiment of this application.

[0083] Figure 9 This is a flowchart illustrating how the display device 200 processes the video stream of the first video through a hardware processor, as described in this embodiment of the application.

[0084] Figure 10 This is a flowchart illustrating how the display device 200 processes the video stream of the second video through an image rendering engine, as described in this application embodiment.

[0085] Figure 11 This is a flowchart illustrating how the display device 200 processes the second video frame in an embodiment of this application.

[0086] Figure 12 This is a flowchart illustrating how the display device 200 processes multiple video streams via a hardware processor, as described in this application embodiment.

[0087] Figure 13 This is a flowchart illustrating the display device 200 displaying three video streams in an embodiment of this application.

[0088] Figure 14 This is a schematic diagram of the display device 200 displaying three video streams in an embodiment of this application. Detailed Implementation

[0089] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0090] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0091] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0092] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0093] In this application embodiment, "display device" refers to a device with screen display and data processing capabilities. For example, display devices include, but are not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, and augmented reality devices.

[0094] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1As shown, users can operate the display device 200 via touch, voice, mobile terminal 300, and control device. For example, the control device can be a remote control 100, a stylus, a gamepad, etc.

[0095] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0096] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0097] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0098] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a user input interface 280, a memory, and a power supply.

[0099] In some embodiments, the communication device 220 is a component for communicating with external devices or the server 400 according to various communication protocol types. The display device 200 may be equipped with multiple communication devices 220 depending on the supported communication methods. The communication devices 220 can enable the display device 200 to communicate with the external devices or the server 400 via wireless or wired connections.

[0100] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0101] In some embodiments, device interface 240 is used to connect to an external device.

[0102] In some embodiments, the controller 250 is used to control the overall operation of the display device 200. The controller 250 may include at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device 200 and responds to user operations through various software control programs stored in memory.

[0103] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0104] In some embodiments, the display 260 is used to receive and display image signals output from the controller 250. The display 260 may include display function components for presenting images and driving components for driving image display.

[0105] In some embodiments, a user can input user commands on a graphical user interface (GUI) displayed on a display 260, and a user input interface 280 can receive user commands through the GUI.

[0106] In some embodiments, the audio output device 270 may be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200.

[0107] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0108] In some embodiments, to enable user interaction, the display device 200 may run an operating system. An operating system is a computer program that manages and controls the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running applications. The operating system also allows users to interact with the display device 200.

[0109] The operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices 200.

[0110] like Figure 3 As shown, Figure 3The diagram below illustrates the software configuration of a display device according to some embodiments of this application. In some embodiments, the system of the display device 200 can be divided into three layers, from top to bottom: the application layer, the middleware layer, and the hardware layer.

[0111] The application layer mainly includes TV applications and application frameworks. The applications are mainly browser-based applications, such as HTML5 apps, and native apps.

[0112] An application framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, and the user interface for these functions (toolbar, status bar, menu, dialog box).

[0113] Native apps can support online or offline access, push notifications, or access to local resources.

[0114] The middleware layer includes various television protocols, multimedia protocols, and system components. Middleware can use the basic services (functions) provided by system software to connect different parts of application systems or different applications on the network, achieving resource sharing and function sharing.

[0115] The hardware layer mainly includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all TV chips, with the specific logic implemented by each chip. The drivers mainly include: audio drivers, display drivers, Bluetooth drivers, camera drivers, Wi-Fi drivers, USB drivers, HDMI drivers, sensor drivers (such as fingerprint sensors, temperature sensors, pressure sensors, etc.), and power drivers.

[0116] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0117] When displaying video, the display device 200 needs to perform decoding and rendering processes on the video stream. This processing is typically handled by the display device 200's hardware processor. If the display device 200's hardware display capability supports multi-channel display, meaning it can process multiple video streams simultaneously, a multi-window mode can be achieved. However, if the display device 200's hardware display capability only supports single-channel display, meaning it can only process one video stream at a time, then only one video can be displayed at a time, and multi-window mode is not supported.

[0118] Therefore, for display devices 200 whose hardware display capabilities do not support multi-channel display, it will be unable to meet users' needs for video viewing in multi-window mode.

[0119] To address the aforementioned issues, the display device 200 provided in this application embodiment can be configured with a software-assisted rendering mode. In this mode, the display device 200 can process one video stream through software. Thus, by processing multiple video streams simultaneously using both hardware processors and software, the effect of processing multiple video streams at the same time can be achieved, thereby realizing a multi-window display effect.

[0120] Figure 4 This is a flowchart illustrating the display device 200 displaying multiple video streams in an embodiment of this application. The specific steps are as follows:

[0121] In step S401, in response to the start-up command of the first video, the video stream of the first video is processed by the hardware processor to obtain the video frame of the first video.

[0122] When a user needs to watch the first video, they input a start command for the first video to the display device 200. The first video can be a video provided by the native application of the display device 200, such as a live program, or a video provided by a third-party application, such as a web video.

[0123] When the display device 200 starts playing the first video, there is only the first video to be displayed, meaning that the display device 200 only needs to process one video stream. The hardware display capability of the display device 200 can support one-stream display; therefore, the display device 200 can process the video stream of the first video through its hardware processor to obtain the video image of the first video.

[0124] In some embodiments, when the display device 200 processes the video stream of the first video through a hardware processor, it creates a corresponding first window for displaying the video frame of the first video. After creating the first window, the hardware processor processes the video stream of the first video based on the window parameters of the first window to match the resulting video frame with the first window. The window parameters of the first window may include its size and position on the display screen.

[0125] In some embodiments, when the display device 200 displays only one video at a time, it is configured to preferentially use a full-screen state. When the display device 200 displays a video within a window, it preferentially uses a fill-in display mode, that is, the video frame matches the size of the corresponding window, and the video frame within the corresponding window is equivalent to a full-screen state. Based on this, when the hardware processor processes the video stream of the first video, the first window it creates is a full-screen window, that is, the size of the first window matches the size of the display area on the screen. The video frame of the first video processed by the hardware processor based on the first window can fill the first window, thereby achieving the display effect of full-screen display of the first video.

[0126] Figure 5 This is a timing diagram of the display device 200 displaying multiple video streams in an embodiment of this application.

[0127] Combination Figure 5 After receiving the start command for the first video, the display device 200 responds to the start command by having its hardware processor process the video stream of the first video to obtain the video image of the first video.

[0128] Step S402: Display the video frame of the first video in the video layer.

[0129] The video footage processed by the hardware processor will be displayed on the video layer.

[0130] In some embodiments, the display device 200 displays the video frame of the first video in full screen on the video layer.

[0131] Combination Figure 5 After the hardware processor processes and obtains the video frame of the first video, it transmits the corresponding video frame to the display controller. The display controller then performs rendering and other processing on the corresponding video frame in the video layer to display the video frame of the first video in the video layer.

[0132] Figures 6A-6B This is a schematic diagram illustrating the interface changes of a display device 200 displaying multiple video streams, as provided in an embodiment of this application. Figure 6A As shown, in response to the start-up command of the first video, the display device 200 displays the video frame of the first video in full screen on the video layer.

[0133] In step S403, in response to the start-up command of the second video, the first multi-window mode currently set is read, and the hardware display capability of the hardware processor is read.

[0134] When the display device 200 is displaying the first video, if the user wants to watch the second video at the same time, the user can input a start playback command for the second video into the display device 200.

[0135] In some embodiments, the second video may be a video provided by a native application of the display device 200, such as a live broadcast, or a video provided by a third-party application, such as a web video.

[0136] In some embodiments, the display device 200 may be configured to provide at least one multi-window mode. This multi-window mode may include picture-in-picture (PIP) mode, split-screen mode, free mode, etc. In different multi-window modes, the size and position of the different windows differ, and correspondingly, the display effect of each video frame also differs.

[0137] In some embodiments, the display device 200 may be preset with a default multi-window mode. In one example, the default multi-window mode is a split-screen mode.

[0138] In some embodiments, the multi-window mode can be preset by the user. In one example, the user can set the multi-window mode before instructing the start of the second video, or the user can indicate the multi-window mode to be used when instructing the start of the second video, in which case the start instruction for the second video also includes the multi-window mode indicated by the user.

[0139] In response to the start command of the second video, the display device 200 reads the currently set first multi-window mode and the hardware display capability of the hardware processor, thereby determining whether the hardware processor can support multi-channel display.

[0140] In some embodiments, the display device 200 is configured with a multiplexing management middleware, through which the display device 200 can read the hardware display capabilities and the currently set first multi-window mode.

[0141] Combination Figure 5 In response to the start command of the second video, the display device 200 can read the hardware display capabilities and the currently set first multi-window mode through the multiplexing management middleware.

[0142] In some embodiments, the display device 200 may read the hardware display capabilities of the hardware processor from a local configuration file.

[0143] In some embodiments, the display device 200 may request the hardware display capabilities of the hardware processor from the server 400.

[0144] In some embodiments, the display device 200 may read the hardware display capabilities by sequentially reading the hardware display capabilities from a local configuration file and requesting the hardware display capabilities from the server 200.

[0145] In one example Figure 7 This is a flowchart illustrating how the display device 200 reads the hardware display capability in this embodiment of the application. The specific steps are as follows:

[0146] In step S701, in response to the start playback command of the second video, the hardware display capabilities of the hardware processor are read from the local configuration file.

[0147] The display device 200 can be configured to prioritize reading the hardware display capabilities of the hardware processor from a local configuration file. Once the hardware display capabilities of the hardware processor are read, the subsequent video stream processing flow can be directly executed based on the read result.

[0148] Step S702: If the hardware display capability is not detected, a configuration request is sent to the server.

[0149] If the display device 200 fails to read the hardware display capability from the local configuration file, it can send a configuration request to the server 400 to obtain the hardware display capability of the hardware processor from the server 400.

[0150] In some embodiments, the configuration request may carry the device model of the display device 200, the model of the hardware processor, etc., which can be used to find parameters / information of the hardware display capabilities of the hardware processor.

[0151] Step S703: Receive the hardware display capability of the hardware processor from the server based on the configuration request.

[0152] After receiving a configuration request from the display device 200, the server 400 can respond to the configuration request by obtaining the hardware display capabilities corresponding to the hardware processor configured by the display device 200, and encapsulate it as feedback information and send it to the display device 200 so that the display device 200 can know the hardware display capabilities corresponding to the hardware processor.

[0153] Therefore, the display device 200 can accurately determine the hardware display capability, so as to ensure that when the hardware display capability supports multi-channel display, the hardware display capability can be fully utilized to improve the quality and efficiency of multi-channel display, and to ensure that when the hardware display capability does not support multi-channel display, the software-assisted rendering mode can be started in a timely manner to assist in realizing the multi-window mode.

[0154] Step S404: If the hardware display capability does not support multi-channel display, start the software-assisted rendering mode.

[0155] If the hardware display capability does not support multi-channel display, it means that the display device 200 cannot process multiple video streams and achieve a multi-window display effect solely through the hardware processor. In this case, the display device 200 will control the activation of software-assisted rendering mode.

[0156] Combination Figure 5 When the hardware display capability does not support multi-channel display, the multi-channel management middleware controls the start of the software-assisted rendering mode.

[0157] Step S405: In software-assisted rendering mode, the video stream of the first video is processed by the hardware processor to obtain the first video frame of the first video in the first multi-window mode; and the image rendering engine is invoked to process the video stream of the second video to obtain the second video frame of the second video in the first multi-window mode.

[0158] In software-assisted rendering mode, the hardware processor will continue to process the video stream of the first video (which can also be called the main video in the first multi-window mode) to obtain the first video frame in the first multi-window mode. At the same time, software will be started to process the video stream of the second video (which can also be called the auxiliary video in the first multi-window mode).

[0159] The display device 200 is specifically configured to process the video stream of the second video through software by calling the image rendering engine through the corresponding application programming interface (API) to process the video stream of the second video through the image rendering engine, thereby obtaining the second video frame in the first multi-window mode.

[0160] In some embodiments, the image rendering engine may include an open graphics library (Open Graphics Library, OpenGL), a lightweight hardware graphics acceleration engine (DirectFB), etc.

[0161] Combination Figure 5 The video stream of the first video is controlled by the multiplexing management middleware, which then processes it through the hardware processor. In one example, control commands are sent to the hardware processor. The multiplexing management middleware then invokes the image rendering engine to process the video stream of the second video.

[0162] In some embodiments, the display device 200 determines the image rendering engine to be invoked based on the software rendering capabilities before invoking the image rendering engine.

[0163] Figure 8 This is a flowchart illustrating the process of the display device 200 calling the image rendering engine in this embodiment of the application. The specific steps are as follows:

[0164] Step S801: Read the software rendering capabilities of the display device.

[0165] Display device 200 reads software rendering capabilities, which can indicate the image rendering engines supported by display device 200.

[0166] Step S802: Based on the software rendering capabilities, determine at least one image rendering engine supported by the display device.

[0167] Based on software rendering capabilities, at least one image rendering engine currently supported by the display device 200 can be determined.

[0168] Step S803: Invoke one of the image rendering engines in at least one image rendering engine to process the video stream of the second video through the image rendering engine, and obtain the second video frame of the second video in the first multi-window mode.

[0169] In some embodiments, if the display device 200 only supports one image rendering engine, the image rendering engine is invoked to process the video stream of the second video to obtain the second video frame in the first multi-window mode.

[0170] In some embodiments, if the display device 200 supports multiple image rendering engines, the image rendering engine with higher priority is called according to a preset priority, so as to process the video stream of the second video through the image rendering engine and obtain the second video frame of the second video in the first multi-window mode.

[0171] Combination Figure 5 Before calling the image rendering engine, the multi-path management middleware reads the software rendering capabilities and determines the image rendering engine based on those capabilities.

[0172] Therefore, after starting the software-assisted rendering mode, the display device 200 first reads the software rendering capabilities of the display device 200 to accurately determine the image rendering engine that the display device 200 can support, and then calls the image rendering engine supported by the display device 200 to process the video stream of the second video, ensuring the effective display of the second video.

[0173] In some embodiments, the display device 200 configures corresponding split-screen middleware for the hardware processor and the image rendering engine, respectively.

[0174] In one example, the hardware processor has a corresponding first split-screen middleware, and the image rendering engine has a corresponding second split-screen middleware.

[0175] Figure 9 This is a flowchart illustrating how the display device 200 processes the video stream of the first video through a hardware processor, as described in this embodiment. The specific steps are as follows:

[0176] In step S901, the first split-screen middleware corresponding to the hardware processor calculates the first size and first position of the first window used to display the first video according to the first multi-window mode.

[0177] The first video is the main video in the first multi-window mode. The first split-screen middleware will calculate the window parameters of the corresponding first window, such as the first size and first position of the first window, according to the configuration of the main video in the first multi-window mode.

[0178] In one example, if the first multi-window mode is PiP mode, the first size of the first window is the size of the display area on the screen, that is, the size of the full-screen display, and the first position of the first window is the entire screen.

[0179] In another example, if the first multi-window mode is a split-screen mode, the first size of the first window is half of the display area on the screen, that is, the size of half-screen display, and the first position of the first window is the left side of the screen.

[0180] In another example, if the first multi-window mode is free mode, the first size of the first window is the size set by the user or the default size, and the first position of the first window is the position set by the user or the default position.

[0181] Step S902: The hardware processor creates a first window based on the first size and the first position.

[0182] A first window is created by a hardware processor based on a first size and a first position, and the first window is used to display the video frame of the first video.

[0183] Step S903: Based on the first window, process the video stream of the first video to obtain the first video frame of the first video in the first multi-window mode, wherein the first video frame is matched with the first window.

[0184] The hardware processor processes the video stream of the first video according to the first window, so that the display position and screen size of the first video screen match the first window, and displays the first video in the first window in a way that fills the first window.

[0185] Combination Figure 5 The multi-channel management middleware sends the first multi-window mode to the first split-screen middleware, which calculates the first size and first position of the first window. The hardware processor processes the video stream of the first video based on the first size and first position to obtain the first video frame.

[0186] Therefore, the display device 200 configures a corresponding first split-screen middleware for the hardware processor. In the first multi-window mode, the first split-screen middleware can accurately calculate the size and position of the first window, so that the display effect of the first video can match the first multi-window mode.

[0187] Figure 10 This is a flowchart illustrating how the display device 200 processes the video stream of the second video through an image rendering engine, as described in this embodiment of the application. The specific steps are as follows:

[0188] In step S1001, the second split-screen middleware corresponding to the image rendering engine calculates the second size and second position of the second window used to display the second video according to the first multi-window mode.

[0189] The second video is the auxiliary video in the first multi-window mode. The second split-screen middleware will calculate the window parameters of the corresponding second window, such as the second size and the second position of the second window, according to the configuration of the auxiliary video in the first multi-window mode.

[0190] In one example, if the first multi-window mode is PiP mode, the second size of the second window is smaller than the size of the display area on the screen, that is, the size of the small window, and the second position of the second window can be the upper right corner.

[0191] In another example, if the first multi-window mode is a split-screen mode, the second size of the second window is half of the display area on the screen, that is, the size of half-screen display, and the second position of the second window is the right side of the screen.

[0192] In another example, if the first multi-window mode is free mode, the second size of the second window is a user-set or default size, and the second position of the second window is a user-set or default position.

[0193] Step S1002: Create a second window using the image rendering engine based on the second size and the second position.

[0194] A second window is created by the image rendering engine based on the second size and the second position. This second window is used to display the video frame of the second video.

[0195] Step S1003: Based on the second window, process the video stream of the second video to obtain the second video frame in the first multi-window mode, wherein the second video frame matches the second window.

[0196] The image rendering engine processes the video stream of the second video according to the second window, so that the display position and screen size of the second video frame match the second window, and the second video is displayed in a way that fills the second window.

[0197] Combination Figure 5 The multi-channel management middleware sends the first multi-window mode to the second split-screen middleware, which calculates the second size and second position of the second window. The image rendering engine then processes the video stream of the second video based on the second size and second position to obtain the second video frame.

[0198] Therefore, the display device 200 configures a corresponding second split-screen middleware for the image rendering engine. In the first multi-window mode, the second split-screen middleware can accurately calculate the size and position of the second window, so that the display effect of the second video can match the first multi-window mode.

[0199] In some embodiments, after the display device 200 obtains the second video image through the image rendering engine, it also needs to perform corresponding processing on the second video image based on its resolution.

[0200] Figure 11 This is a flowchart illustrating how the display device 200 processes the second video frame in this embodiment of the application. The specific steps are as follows:

[0201] Step S1101: Determine whether the resolution of the second video frame is higher than the resolution threshold using the image rendering engine.

[0202] The resolution threshold is the resolution that the display device 200 can support.

[0203] Step S1102: If the resolution of the second video frame is higher than the resolution threshold, perform image quality compression processing on the second video frame according to the resolution threshold.

[0204] Combination Figure 5 After the image rendering engine obtains the second video frame, it compares the resolution of the second video frame with the resolution threshold. If the resolution of the second video frame is less than or equal to the resolution threshold, the second video frame can be directly output. If the resolution of the second video frame is higher than the resolution threshold, the second video frame is compressed before being output.

[0205] Therefore, the display device 200 processes the video stream of the second video through its image rendering engine. After obtaining the second video frame, it compares the resolution of the second video frame with a resolution threshold to identify whether the display device 200 supports the resolution of the second video. If the resolution of the second video frame exceeds the resolution threshold, that is, if the display device 200 does not support the resolution of the second video, the second video frame undergoes image quality compression processing to adapt to the resolution supported by the display device 200, reduce system bandwidth usage, and avoid other abnormal situations.

[0206] Step S406: Display the first video frame on the video layer and the second video frame on the graphics layer, wherein the graphics layer is located above the video layer.

[0207] The hardware processor outputs the first video frame and displays it on the video layer. The image rendering engine outputs the second video frame and displays it on the graphics layer (surface).

[0208] The graphics layer is located above the video layer, so the display device 200 can display the first video frame and the second video frame simultaneously, and the second video frame is superimposed on the first video frame.

[0209] Combination Figure 5 The hardware processor outputs the first video frame to the display controller, which then renders the first video frame at the video layer for display. The image rendering engine renders the second video frame at the graphics layer for display.

[0210] Combination Figure 6B If the first multi-window mode is PiP mode, the first video screen is displayed in the video layer in full-screen mode, and the second video screen is displayed in the graphics layer in a small window mode.

[0211] Based on the above embodiments, when the display device 200 is playing the first video, in response to the start command of the second video, it first identifies whether the hardware display capability of the hardware processor supports multi-channel display. If the hardware display capability does not support multi-channel display, it can start the software-assisted rendering mode to continue processing the video stream of the first video through the hardware processor, and process the video stream of the second video through the image rendering engine in a software manner. In this way, the video streams of multiple videos can be processed simultaneously, and the video images of two videos can be displayed at the same time, so that the display effect of multi-window mode can still be achieved even when the hardware display capability of the display device 200 does not support multi-channel display.

[0212] In other embodiments, if, based on step S403, the hardware display capability of the display device 200 supports multi-channel display, the hardware processor shall preferentially process the multi-channel video.

[0213] Figure 12 This is a flowchart illustrating how the display device 200 processes multiple video streams via a hardware processor, as described in this embodiment. The specific steps are as follows:

[0214] Step S1201: If the hardware display capability supports multi-channel display, according to the first multi-window mode, the video stream of the first video is processed by the hardware processor to obtain the first video frame of the first video in the first multi-window mode; and the video stream of the second video is processed by the hardware processor to obtain the second video frame of the second video in the first multi-window mode.

[0215] When the multi-channel management middleware determines that the hardware display capability of the display device 200 supports multi-channel display, it controls the hardware processor to continue processing the video stream of the first video, and also needs to process the video stream of the second video. In other words, by processing the video streams of the two videos through the hardware processor, a multi-window display effect is achieved.

[0216] In some embodiments, the multi-channel management middleware also sends a first multi-window mode to a first split-screen middleware, which calculates, based on the first multi-window mode, a first size and a first position of a first window for displaying the first video, and a second size and a second position of a second window for displaying the second video. The hardware processor then creates the first and second windows based on the calculation results from the first split-screen middleware. The hardware processor further processes the first video to obtain a first video frame and processes the second video to obtain a second video frame, respectively, based on the first and second windows. The first video frame is matched with the first window, and the second video frame is matched with the second window.

[0217] Step S1202: Combine the first video frame and the second video frame to obtain a combined video frame, which includes the first video frame and the second video frame.

[0218] The hardware processor processes the first and second video frames and transmits them to the display controller. The display controller then performs a composite processing on the first and second video frames to obtain the composite video frame.

[0219] Step S1203: Display the composited video image on the video layer.

[0220] The display controller renders and composites the video images at the video layer so that the first video image and the second video image can be displayed simultaneously at the video layer.

[0221] Based on this, when the display device 200 supports multi-channel display, it prioritizes processing the video streams of the first and second videos simultaneously through the hardware processor to fully utilize the hardware display capabilities and achieve ultra-low latency, high image quality, and low power consumption for multi-channel display. After obtaining the video frames of the two videos through the hardware processor, the display device can combine the two video frames into one video stream for display, achieving a multi-window display effect.

[0222] In some embodiments, if the hardware display capability of the display device 200 supports multi-channel display, and if the user instructs to add a third video to be displayed when the display device 200 is currently performing multi-channel display, i.e., the number of videos to be displayed is 3, the display device 200 can display the three videos based on the multi-channel display capability supported by the hardware display capability.

[0223] Figure 13 This is a flowchart illustrating the display device 200 displaying three video streams in this embodiment of the application. The specific steps are as follows:

[0224] In step S1301, in response to the start playback command of the third video, if the total number of videos to be displayed exceeds the number of channels supported by the hardware display capability, the software-assisted rendering mode is started, and the currently set first multi-window mode is read.

[0225] If a user wants to watch a third video while simultaneously displaying the first and second videos on the display device 200, the user can input a start command for the third video. In response to this start command, the display device 200 can use a multiplexing management middleware to determine whether the total number of videos to be displayed exceeds the hardware display capacity. For example, if the hardware display capacity supports two videos, meaning the hardware processor can handle a maximum of two videos, then displaying three videos exceeds the hardware display capacity.

[0226] If the total number of videos to be displayed exceeds the hardware display capacity, exceeding the hardware display capability of the display device 200, the multi-channel management middleware will control the activation of the software-assisted rendering mode and read the currently set first multi-window mode. This process can be referred to in step S403, and will not be elaborated here.

[0227] In step S1302, under software-assisted rendering mode, the video stream of the first video is processed by the hardware processor to obtain the first video frame of the first video in the first multi-window mode; the video stream of the second video is processed by the hardware processor to obtain the second video frame of the second video in the first multi-window mode; and the image rendering engine is invoked to process the video stream of the third video to obtain the third video frame of the third video in the first multi-window mode.

[0228] The process of processing the first and second videos using a hardware processor can be referred to in step S1201, and the process of processing the third video using an image rendering engine can be referred to in step S405, which describes the process of processing the second video using an image rendering engine. These details will not be repeated here.

[0229] Step S1303: Combine the first video frame and the second video frame to obtain a combined video frame, which includes the first video frame and the second video frame.

[0230] Step S1303 can be referred to step S1202, and will not be repeated here.

[0231] Step S1304: Display the composited video image on the video layer and the third video image on the graphics layer.

[0232] Step S1304 can be referred to step S406, and will not be repeated here.

[0233] Figure 14 This is a schematic diagram of the display device 200 displaying three video streams in an embodiment of this application, as shown below. Figure 14 As shown, if the first multi-window mode is a split-screen mode, in the composite video frame, the first video frame is located on the left, and the second video frame is located on the right. The composite video frame is displayed on the video layer and is located on the left side of the screen. The third video frame is displayed on the graphics layer and is located on the right side of the screen.

[0234] Therefore, when supporting multi-channel display, the hardware display capability of display device 200 can simultaneously process multiple video streams through the hardware processor to achieve simultaneous display of multiple videos. In response to the start command of a third video, if the number of videos to be displayed exceeds the number of channels that the hardware processor can handle simultaneously (i.e., exceeding the hardware display capability), display device 200 can activate a software-assisted rendering mode to expand its multi-channel display capability and achieve simultaneous display of more videos.

[0235] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the embodiments.

Claims

1. A display device, characterized in that, include: monitor; The controller is configured as follows: In response to the start playback command of the first video, the video stream of the first video is processed by the hardware processor to obtain the video frame of the first video; Control the display to show the video frame of the first video on the video layer; In response to the start command of the second video, the system reads the currently set first multi-window mode and the hardware display capabilities of the hardware processor. If the hardware display capability does not support multi-channel display, start the software-assisted rendering mode; In the software-assisted rendering mode, the video stream of the first video is processed by the hardware processor to obtain the first video frame of the first video in the first multi-window mode. In addition, the image rendering engine is invoked to process the video stream of the second video through the image rendering engine, so as to obtain the second video frame of the second video in the first multi-window mode; The display is controlled to display the first video image on the video layer and the second video image on the graphics layer, wherein the graphics layer is located above the video layer.

2. The display device according to claim 1, characterized in that, In the software-assisted rendering mode, the controller processes the video stream of the first video through the hardware processor to obtain the first video frame of the first video in the first multi-window mode, which is specifically configured as follows: The first split-screen middleware corresponding to the hardware processor calculates the first size and first position of the first window used to display the first video according to the first multi-window mode; The hardware processor creates the first window based on the first size and the first position. Based on the first window, the video stream of the first video is processed to obtain the first video frame of the first video in the first multi-window mode, wherein the first video frame is matched with the first window.

3. The display device according to claim 1, characterized in that, In the software-assisted rendering mode, the controller invokes the image rendering engine to process the video stream of the second video, obtaining the second video frame in the first multi-window mode, specifically configured as follows: The second split-screen middleware corresponding to the image rendering engine calculates the second size and second position of the second window used to display the second video according to the first multi-window mode; The second window is created by the image rendering engine based on the second size and the second position; Based on the second window, the video stream of the second video is processed to obtain the second video frame of the second video in the first multi-window mode, wherein the second video frame is matched with the second window.

4. The display device according to claim 1, characterized in that, In the software-assisted rendering mode, after the controller invokes the image rendering engine to process the video stream of the second video and obtains the second video frame in the first multi-window mode, it is further configured as follows: The image rendering engine determines whether the resolution of the second video frame is higher than a resolution threshold. If the resolution of the second video frame is higher than the resolution threshold, the second video frame is subjected to image quality compression processing according to the resolution threshold.

5. The display device according to any one of claims 1-4, characterized in that, In response to the start command of the second video, the controller reads the hardware display capabilities of the hardware processor and is specifically configured as follows: In response to the start command of the second video, the hardware display capabilities of the hardware processor are read from the local configuration file; or, In response to the start-up command of the second video, a configuration request is sent to the server; The server receives feedback from the hardware processor regarding its hardware display capabilities based on the configuration request.

6. The display device according to any one of claims 1-4, characterized in that, In the software-assisted rendering mode, the controller invokes the image rendering engine to process the video stream of the second video, obtaining the second video frame in the first multi-window mode, specifically configured as follows: Read the software rendering capabilities of the display device; Based on the software rendering capabilities, determine at least one image rendering engine supported by the display device; Call one of the at least one image rendering engines to process the video stream of the second video through the image rendering engine, and obtain the second video frame of the second video in the first multi-window mode.

7. The display device according to claim 1, characterized in that, The controller is also configured to: If the hardware display capability supports multi-channel display, the video stream of the first video is processed by the hardware processor according to the first multi-window mode to obtain the first video frame of the first video in the first multi-window mode. And, by processing the video stream of the second video through the hardware processor, a second video frame of the second video in the first multi-window mode is obtained; The first video frame and the second video frame are combined to obtain a combined video frame, which includes the first video frame and the second video frame. The display is controlled to show the synthesized video image on the video layer.

8. The display device according to claim 7, characterized in that, After controlling the display to show the composited video image on the video layer, the controller is further configured to: In response to the start-up command of the third video, if the total number of videos to be displayed exceeds the number of channels supported by the hardware display capability, the software-assisted rendering mode is activated, and the currently set first multi-window mode is read. In the software-assisted rendering mode, the video stream of the first video is processed by the hardware processor to obtain the first video frame of the first video in the first multi-window mode. The hardware processor processes the video stream of the second video to obtain the second video frame in the first multi-window mode. In addition, the image rendering engine is invoked to process the video stream of the third video through the image rendering engine to obtain the third video frame of the third video in the first multi-window mode; The first video frame and the second video frame are combined to obtain a combined video frame, which includes the first video frame and the second video frame. The display is controlled to show the composited video image on the video layer and the third video image on the graphics layer.

9. A method for displaying multiple video streams, applied to a display device, characterized in that, The method includes: In response to the start playback command of the first video, the video stream of the first video is processed by the hardware processor to obtain the video frame of the first video; Display the video frame of the first video in the video layer; In response to the start command of the second video, the system reads the currently set first multi-window mode and the hardware display capabilities of the hardware processor. If the hardware display capability does not support multi-channel display, start the software-assisted rendering mode; In the software-assisted rendering mode, the hardware processor processes the video stream of the first video to obtain the first video frame of the first video in the first multi-window mode; and the image rendering engine is invoked to process the video stream of the second video to obtain the second video frame of the second video in the first multi-window mode. The first video frame is displayed in the video layer, and the second video frame is displayed in the graphics layer, wherein the graphics layer is located above the video layer.

10. The method according to claim 9, characterized in that, The method further includes: If the hardware display capability supports multi-channel display, according to the first multi-window mode, the hardware processor processes the video stream of the first video to obtain the first video frame of the first video in the first multi-window mode; and the hardware processor processes the video stream of the second video to obtain the second video frame of the second video in the first multi-window mode. The first video frame and the second video frame are combined to obtain a combined video frame, which includes the first video frame and the second video frame. The synthesized video image is displayed on the video layer.

Images