Display apparatus and control method

Abstract

Some embodiments of the present application provide a display apparatus and a resource control method in background mode. In the method, in response to a background operation command for a media playback application, the target page of the media playback application is hidden and video elements of the target page are detected. The playback of video elements in the media data is paused and the playback progress of the media data is recorded. Then redundant data is released from a first buffer queue and target information based on the redundant data is recorded. The redundant data is media data in the buffer queue other than the reference data, where the reference data is segment data corresponding to the playback progress.

Description

VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946PDISPLAY APPARATUS AND CONTROL METHODCross-reference to related applications

[0001] The present application claims priority to Chinese patent application No. 202411121150.8, filed on August 14, 2024, Chinese patent application No. 202411379744.9, filed on September 29, 2024, and Chinese patent application No. 202411538828.2, filed on October 31, 2024, the entire contents of which are incorporated herein by reference.Technical Field

[0002] The present application relates to the field of display apparatus technology, and in particular to a display apparatus and a resource control method in a background mode.Background

[0003] Display apparatuses are smart devices capable of presenting user interfaces and supporting user interaction. Smart televisions are taken as an example. Smart televisions are based on Internet application technology, provided with an open operating system and chip, and an open application platform. Television products with the two-way human-machine interaction function and integrated with multiple functions such as audio-visual, entertainment, and data, etc., can be realized, meeting diverse and personalized needs of the user.

[0004] Display apparatuses can be configured with various types of applications to achieve different functions. For example, display apparatuses can be equipped with built-in media playback applications. The media playback application can download media data via a browser, and decode and play the media data, to enable the display apparatus to present the playback image corresponding to the media data. When the display apparatus plays media via the media playback application, the page of media playback application is displayed in the foreground. To facilitate operation, the display apparatus can retain the browser process of the media playback application based on specific user operations and switch the media playback application to the background. In this way, the media playback application can be quickly switched back to the foreground when the user re-uses the media playback application, enabling rapid display of the page of the media playback application.

[0005] After switching the media playback application to the background, although the display apparatus no longer displays the page of the media playback application, the browser process of the media playback application continues to download and decode media data, resulting in significant memory usage and impacting the operational efficiency of the display apparatus.Summary

[0006] Some embodiments of the present application provide a display apparatus, including: a display, configured to display images and / or a user interface; at least one processor, configured to execute computer instructions to cause the display apparatus to: in response to a background operation command for a media playback application, hide a target page of the media playback application and detect video elements of the target page, where the video elements are used to embed a playback image of first media data; pause playbackVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P of the first media data of the video elements; record a playback progress of the first media data; release redundant data from a first buffer queue, where the first buffer queue is used to cache the first media data of the media playback application, the redundant data is the first media data in the first buffer queue except for reference data, and the reference data is segment data corresponding to the playback progress; record target information based on the redundant data; receive a startup command for the media playback application; and in response to the startup command, start the media playback application based on the target information and the playback progress, and control the display to display the playback image of the first media data based on the video elements of the target page.

[0007] Some embodiments of the present application also provide a resource control method, including: in response to a background operation command for a media playback application, hiding a target page of the media playback application and detecting video elements of the target page, where the video elements are used to embed a playback image of first media data; pausing playback of the first media data of the video elements; recording a playback progress of the first media data; releasing redundant data from a first buffer queue, where the first buffer queue is used to cache the first media data of the media playback application, the redundant data is the first media data in the first buffer queue except for reference data, and the reference data is segment data corresponding to the playback progress; recording target information based on the redundant data; receiving a startup command for the media playback application; and in response to the startup command, starting the media playback application based on the target information and the playback progress, and controlling a display to display the playback image of the first media data based on the video elements of the target page.Brief Description of Figures

[0008] FIG. 1 is a schematic diagram of an operational scenario between a display apparatus and a control device according to some embodiments of the present application.

[0009] FIG. 2 is a schematic diagram of a hardware configuration of the display apparatus according to some embodiments of the present application.

[0010] FIG. 3 is a schematic diagram of a software configuration of the display apparatus according to some embodiments of the present application.

[0011] FIG. 4 is a flowchart of a media data download and parsing process according to some embodiments of the present application.

[0012] FIG. 5 is a schematic diagram of a media data decoding and playback process according to some embodiments of the present application.

[0013] FIG. 6 is a schematic diagram of an effect of switching a media playback application to a background according to some embodiments of the present application.

[0014] FIG. 7 is a timing chart of a resource control method in a background mode according to some embodiments of the present application.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0015] FIG. 8 is a flowchart of a resource control method in the background mode according to some embodiments of the present application.

[0016] FIG. 9 is a schematic diagram of releasing redundant data according to some embodiments of the present application.

[0017] FIG. 10 is a schematic diagram of background recording information according to some embodiments of the present application.

[0018] FIG. 11 is a schematic diagram of an effect of a background tab according to some embodiments of the present application.

[0019] FIG. 12 is a schematic diagram of an effect of switching the media playback application from the background to a foreground according to some embodiments of the present application.

[0020] FIG. 13 is a flowchart of downloading redundant data according to some embodiments of the present application.

[0021] FIG. 14 is a flowchart of downloading supplementary media data according to some embodiments of the present application.

[0022] FIG. 15 is a flowchart of merging buffer queues according to some embodiments of the present application.

[0023] FIG. 16 is a schematic diagram of an effect of switching the media playback application from the background to the foreground under an abnormal condition according to some embodiments of the present application.

[0024] FIG. 17 is a calculation principle diagram of live-streaming media playback time according to some embodiments of the present application.

[0025] FIG. 18 is a flowchart of a media image display process based on hole-digging operations according to some embodiments of the present application.

[0026] FIG. 19 is a flow architecture diagram of a display layer hole-digging operation according to some embodiments of the present application.

[0027] FIG. 20 is a flowchart of a media image display method according to some embodiments of the present application.

[0028] FIG. 21 is a timing chart of a media image display method according to some embodiments of the present application.

[0029] FIG. 22 is a schematic diagram of a display level of display layers in a browser page according to some embodiments of the present application.

[0030] FIG. 23 is a flowchart of calculating a target region corresponding to a display layer according to some embodiments of the present application.

[0031] FIG. 24 is a flowchart of a calculation principle of the target region according to some embodiments of the present application.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0032] FIG. 25 is a flowchart of drawing a display image and performing a hole-digging operation according to some embodiments of the present application.

[0033] FIG. 26 is a schematic diagram of a media image display effect in a non-full-screen state without converting returned coordinates according to some embodiments of the present application.

[0034] FIG. 27 is a flowchart of calculating the returned coordinates according to some embodiments of the present application.

[0035] FIG. 28 is a schematic diagram of a media image display effect in a non-full-screen state with converted returned coordinates according to some embodiments of the present application.

[0036] FIG. 29 is a timing chart of the display apparatus decoding page loading data based on hardware decoding function according to some embodiments of the present application.

[0037] FIG. 30 is a timing chart of the display apparatus decoding page loading data based on software decoding function according to some embodiments of the present application.

[0038] FIG. 31 is a determination flowchart of decoding page loading data based on a data format of a page loading data according to some embodiments of the present application.

[0039] FIG. 32 is a schematic diagram of a level relationship between a graphics layer and a video layer according to some embodiments of the present application.

[0040] FIG. 33 is a schematic diagram of an operation of a first process according to some embodiments of the present application.

[0041] FIG. 34 is a schematic diagram of a migration of page loading data from the first process to a second process according to some embodiments of the present application.

[0042] FIG. 35 is a schematic diagram of transparent processing of the graphics layer according to some embodiments of the present application.Detailed Description

[0043] The following will provide a detailed description of the embodiments, with examples illustrated in the accompanying drawings. When referring to the drawings in the following description, unless otherwise indicated, the same reference numerals denote the same or similar elements in different drawings. The embodiments described herein do not represent all possible embodiments consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application as detailed in the claims.

[0044] It should be noted that the brief explanations of terms in the present application are provided solely for the convenience of understanding the embodiments described herein and are not intended to limit the scope of the embodiments of the present application. Unless otherwise specified, these terms should be understood in their ordinary and customary meanings.

[0045] The terms “first,” “second,” “third,” etc., used in the specification, claims, and the above-mentioned drawings of the present application are used to distinguish similar or identical objects or entities and do notVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P necessarily imply a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchangeable in appropriate contexts.

[0046] The terms “including” and “having” and any variations thereof are intended to cover but not be limited to the inclusion of a series of components in a product or device, which need not be limited to all components clearly listed, but can include other components not clearly listed or inherent to such products or devices.

[0047] The term “module” refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing functions related to the component.

[0048] In the embodiments of the present application, the display apparatus 200 generally refers to an apparatus with image display and data processing capabilities. For example, the display apparatus 200 can include but is not limited to smart televisions, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0049] FIG. 1 is a schematic diagram of an operational scenario between the display apparatus and the control device according to some embodiments of the present application. As shown in FIG. 1, a user can operate the display apparatus 200 through touch operations, the mobile terminal 300, and the control device 100. The control device 100 is used to receive operation commands input from the user and convert them into control commands that the display apparatus 200 can recognize and respond to. For example, the control device 100 can be a remote control, a touch pen, a joystick, etc.

[0050] The mobile terminal 300 can serve as a control device for executing human-machine interaction between the user and the display apparatus 200. The mobile terminal 300 can also function as a communicating device for establishing a communication connection with the display apparatus 200 and exchanging data. In some embodiments, the mobile terminal 300 can install software applications on the display apparatus 200 and establish a connection via a network communication protocol to achieve one-to-one control operations and data communication. Audio-visual content displayed on the mobile terminal 300 can also be transmitted to the display apparatus 200 to achieve synchronized display function.

[0051] In some embodiments, the mobile terminal 300 or other electronic devices can also simulate the function of the control device 100 by running an application that can control the display apparatus 200.

[0052] As shown in FIG. 1, the display apparatus 200 can also communicate with the server 400 via various communication methods. The display apparatus 200 can communicate via a local area network (LAN), wireless local area network (WLAN), and other networks.

[0053] The display apparatus 200 can provide broadcast television reception function and can additionally provide smart network television function with computer -supported features, including but not limited to network television, smart television, and Internet Protocol Television (IPTV).

[0054] FIG. 2 is a hardware configuration block diagram of the display apparatus 200 shown in FIG. 1 according to some embodiments of the present application.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0055] In some embodiments, the display apparatus 200 can include a tuning demodulator 210, a communicating device 220, a detector 230, a device interface 240, a processor 250 (the processor can be at least one, such as one or more), a display 260, an audio output interface 270, a memory, a power supply, and at least one of a user input interface.

[0056] In some embodiments, the detector 230 is used to collect signals from the external environment or from interactions with the external environment. For example, the detector 230 can include a light receiver, such as a sensor for measuring ambient light intensity; or, the detector 230 can include an image acquisitor, such as a camera, which can be used to capture external environmental scenes, user attributes, or user interaction gestures; or, the detector 230 can include a sound acquisitor, such as a microphone, for receiving external sounds.

[0057] In some embodiments, the display 260 can include display functional components for presenting images and drive components for driving image display. The display 260 is used to receive image signals output from the processor 250 for display. For example, the display 260 can be used to display video content, image content, components of menu control interfaces, and user interfaces, etc.

[0058] In some embodiments, the communicating device 220 is a component used to communicate with external devices or servers 400 according to various communication protocol types. The display apparatus 200 can be configured with multiple communicating devices 220 depending on the supported communication methods. For example, when the display apparatus 200 can support wireless network communication, it can be equipped with a communicating device 220 that can include WiFi function. When the display apparatus 200 can support Bluetooth connection communication, the display apparatus 200 must be equipped with a communicating device 220 that can include Bluetooth function.

[0059] The communicating device 220 can enable the display apparatus 200 to establish a communication connection with external devices or servers 400 via wireless or wired connections. Wired connections can connect the display apparatus 200 to external devices through components such as data cables or interfaces. Wireless connections can connect the display apparatus 200 to external devices via wireless signals or wireless networks. The display apparatus 200 can establish a direct connection with external devices or an indirect connection through gateways, routers, or connection devices.

[0060] In some embodiments, the processor 250 can include at least one of a central processor, a video processor, an audio processor, a graphics processor, or a power processor, and can include first to nthinterfaces for input / output. The processor 250 can control the operation of the display apparatus and respond to user operations through various software control programs stored in the memory. The processor 250 can control the overall operation of the display apparatus 200.

[0061] In some embodiments, the processor 250 and the tuning demodulator 210 can be located in separate devices, i.e., the tuning demodulator 210 can also be in an external device, such as an external set-top box, of the main device where the processor 250 is located.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0062] In some embodiments, the user can input user commands via the graphical user interface (GUI) displayed on the display 260, and the user input interface can receive user input commands through the GUI.

[0063] In some embodiments, the audio output device 270 can be the built-in speakers of the display apparatus 200 or an external audio output device connected to the display apparatus 200. For audio output devices connected to the display apparatus 200, the display apparatus 200 can also be equipped with an external audio output terminal. The audio output device can be connected to the display apparatus 200 via the external audio output terminal to output sound from the display apparatus 200.

[0064] In some embodiments, the user input interface 280 can be used to receive commands input from user.

[0065] To perform user interaction, in some embodiments, the display apparatus 200 can run an operating system. The operating system is a computer program used to manage and control hardware and software resources within the display apparatus 200. The operating system can control the display apparatus to provide a user interface, for example, the operating system can directly control the display apparatus to provide a user interface, or it can provide a user interface through running applications. The operating system also allows the user to interact with the display apparatus 200.

[0066] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized for a specific operating platform, or an independent operating system specifically developed for the display apparatus.

[0067] The operating system can be divided into different modules or layers based on the functions it implement. For example, as shown in FIG. 3, FIG. 3 is a schematic diagram of the software configuration of the display apparatus according to some embodiments of the present application. In some embodiments, the system of the display apparatus 200 can be divided into three layers, from top to bottom: the application layer, the middleware layer, and the hardware layer.

[0068] The application layer can primarily include applications on the television and the application framework. Applications are primarily browser-based applications developed using HTML5, such as HTML5 apps, and native applications (Native Apps).

[0069] The application framework is a complete program model that can provide all the basic functions, such as file access and data exchange, etc., required by standard application software, and the interfaces (toolbars, status bars, menus, dialog boxes) for using these functions.

[0070] Native apps (Native APPs) can support online or offline function, push notifications, or access to local resources.

[0071] The middleware layer can include various television protocols, multimedia protocols, and system components. Middleware utilizes the basic services (functions) provided by system software to connect different parts of application systems or distinct applications over a network, enabling resource sharing and functional integration.

[0072] The hardware layer can primarily include HAL interfaces, hardware, and drivers. The HAL interface serves as the unified interface for all television chips, with specific logic implemented by each chip. DriversVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P can primarily include: audio drivers, display drivers, Bluetooth drivers, camera drivers, Wi-Fi drivers, USB drivers, HDMI drivers, sensor drivers (such as fingerprint sensors, temperature sensors, pressure sensors, etc.), and power drivers.

[0073] It should be noted that the above example is merely a simple division of operating system functions and does not constitute a limitation on the specific form of the operating system in the display apparatus 200 of the present application. Depending on factors such as the function of the display apparatus and the type of operating system, the number of layers and specific layer types included in the operating system can take other forms.

[0074] To expand the function of display apparatus 200, in some embodiments, as shown in FIG. 3, the application layer of display apparatus 200 can be configured with applications, such as media playback applications, game applications, etc., for implementing different functions. For example, the media playback application is an application developed based on the browser, which can use the MSE (Media Source Extensions) playback method to play media data. The browser can download the media data and actively pushes it via JavaScript (JS) to achieve media data playback.

[0075] In some embodiments, the media playback application can use the video element(s) to display the playback image of the media data. The video element is the <video> tag of HTML5, which can provide native support for video playback. The video element can directly embed media data into the HTML document, and the media data can support multiple formats, such as MP4, WebM, Ogg, etc.

[0076] In some embodiments, the display apparatus 200 can communicate with the server 400 via a browser to enable media data interaction. That is, during communication between the display apparatus 200 and the server 400, the display apparatus 200 can send an obtaining request including a network link to the server 400 to obtain the media data corresponding to the media. Upon receiving the obtaining request sent by the display apparatus 200, the server 400 can extract the corresponding media data based on the obtaining request and return the extracted media data to the display apparatus 200, thereby downloading the media data.

[0077] To achieve data interaction between the display apparatus 200 and the server 400, in some embodiments, the display apparatus 200 can establish a communication connection with the server 400 through the browser of the media playback application to establish a transmission channel for media data. For example, both the display apparatus 200 and the server 400 can be connected to the internet and transmit interactive data between the display apparatus 200 and the server 400 according to internet transmission protocols.

[0078] It should be noted that the display apparatus 200 and the server 400 can also establish a communication connection using other connection methods. For example, wired broadband, wireless local area networks, cellular networks, Bluetooth, infrared, or radio frequency communication.

[0079] In some embodiments, the connection between display apparatus 200 and server 400 is a “many-to-one” relationship, meaning that multiple display apparatuses 200 can establish a communication connection with the same one server 400, enabling server 400 to provide services to multiple display apparatuses 200. Alternatively,VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P the connection between the display apparatus 200 and the server 400 can also be a “many-to-many” connection, meaning that multiple display apparatuses 200 can establish communication connections with multiple servers 400, enabling multiple servers 400 to provide different services to the display apparatuses 200. Clearly, in certain application scenarios, the connection between display apparatus 200 and server 400 can also be a “one- to-one” relationship, meaning that one server 400 can provide services to one display apparatus 200.

[0080] To improve the smoothness of media playback, in some embodiments, the media playback application on the display apparatus 200 can pre-download a certain amount of media data into memory via a browser, parse the downloaded media data, and cache the parsed media data in a buffer queue. For example, as shown in FIG. 4, the display apparatus 200 can download segment data of the media data via the browser, parse the segment data, and then send the parsed segment data to the buffer queue for caching.

[0081] In some embodiments, after downloading the media data via the browser, the display apparatus 200 can decode the media data using the browser’s decoder, and then render the decoding data frames into the corresponding video element, enabling the media playback application to display the playback image of the media data via the browser page. For example, as shown in FIG. 5, the display apparatus 200 can read the media data cached in the buffer queue through the browser, decode the read media data using the decoder, and play the decoded media data to display the playback image of the media data through the browser page.

[0082] In some embodiments, the media data frames can include I-frames (intra-coded frames) and P-frames (forward-predicted coded frames). I-frames can appear in different forms, such as IDR, CRA, or BLA, etc., in different codecs. I-frames can consist solely of intra-predicted macroblocks, where each macroblock within one frame can only match other macroblocks within this one frame and can only be compressed using intraframe spatial redundancy, i.e., compressed based on the similarity between pixels within a single frame. Different types of I-frames are fundamentally the same and do not involve temporal prediction. Therefore, I- frames can be encoded and decoded independently. P frames can represent predictive frames. In addition to spatial prediction, P frames can also be compressed using temporal prediction. P frames can require reference to previous frames for motion estimation, and each macroblock within a P frame can be subject to temporal prediction, or spatial prediction, or can be skipped. Skipping involves the decoder copying the macroblock from the previous frame located at the same positions, i.e., zero motion direction.

[0083] To facilitate user control, in some embodiments, after starting the media playback application, the display apparatus 200 display the target page of the media playback application in the foreground. At this point, the user can input a background operation command via a specific interaction action, causing the display apparatus 200, in response to the background operation command, to switch the media playback application to the background and hide the display of the target page. For example, as shown in FIG. 6, the user can input the background operation command via the “HOME” button on the remote control accompanying the display apparatus 200, causing the display apparatus 200 to switch to the home page and hide a playback page of the media playback application in the background; or, when the display apparatus 200 can support touch operations, the user can input the background operation command via an upward swipe gesture, etc.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0084] In some embodiments, when the display apparatus 200, in response to the background operation command, switches the media playback application to the background, it can retain the browser process of the media playback application. Thus, when the media playback application is re-entered by the user, the display apparatus 200 can quickly switch the media playback application to the foreground using the retained browser process, thereby restoring the playback effect of the browser application.

[0085] In some embodiments, when the display apparatus 200 switches the media playback application to the background, the display apparatus 200 can freeze the JavaScript (JS) of the media playback application, and prevent the JS from executing any playback operations. However, the browser of the media playback application will continue to perform the download, parsing, and decoding operations shown in FIGS. 4 and 5. In other words, the display apparatus 200 can only pause the media playback operations at the JavaScript layer of the media playback application and hide the target page of the media playback application, but the browser can continue to download and play media data, resulting in significant memory usage and affecting the operational efficiency of the display apparatus 200.

[0086] Additionally, since the browser can continue to download and play media data, it can cause the playback progress of media data in the media playback application to be pushed backward and occupy the playback channel of display apparatus 200. As a result, when the display apparatus 200 re-runs the media playback application, there will be a mismatch between the playback progress of the media data and the playback progress when the media playback application was switched to the background. Media data from other channels also cannot be played normally, which not only affects the user’s viewing experience but also makes it easy for the media playback application to encounter errors.

[0087] For example, when the media playback application is playing media data, and the playback progress reaches point A, the user can switch the media playback application to the background. Over time, the browser can continue to download, decode, and play the media data. When the media playback application is re-entered by the user, the playback progress of the media data in the browser has already reached point B. The media playback application will then determine that the playback states are inconsistent and trigger an error.

[0088] To address this, a display apparatus 200 is provided in some embodiments of the present application can release most of the memory when the media playback application is moved to the background, thereby reducing the resource consumption of the media playback application. Additionally, the display apparatus 200 can quickly seek to the playback progress of the media playback application when the media playback application was moved to the background upon re-entering the display apparatus 200, and resume the playback of the media data, thereby achieving a seamless switching effect, and enhancing the user experience.

[0089] As shown in FIG. 7, in some embodiments, at least one processor 250 can be configured to execute computer instructions to cause the display apparatus to perform the following steps.

[0090] S7001: the display apparatus can receive a background operation command sent by the user via a remote control or other device.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0091] S7002: in response to the background operation command sent by the user via the remote control or other device to the display apparatus, the display apparatus can hide the target page of the media playback application and detect the video elements of the target page.

[0092] S7003: the target page on the display of the display apparatus is hidden.

[0093] S7004, the detected video elements are obtained.

[0094] S7005, the playback of the media data for the video elements is instructed to be paused.

[0095] S7006, the playback of the media data in the media playback application is paused.

[0096] S7007, the playback progress of the media data is obtained.

[0097] S7008, the playback progress of the media data is recorded.

[0098] S7009, redundant data is instructed to be released.

[0099] S7010, the media data in the media playback application is deleted.

[0100] S7011, target information is recorded based on the redundant data.

[0101] S7012, a startup command issued by the user for the media playback application is received.

[0102] S7013, in response to the startup command, the target information and playback progress are obtained.

[0103] S7014, the media playback application is started based on the target information and playback progress.

[0104] S7015, the display is instructed to display the target page.

[0105] S7016, the media data can continue to be played in the media playback application.

[0106] S7017, the display is controlled to display the playback image of the media data.

[0107] S7018, the media data on the display can be presented to the user.

[0108] As shown in FIGS. 7 and 8, in some embodiments, the display apparatus 200 can include a display 260 and at least one processor 250. The display 260 can be configured to display a user interface, and the at least one processor 250 can be configured to execute computer instructions to cause the display apparatus to perform the following steps.

[0109] S801: in response to a background operation command for a media playback application, a target page of the media playback application is hidden and video elements of the target page are detected.

[0110] The display apparatus 200 can play media data via the media playback application, with the playback method of the media data including live-streaming or on-demand playback. In some embodiments, the display apparatus 200 can establish a connection with the server 400 via the browser of the media playback application in response to the media playback command input from the user, to download, parse, decode, and play the media data in real time, thereby displaying the playback image of the media data via the target page of the media playback application, i.e., the browser page. For example, the display apparatus 200 can download and play media data as shown in FIGS. 4 and 5.

[0111] In some embodiments, the display apparatus 200 can render the decoded media data to the video element, i.e., the video element can be used to embed the playback image of the media data. Each video element can be correspondingly set with a video object, i.e., a HTML video element object. When the videoVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P element is loaded into the browser, it is parsed by the browser as the HTML video element object, which can provide methods and properties for operating the playback of media data, e.g., pausing of the media data playback, and a current time property, etc.

[0112] During the playback of media data by the media playback application, the user can input the background operation command for the media playback application to the display apparatus 200 through specific the interactive operation to switch the media playback application to the background, so that other operations can be performed or the playback of media data can be temporarily paused via the display apparatus 200. Therefore, the display apparatus 200 can hide the target page of the media playback application in response to the background operation command for the media playback application.

[0113] In some embodiments, when the display apparatus 200 switches the media playback application to the background, it can freeze the JavaScript of the media playback application and hide the target page. At this point, the JavaScript of the media playback application can no longer execute any operations, and the user cannot perceive the display effect of the target page. However, since JavaScript cannot affect the browser or the hardware platform of the display apparatus 200, the browser and the hardware platform will continue to perform the download and playback operations for the media data as shown in FIGS. 4 to 5. Therefore, the display apparatus 200 can also detect video elements within the target page to process the video elements of the target page.

[0114] S802: the playback of the media data of the video elements is paused.

[0115] After the display apparatus 200 detects the video elements of the target page, it can pause the playback of the media data in the video elements. At this point, the browser of the media playback application will no longer continue to download media data from the server 400, thereby reducing memory usage and freeing up system resources.

[0116] In some embodiments, after the display apparatus 200 detects the video elements of the target page, it can iterate through the video objects of the video elements. It then can call a predefined pause function to stop the video objects from playing the media data and release the playback resource used for playing the media data, thereby reducing memory usage when the media playback application is running in the background. For example, the display apparatus 200 can call “pause()” method to perform a pause operation on the video object.

[0117] In some embodiments, the playback resource released by the display apparatus 200 can include at least one of decoder resources, memory resources, network resources, or GPU resources. Memory resources can include memory space for storing video frames, decoding data, and playback status information, and network resources can include connections established with the video stream server and related network buffers, etc.

[0118] S803: a playback progress of the media data is recorded.

[0119] While the display apparatus 200 pauses the playback of the media data by the video element, it can also record the playback progress of the media data within the video element to track the current playback progress of the media data in the media playback application.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0120] In some embodiments, the display apparatus 200 can query the current time attribute of the video object and parse the current playback position of the media data through the current time attribute to generate the playback progress of the media data. In this embodiment, the current time attribute is recorded as the playback progress of the media data, which facilitates the display apparatus 200 to quickly locate the playback progress of the media playback data.

[0121] In some embodiments, the display apparatus 200 can also record the playback progress based on the frame number or timestamp of the media data, i.e., determine the frame number or timestamp of the data frame at the current playback position of the media data, and record the determined frame number or timestamp to record the playback progress of the media data.

[0122] S804: redundant data is released from the first buffer queue.

[0123] Since the display apparatus 200 can preload part of the media data, and this part of the media data is useless for the media playback application running in the background and can also occupy a large amount of memory. Therefore, after the display apparatus 200 pauses the playback of the video element and records the playback progress of the media data, the display apparatus 200 can also release the redundant data in the buffer queue. For ease of distinction, in the embodiments of the present application, when the media playback application is running in the foreground, the buffer queue for caching media data is referred to as the first buffer queue.

[0124] The redundant data refers to media data in the first buffer queue other than the reference data, and the reference data refers to the segment data corresponding to the playback progress. That is, in some embodiments, the display apparatus 200 can query the target key frame based on the playback progress. The target key frame is a key frame, that is closest to a key point, of the media data in the first buffer queue. Then, the target segment data including the target key frame is queried, and the media data in the first buffer queue whose playback position is ahead of the target segment data is marked as the reference data. The media data in the first buffer queue, except for the reference data, is then deleted to release the redundant data and reduce the memory usage of the media playback application running in the background.

[0125] For example, as shown in FIG. 9, the display apparatus 200 can determine the segment data containing the key frame based on the playback progress of the media data, retain that segment data, and delete the remaining segment data. In this way, only the media data corresponding to the playback progress, i.e., the reference data, is retained in the first buffer queue, effectively reducing the memory usage of cached data. Additionally, the retained reference data can support fast playback when re-entering the media playback application, enabling seamless switching from background to foreground for the media playback application.

[0126] S805: target information is recorded based on the redundant data.

[0127] Since the media data pre -downloaded by the display apparatus 200 is released when the media playback application switches to the background, if the user restarts the media playback application, the media playback application can unfreeze the Js and pause the video elements, and the browser can continue toVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P download the media data. However, the media data downloaded by the browser can continue based on the previously preloaded media data, which can affect the integrity of the media data and cause playback errors.

[0128] Therefore, to enable the media playback application to continue playing the media data when reentering the media playback application, the display apparatus 200 can record the cache time and segment address of the redundant data while releasing the redundant information, and generate target information to re- download and cache the released media data based on the target information.

[0129] In some embodiments, when the display apparatus 200 records target information based on redundant data, it can detect the cache time and segment address of the redundant data, then add the cache time and segment address to the first buffer queue to record the target information. That is, it can record the buffer range and segment address of each segment data in the redundant data to facilitate that the display apparatus 200 can subsequently re-cache the redundant data. For example, as shown in FIG. 10, the playback progress, target information, and retained reference data recorded by the display apparatus 200 can only occupy a small amount of memory and cannot affect the playback of other channels, such as HDMI, in the display apparatus 200, thereby reducing the occupation of system resources and ensuring the operational efficiency of the display apparatus 200.

[0130] S806: the startup command for the media playback application is received.

[0131] After the display apparatus 200 switches the media playback application to the background, the user can still switch the media playback application back to the foreground based on specific interactive operations. For example, the user can select the media playback application’s icon to input the startup command for the media playback application to the display apparatus 200; or, as shown in FIG. 11, the user can also control the display apparatus to display the background application tab by long-pressing the “HOME” button on the remote control accompanying the display apparatus 200, and then input the startup command for the media playback application to the display apparatus 200 by performing a selection operation on the media playback application tab 1101.

[0132] S807: in response to the startup command, the media playback application is started based on the target information and playback progress, and the display apparatus is controlled to display the playback image of the media data based on the video elements of the target page.

[0133] After the display apparatus 200 receives the startup command for the media playback application again, it can, in response to the start command, start the media playback application based on the target information and playback progress, and control the display 260 to continue displaying the playback image of the media data based on the video elements of the target page, i.e., cancelling the pause playback of video elements. In this way, the progress of the media data can be quickly located based on the playback progress, and then the media data can continue to still be downloaded and cached based on the target information, thereby improving the playback continuity and smoothness of the media data, which can achieve the seamless transition of the media playback application from the background to the foreground as shown in FIG. 12, and enhance the user experience.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0134] In some embodiments, when starting the media playback application based on the target information and playback progress, the display apparatus 200 can revoke the hidden state of the target page to allow the display 260 of the display apparatus 200 to re-display the display effect of the target page and switch the target page of the media playback application to the foreground. Next, the reference data in the fust buffer queue is located based on the playback progress, and the reference data in the first buffer queue is decoded starting from the playback progress. Then the decoded reference data is rendered to the video element to continue displaying the playback image of the media data via the video elements.

[0135] In other words, the display apparatus 200 can seek to the playback time of the media data when the media playback application is switched to the background based on the recorded playback progress, and then continue to decode and play the media data at that time. Since the display apparatus 200 has retained the segment data corresponding to the playback progress when releasing the data, the media playback application can quickly present the media data image at this time, achieving a seamless switch of the media playback application.

[0136] In some embodiments, after the display apparatus 200 receives the startup command for the media playback application, it can also, in response to the startup command, read the target information recorded in the first buffer queue, download the redundant data according to the target information, and cache the downloaded redundant data in the first buffer queue. It then can play the media data based on the first buffer queue.

[0137] In other words, after the display apparatus 200 switches the media playback application to the background, if the media playback application is restarted, the display apparatus 200 will unfreeze the application JS of the media playback application to continue downloading subsequent media data, i.e., media data whose playback position is behind the redundant data. To enhance the continuity of media playback, the display apparatus 200 needs to re-download the already released redundant data to the first buffer queue based on the recorded target information for decoding and playback.

[0138] In some embodiments, when the display apparatus 200 downloads redundant data according to the target information, it can download the redundant data based on the recorded cache time and segment address. For example, as shown in FIG. 13, the display apparatus 200 can download each segment data sequentially according to the recorded target information 1301, cache the downloaded data in the first buffer queue, and decode and play the media data through the first buffer queue.

[0139] In some embodiments, the display apparatus 200 can also, in response to the startup command, download supplementary media data based on the media playback application. The supplementary media data can refer to media data whose playback position is located behind the redundant data. The supplementary media data is then cached in the second buffer queue, and the media data is played back using the second buffer queue and the first buffer queue.

[0140] For example, as shown in FIG. 14, after the display apparatus 200 unfreezes the application Js, the media playback application can continue to download subsequent supplementary media data and cache theVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P supplementary media data in the second buffer queue, enabling the display apparatus 200 to decode and play the media data based on the first buffer queue and the second buffer queue.

[0141] To improve the continuity and accuracy of the playback process, in some embodiments, after caching the downloaded redundant data in the first buffer queue, the display apparatus 200 can merge the first buffer queue and the second buffer queue to generate a composite buffer queue. It then can read the target playback position of the media data in the video element and decode the media data in the composite buffer queue according to the target playback position. The decoded media data is then rendered to the video element of the target page to display the playback image of the media data via the video element.

[0142] For example, after the display apparatus 200 downloads and caches the redundant data in the first buffer queue based on the target information, it can merge the second buffer queue containing the downloaded supplementary media data into the end of the first buffer queue, forming a composite buffer queue as shown in FIG. 15. In this way, the display apparatus 200 can decode and play the media data based on the composite buffer queue, and continuously render the playback image of the media data in the video element of the target page.

[0143] The first buffer queue is the additional segment data downloaded by the browser compared to the application JS after switching to the background, while the second buffer queue is the newly downloaded segment data after the application JS is thawed when restarting the media playback application. These two can be merged to form a unified buffer queue.

[0144] For certain abnormal events, such as invalid segment address or seek abnormalities in the media playback application, in some embodiments, the display apparatus 200 can, when detecting the abnormal event, exit the video element playback of the media playback application, and close the target page. At this point, the user can see the application page one level of which above the playback page in the media playback application on the display apparatus 200. Since the browser process of the media playback application is not closed when the media playback application is moved to the background, there is no need to reload the page resources of the upper-level page. Therefore, the display apparatus 200 can quickly display the display image of the upper-level page in response to the startup command, achieving rapid response to the startup command during the transition from background to foreground.

[0145] For example, as shown in FIG. 16, while the display apparatus 200 is playing media via the media playback application, the user can input the background operation command to switch the media playback application to the background. After a period of time, the media playback application can be re-entered by the user. At this point, the display apparatus 200 can detect an abnormal event where the segment address has become invalid, so it can stop the playback of the video element of the target page, close the target page, and return to the upper-level menu page of the media playback application, thereby achieving a rapid response to the startup command.

[0146] For live-streaming media data, since live-streaming media requires the playback progress to be adjusted in real-time, when the display apparatus 200 transitions from the background to the foreground, theVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P playback progress of the media data must be updated according to real-time. Therefore, in some embodiments, when display apparatus 200 detects that the media data playback type is live streaming, it can, in response to the background operation command for the media playback command, execute the above steps S801 and S802. Then, it can record the current first playback time and the first network time corresponding to network UTC (Coordinated Universal Time). Then, all media data in the first buffer queue is released to reduce memory usage by the media playback application when it is running in the background.

[0147] For live-streaming media data, in some embodiments, after the display apparatus 200 switches the media playback application to the background, it can receive a startup command for the media playback application. In response to the startup command, it can obtain the second network time corresponding to the current network UTC. Then, it can calculate the difference between the second network time and the first network time, and it can calculate the current second playback time based on the difference and the first playback time, enabling the media playback application to continue playing the media data according to the second playback time.

[0148] For example, as shown in FIG. 17, the display apparatus 200 can play live -streaming media via the media playback application. The user can input the background operation command for the media playback application, and switch the media playback application to the background without displaying. At this point, the playback time recorded by the display apparatus 200 is 1719973655, corresponding to UTC time 1719944855. After some time, the user can restart the media playback application, and the display apparatus 200 can switch the media playback application from the background to the foreground. At this point, the UTC time obtained by display apparatus 200 is 1719948455. By calculating the current latest playback time using the two UTC times, the latest playback time is determined to be 1719977255. The media playback application is then controlled to continue playing the live -streaming media according to the playback time of 1719977255, ensuring the correct playback of the live-streaming media and avoiding playback errors caused by time inconsistencies.

[0149] Based on the above display apparatus 200, some embodiments of the present application also provide a resource control method in background mode, which can be performed by the display apparatus 200 provided in the above embodiments, as shown in FIG. 8. The method can include the following steps.

[0150] S801: in response to the background operation command for the media playback application, the target page of the media playback application is hidden and the video elements of the target page is detected, and the video elements are used to embed the playback image of the media data.

[0151] S802: the playback of the media data of the video elements is paused.

[0152] S803: the playback progress of the media data is recorded.

[0153] S804: the redundant data is released from the first buffer queue. The first buffer queue is used to cache the media data of the media playback application. The redundant data refers to media data in the first buffer queue other than the reference data, where the reference data is the segment data corresponding to the playback progress.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0154] S805: the target information is recorded based on the redundant data.

[0155] S806: the startup command for the media playback application is received.

[0156] S807: in response to the startup command, the media playback application is started based on the target information and playback progress, and the display apparatus is controlled to display the playback image of the media data based on the video elements of the target page.

[0157] As can be seen from the above embodiments, according to the display apparatus and resource control method in background mode in some embodiments of the present application, in the method, in response to the background operation command for the media playback application, the target page of the media playback application is hidden, and the video elements of the target page is detected. The playback of the media data of the video elements is paused, and the playback progress of the media data is recorded. Then the redundant data is released from the first buffer queue and the target information is recorded based on the redundant data. The redundant data is media data in the first buffer queue excluding the reference data, where the reference data is the segment data corresponding to the playback progress. The startup command for the media playback application is received, and in response to the startup command, the media playback application is started based on the target information and playback progress. The display 260 is controlled to continue displaying the playback image of the media data based on the video elements of the target page. The method can reduce memory usage when the media playback application is running in the background and enable quick resume playback upon restarting the media playback application, thereby improving the user experience.

[0158] To address the above problem, the display apparatus and media image display method according to some embodiments of the present application are provided. In the method, in response to the media playback command, the display 260 is controlled to display a playback window for displaying the browser page, and the regions of the display layers overlapping with the media image on the browser page are calculated. It then can draw the display layers and set the overlapping regions identified in the display layers as transparent regions. It then can decode the media image based on the returned coordinates to display the media image in the display region corresponding to the returned coordinates. The returned coordinates are determined based on the display state of the playback window, and the display state can include full-screen and non-full-screen states. In the method, the layer hole-digging operation is decoupled from the browser compositing and the returned coordinates are converted based on the window’s display state, thereby reducing browser playback limitations while ensuring the correct display of media image, thus enhancing the user experience. The display apparatus 200 can implement different functions based on the applications or services configured in the application layer as shown in FIG. 3. In some embodiments, the display apparatus 200 can display different browser pages based on browser applications.

[0159] In other words, the user interface presented by the display apparatus can include browser pages (such as the aforementioned target pages) generated by the browser, these pages can include elements such as text, images, and videos. Taking the scenario of playing a video in a browser as an example, when the display apparatus plays a video, the webpage elements are layered into a picture layer, while the video elements areVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P layered into a video layer. The picture layer is rasterized to generate a texture, while the video layer is decoded by the media module to generate video frame textures. These two layers are combined into a composite frame within the render of the browser kernel and then displayed on the screen after compositing. Meanwhile, the video layer can feed back the position information of the video it carries to the display apparatus’s backend, enabling the backend to utilize its own hardware decoding capabilities to decode and display the video. During this process, the display apparatus can employ layer hole-digging technology to set the corresponding regions of the layers below the video layer to be transparent, ensuring that the video played by the backend is directly visible.

[0160] However, the position information fed back by the video layer to the backend is based on the coordinate system of the browser page. If the browser page is not displayed in full screen on the display apparatus, this can result in the video not being correctly displayed; furthermore, the aforementioned layer hole-digging technology can rely on the browser’s compositing mechanism. When the browser’s compositing method is incompatible with the layer hole-digging technology, the hole-digging effect becomes ineffective, thereby affecting the video playback quality of the display apparatus and reducing the user experience.

[0161] In some embodiments, when the display apparatus 200 starts the browser application, the browser kernel can first parse the content of the browser page (webpage) and convert the content of the browser page into a format supported by the browser. After the browser kernel completes parsing the browser page, the browser can perform layering processing on webpage elements in the CC (Compositing and Constant) Tenderer. For example, the webpage elements, e.g., texts and images are layered into the picture layer, while video elements are layered into the video layer. The picture layer can undergo rasterization to generate a tile texture.

[0162] For the media playback process in the browser application, in some embodiments, when the browser application plays media (i.e., video), the browser can layer the video elements into a video layer and layer other elements into a picture layer. The picture layer can undergo rasterization to generate a tile texture, while the video layer can generate a video frame texture after video decoding by the media module. Then, during the drawing process on the CC tenderer side, the two textures are combined into a compositor frame and sent to the browser side for compositing. After compositing, the browser page and video can be displayed together on the screen.

[0163] As shown in FIG. 18, in some embodiments, the decoding and displaying by the display apparatus 200 are based on the chip decoding capabilities of the processor 250. Therefore, the media playback of the browser application can be achieved through browser layer hole-digging operations. That is, the video layer is not decoded through the media module but directly hands over to the backend of the display apparatus 200 for decoding and displaying the video frame. The video layer transmits the calculated display region of the media image back to the backend. Then, the display apparatus 200 can use the layer hole-digging operation to make the corresponding region of the layer below the video layer (the display layer with a higher display level than the video layer) transparent, thereby revealing and displaying the video played by the backend.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0164] As shown in FIG. 19, in some embodiments, when the display apparatus 200 performs a hole-digging operation in the display layer, it can construct a transparent DrawQuad (drawing a quadrilateral) to achieve hole-digging in the same-layer RenderPassQuad (rendering channel quadrilateral), and pass the lower-layer hole-digging flag to the compositing phase to achieve the hole-digging operation in the display layer. The compositing phase can use different compositing methods, such as glRenderer or skiaRenderer. Different compositing methods correspond to different operation methods. For example, glRenderer can use gl commands, while skiaRenderer can use canvas methods. Therefore, the hole-digging operation in display layer needs to be set according to the compositing method. If the browser’s compositing method does not match, the hole-digging operation in display layer can fail, resulting in the media image not being correctly displayed.

[0165] Additionally, in the decoding and displaying performed by the backend of the display apparatus 200 shown in FIG. 18, the media image position is determined based on the coordinates returned by the video layer. When the playback window of the browser application is displayed in full-screen mode, the coordinate system of the browser page is fully mapped to the screen coordinate system. The display apparatus 200 can then correctly display the media image at the appropriate position based on the coordinates returned by the video layer, allowing the media image to be properly displayed through the hole-digging region. However, when the playback window of the browser application is not displayed in full-screen mode, although the browser page can be correctly displayed in the playback window via the browser’s zoom function, the coordinates transmitted back to the backend of display apparatus 200 cannot be adapted to the playback window, leading to the problem such as offset display positions and obstruction for the media image, thereby affecting the user’s viewing experience.

[0166] Therefore, some embodiments of the present application provide a display apparatus 200 that can be applied to playback windows in different states, enabling the returned coordinates of the media image to be more accurately matched to windows in various display states, thereby ensuring the correct display of the media image. Furthermore, the display apparatus 200 provided by the embodiments of the present application can perform hole -digging operations in the display layer without relying on browser compositing. By decoupling the hole-digging operations during media playback from compositing by browser, the limitations of hole-digging operations can be improved.

[0167] As shown in FIG. 21, in some embodiments, the display 260 can be configured to display a user interface, and at least one processor 250 can be configured to execute computer instructions to cause the display apparatus to perform the following steps.

[0168] S2101: the display apparatus can receive a media playback command sent by the user via a remote control or other device.

[0169] S2102: in response to the media playback command, the browser page in the browser is obtained.

[0170] S2103, the display of the display apparatus is instructed to display the playback window.

[0171] S2104, the display is controlled to display the playback window, which is used to display the browser page, which can include the media image and at least two display layers.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0172] S2105, a target region of the display layer is calculated. The target region is the region of the media image overlapping with the display layer.

[0173] S2106, the display layer is instructed to be drawn.

[0174] S2107, the display layer is drawn to set the target region in the display layer as a transparent region.

[0175] S2108, the media image is decoded based on the returned coordinates to display the media image in the display region corresponding to the returned coordinates. The returned coordinates are determined based on the display state of the playback window, and the display state can include full-screen mode and non-fullscreen mode.

[0176] S2109, the media image is sent to the display apparatus.

[0177] S2110, the media image and the display layer are displayed on the display apparatus.

[0178] As shown in FIGS. 20 to 21, in some embodiments, the display apparatus 200 can include a display 260 and at least one processor 250. The display 260 is coupled to the processor 250, and the display 260 can be configured to display images and / or a user interface. The processor 250 can be configured to execute computer instructions to cause the display apparatus to perform the following steps.

[0179] S2001: in response to a media playback command, the display is controlled to display a playback window, where the playback window is used to display a browser page, and the browser page can include a media image and at least two display layers.

[0180] The processor 250 of the display apparatus 200 can receive control commands input by the user via different interaction methods, such as touch operations, commands input via the control device 100 of the display apparatus 200, voice-input interaction commands, etc. In some embodiments, the command for playing media is represented as the media playback command, which is used to instruct the display apparatus 200 to decode and play specific media. For example, the media playback command can be input from specific playback controls displayed based on the browser application. When the user performs a selected operation on the playback control, the display apparatus 200 can automatically generate the media playback command corresponding to the media associated with the playback control.

[0181] Upon receiving the media playback command, the processor 250 of the display apparatus 200 can, in response to the media playback command, control the display 260 to display the playback window corresponding to the media. The playback window is used to display the browser page, which can include the media image and at least two display layers. In some embodiments, the display layers of the browser page include a target layer (i.e., the video layer used to display the video elements) for displaying the media image, as well as other display layers (i.e., picture layer) that are not used to display the media image.

[0182] In some embodiments, the display layers can include a fust layer and a second layer. The first layer can include the target layer (i.e., the video layer) for displaying the media image and the display layer with a higher display level than the target layer. The second layer can include the display layer with a lower display level than the target layer. The display level refers to the level relationship among display layers when they areVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P displayed on the foreground (i.e., the screen). When there is the overlapping region, display layers with higher display levels will overlay display layers with lower display levels.

[0183] For example, as shown in FIG. 22, the target layer 2201 is a display layer used to display video elements. The target layer 2201 itself and display layers with higher display levels than the target layer 2201 are the first layer 2202, while display layers with lower display levels than the target layer 2201 are the second layer 2203.

[0184] S2002: the target region of the display layer is calculated, where the target region is the region of the media image overlapping with the display layer.

[0185] The processor 250 of the display apparatus 200 can, in response to the media playback command, display the playback window and calculate the region of each display layer in the browser page overlapping with the media image based on the browser page to be displayed in the playback window. For ease of description, each region of the display layer overlapping with the media image is referred to as the target region in the embodiments of the present application.

[0186] As shown in FIG. 23, due to the different sizes and display positions of different layers, in some embodiments, when the processor 250 of the display apparatus 200 calculates the target region of each display layer, it can first calculate the first position region of the media image based on the target layer (S2301), and then calculate the second position region of the media image based on the first position region and the first layer (S2302). The first position region is the position region of the media image in the screen coordinate system, and the second position region is the position region of the media image in the content coordinate system of the first layer. Then, the intersection region between the second position region and the bounding box corresponding to the first layer is detected to obtain the target region of the first layer (S2303).

[0187] In some embodiments, since the display level of the second layer is lower than the display level of the target layer (video layer), the second layer does not occlude the target layer. Therefore, when calculating the target region, the processor 250 of the display apparatus 200 can calculate only the target region of the first layer without calculating the target region of the second layer, thereby saving system resource consumption in the display apparatus 200.

[0188] For the calculation of the first position region, in some embodiments, the processor 250 of the display apparatus 200 can obtain the first transformation matrix of the target layer and the third position region of the media image. The first transformation matrix is used to convert coordinates in the content coordinate system of the target layer to coordinates in the screen coordinate system, and the third position region is the position region of the media image in the content coordinate system of the target layer. Then, the display apparatus 200 can extract the coordinate matrix of the third position region to obtain the first position region through the converted coordinate matrix.

[0189] For the calculation of the second position region, in some embodiments, the processor 250 of the display apparatus 200 can obtain the inverse matrix of the second transformation matrix of the first layer. The second transformation matrix is used to convert coordinates in the content coordinate system of the first layerVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P to coordinates in the screen coordinate system. Then, the display apparatus 200 can extract the coordinate matrix of the first position region again and convert the coordinate matrix of the first position region using the inverse matrix of the second transformation matrix to obtain the second position region through the converted coordinate matrix.

[0190] In other words, the processor 250 of the display apparatus 200 can determine the position and size of the media image on the display apparatus screen based on the coordinates of the media image in the screen coordinate system, and then convert the position and size of the media image on the screen into the position and size in the coordinate system of the display layer. Then, using the position and size in the coordinate system of the display layer, the media image is intersected with the bounding box (bound) of the display layer to determine the target region where a hole-digging operation is required in the display layer, enabling the media image to be correctly displayed in the target region of the display layer.

[0191] For example, the display process for display apparatus 200 involves the content coordinate system, drawing coordinate system, and screen coordinate system. The content coordinate system is primarily used for drawing the layer itself, the drawing coordinate system is used for drawing relationships between layers, and the screen coordinate system is used for positioning layer content on the screen. The screen coordinate system is the same for all layers, so the display apparatus 200 first needs to obtain the coordinate sizes of the video layer in the screen coordinate system, then can convert them to the content coordinate system of the corresponding layer.

[0192] That is, as shown in FIG. 24, the processor 250 of display apparatus 200 can first use the full-screen transformation matrix (first transformation matrix) of the video layer to obtain the position region of the video in the screen coordinate system; then can use the inverse matrix of the full-screen transformation matrix of the first layer (a display layer with a lower display level than the video layer) to obtain the corresponding position region of the video in the content coordinate system of the current first layer; and then can intersect the obtained position region with the bound of the first layer to obtain the hole-digging region (i.e., the target region) in the first layer.

[0193] S2003 : the display layer is drawn and the target region within the display layer is set as a transparent region.

[0194] After the processor 250 of the display apparatus 200 calculates the target regions for each display layer, it can draw the display layers based on the target regions, and set the corresponding target regions of the display layers as transparent regions, allowing the media image to be displayed through the transparent regions, thereby completing the display and playback of the media image.

[0195] In some embodiments, when the processor 250 of the display apparatus 200 draws the display layers, it can determine the drawing order based on the display levels of the display layers. The drawing order of the second layer precedes that of the first layer, meaning that display layers with higher display levels have later drawing orders, while those with lower display levels have earlier drawing orders. For example, the display apparatus 200 can use the ActiveLayerTreelml (layer tree) to recursively traverse the entire layer tree startingVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P from the root layer. Therefore, the hole-digging operation (i.e., DrawQuad operation) can be uniformly performed in the video layer and the subsequent first layer according to the drawing order.

[0196] After the processor 250 of the display apparatus 200 determines the drawing order of the display layers, it can draw the display layers in the drawing order using the browser thread’s Tenderer (e.g., the CC tenderer). When the display layer being drawn is the second layer, the Tenderer can draw the second layer; when the display layer being drawn is the first layer, the Tenderer can draw the first layer and set the target region of the first layer as a transparent region. For example, as shown in FIG. 25, during the CC rendering phase, the display apparatus 200 can determine the drawing order based on the nodes of the layer tree and draw the browser page according to the drawing order. During the drawing process, the DrawQuad operation is performed on the video layer and display layers whose drawing order follows the video layer to set their corresponding target regions as transparent regions.

[0197] It should be noted that the “lower layer” referred to in FIG. 25 refers to the lower layer of the layer tree, not the lower layer in terms of display level. Display layers closer to the lower layer in the layer tree correspond to higher display levels.

[0198] Therefore, in some embodiments, when the processor 250 of the display apparatus 200 sets the target region as a transparent region, it can generate a unit drawing command (i.e., a DrawQuad command) based on the target region associated with the fust layer, and send the unit drawing command to the first layer. The Tenderer then can respond to the unit drawing command, detect the target region in the first layer, and set the transparency value of the target region in the first layer to the target value. The target value is less than the transparency value of the media image and can be transparent or nearly transparent. For example, the Alpha value of the target region can be set to 0 to set the target region as a transparent region.

[0199] It should be understood that the method of setting the target region as a transparent region in the embodiments of the present application can be varied, and the present application does not limit this aspect.

[0200] S2004: the media image is decoded based on the returned coordinates to display the media image in the display region corresponding to the returned coordinates. The returned coordinates are determined based on the display state of the playback window, and the display state can include full-screen mode and non-fullscreen mode.

[0201] The processor 250 of the display apparatus 200 can draw the various display layers of the browser page, and set the target region of the display layer to be transparent while drawing,. Meanwhile, it can decode the video based on the coordinates returned to the backend by the video layer, causing the video to be displayed at the appropriate position. The coordinates returned by the video layer are coordinates corresponding to the full-screen display state of the playback window. If the playback window is in the nonfull-screen display state, the display apparatus 200 can directly decode the video based on the coordinates returned by the video layer, which can cause the media image to be shifted, as the playback window 1201 and the media image 1202 shown in FIG. 26, where the media image 1202 and the playback window can exhibit significant offset.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0202] Therefore, in some embodiments, the processor 250 of the display apparatus 200 can monitor the display state of the playback window and apply different processing strategies to the returned coordinates based on the different display states. The display state of the playback window can include the full-screen state and non-full-screen state.

[0203] As shown in FIG. 27, in some embodiments, the display state of the playback window is monitored (S2700). If the processor 250 of the display apparatus 200 detects that the display state of the playback window is full-screen state, the coordinate matrix of the media image in the page coordinate system is marked as the returned coordinates, and the returned coordinates are sent to the backend for processing (S2701), where the page coordinate system is the logical coordinate system of the browser page. That is, when the playback window is in full-screen mode, the display apparatus 200 does not need to process the coordinates returned by the video; it can directly decode and display the video according to those coordinates.

[0204] However, if the processor 250 of display apparatus 200 detects that the display state of the playback window is the non-full-screen state, it can convert the coordinate matrix of the media image in the page coordinate system into a target matrix (S2702), mark the target matrix as the returned coordinates, and send the returned coordinates to the backend for processing (S2703). That is, when the playback window is not displayed in full-screen mode, the display apparatus 200 needs to convert and adapt the coordinates returned by the video to ensure that the media image is correctly displayed in the playback window.

[0205] For non-full-screen playback scenarios, in some embodiments, the processor 250 of the display apparatus 200 can obtain the first size information of the media image in the page coordinate system and the second size information of the playback window in the screen coordinate system. It then can analyze the mapping relationship between the first size information and the second size information, and generate the first conversion formula based on the mapping relationship.

[0206] The first conversion formula represents the process of converting the first coordinate matrix to the second coordinate matrix, where the first coordinate matrix is the coordinate matrix of the media image in the page coordinate system, and the second coordinate matrix is the coordinate matrix of the media image in the screen coordinate system. Then, the second conversion formula is generated using the second coordinate matrix and the mapping relationship. The second conversion formula represents the process of converting the second coordinate matrix into the third coordinate matrix, where the third coordinate matrix is the coordinate matrix in the page coordinate system corresponding to the second coordinate matrix in the screen coordinate system.

[0207] In some embodiments, the first size information can be obtained based on the object properties of browser kernel in the display apparatus 200, and the second size information can be obtained through mojom (module-to-module communication protocol) communication in the display apparatus 200.

[0208] For example, the currently returned video coordinate sizes are (x0,y0, w0, h0), and the converted coordinate sizes are (xt,yt, wt, ht). Here, x and y represent the coordinate positions of the video in the logicalVI0418P-WO-0004 PCT-application - TextDocket No. TD250503946P coordinate system, and w and h represent the width and height of the video in the page logical size coordinate system. The logical sizes (Wj, H of the browser page can be obtained through the object properties of browser kernel (such as window.. innerWidth / window.innerHeight). The screen sizes (Ws, Hs) can be obtained through browser startup parameters. Through Mojom communication, the coordinate sizes (Xw, Yw, Ww, Hw) in the screen coordinate system at the time of creating the playback window can be obtained. Since the logical sizes of the browser page are fully mapped to the sizes of the playback window, the display apparatus 200 can first obtain the coordinate sizes (xs, ys, ws, hs) of the video in the screen coordinate system The fust conversion formula is as follows:

[0209] Then, the display apparatus 200 can convert it into coordinate sizes (xt,yt, wt, ht) in the logical size coordinate system. The second conversion formula is as follows:

[0210] Based on the above first conversion formula and second conversion formula, it can obtain the following:

[0211] By marking (xt, yt, wt, ht) as the returned coordinates, the video can be correctly displayed in the appropriate position.

[0212] Therefore, in some embodiments, the processor 250 of the display apparatus 200 can generate a target tconversion formula based on the first conversion formula and the second conversion formula. The target conversion formula is used to represent the process of converting the first coordinate matrix into the third coordinate matrix.

[0213] For example, based on the above example, the target conversion formula can be the following conversion formula:VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0214] This conversion formula is used to represent the relationship between (x0,y0, w0,h0) and (xt,yt, wt,ht).

[0215] After the display apparatus 200 generates the target conversion formula, it can calculate the third coordinate matrix based on the first coordinate matrix and the target conversion formula, then mark the third coordinate matrix as the target matrix to serve as the final returned coordinates. In this way, the display apparatus 200 can convert the coordinates returned by the video layer based on the display state of the playback window, then can decode and display the video according to the converted coordinates, enabling the video to be displayed at an appropriate position within the video layer, and ensuring that the video can correctly appear through the transparent region of each display layer, thereby enhancing the video’s display quality.

[0216] For example, as shown in FIG. 28, the display apparatus 200 can decode and display the media based on the converted returned coordinates, enabling the media display 2802 to appear in the appropriate position even when the playback window 2801 is not in full-screen mode. Compared to the media display effect shown in FIG. 26, the display method provided by the embodiment of the present application can offers superior display quality (as shown in FIG. 28).

[0217] In some embodiments, after the processor 250 of the display apparatus 200 controls the display of the playback window on the display 260, it can either parallel-process the calculation program for the returned coordinates and the calculation program for the target region, or serial-process the calculation program for the returned coordinates and the calculation program for the target region. During serial processing, the returned coordinates can be calculated first followed by the target region, or the target region can be calculated first followed by the returned coordinates. The present application does not impose any restrictions on this.

[0218] Based on the above display apparatus 200, some embodiments of the present application also provide a method for displaying media image, which can be performed by the display apparatus 200 provided in the above embodiments. As shown in FIG. 20, the display method can include the following program steps.

[0219] S2001: in response to a media playback command, the display is controlled to display a playback window, where the playback window is used to display a browser page. The browser page can include a media image and at least two display layers.

[0220] S2002: the target region of the display layer is calculated, where the target region is the region of the media image overlapping with the display layer.

[0221] S2003 : the display layer is drawn and the target region within the display layer is set as a transparent region.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0222] S2004: the media image is decoded based on the returned coordinates to display the media image in the display region corresponding to the returned coordinates. The returned coordinates are determined based on the display state of the playback window, and the display state can include full-screen mode and non-fullscreen mode.

[0223] Additionally, during the process of loading a page (e.g., a page corresponding to a predefined control, or the aforementioned browser page, etc.), the display apparatus can retrieve data from the server and load the page based on the data returned by the server. During the process of loading the page, the display apparatus can decode the data returned by the server using the software decoding function provided by the browser, and then render the page using the decoding data.

[0224] However, the processor performance of the display apparatus is weaker compared to computers and mobile phones, and memory resources are also more limited. Therefore, if the display apparatus relies solely on the software decoding function provided by the browser to decode the data returned by the server during operation, it will increase the utilization of the processor and memory, thereby leading to a decline in the performance of the display apparatus and a reduction in decoding efficiency.

[0225] To address this problem, some embodiments of the present application provide a display apparatus and a page loading method. When the display apparatus receives data to be decoded from the server, it can fust determine whether the decoding format of the data to be decoded is within the hardware decoding format range of the display apparatus. If the decoding format is within the hardware decoding format range, the page loading data is decoded based on the hardware decoding mode to obtain the first video decoding data. It then can control the display to display the media image corresponding to the first video decoding data. By appropriately utilizing the hardware decoding function of the display apparatus, the pressure on the processor and memory resources can be alleviated, enabling the display apparatus to maintain high performance during page loading and improve page loading speed.

[0226] In some embodiments, after starting the display apparatus 200, the user can select a predefined control within the display apparatus to start the page corresponding to the predefined control. In some embodiments, the page corresponding to the predefined control is a pre-set image, so no loading is required. In other embodiments, the page corresponding to the predefined control is a page that needs to be loaded, such as in a cloud gaming scenario where the user selects cloud gaming-related controls via input methods like a remote control or voice commands. The display apparatus 200 can, in response to the user’s input selection command, display the page corresponding to the predefined control.

[0227] It should be understood that the display apparatus 200 needs to request page loading data from the server, load the page using the page loading data returned by the server, and display it to the user. In some embodiments, the display apparatus 200 can establish a first process (render) using an embedded browser. The first process can create a WebRTC instance and receive the page loading data returned by the server via WebRTC. The page loading data can include audio data and video data. Then, the first process can decode the page loading data using the software decoding mode provided by the browser to obtain decoding data, andVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P render the page based on the decoding data to display the media image corresponding to the page loading data to the user.

[0228] However, decoding the page loading data using the software decoding mode provided by the browser can consume CPU and memory resources, etc., of the display apparatus. However, the CPU and memory resources of display apparatuses are more limited compared to high-performance computers, smartphones, and other terminal devices. Additionally, the display apparatuses require extensive interface interactions and data loading processes during operation. Therefore, decoding page loading data solely through software decoding modes can easily lead to performance degradation of the display apparatus due to excessive hardware resource consumption, while also impacting page loading speed and degrading the user experience.

[0229] To address the above problem, the display apparatus provided by the present application can include: a display 260, a communicating device 220, and at least one processor 250. The communicating device can be configured to establish a communication connection with a server. As shown in FIG. 29, the processor 250 can be configured to execute computer instructions to cause the display apparatus to perform the following steps.

[0230] S2901, the processor can receive a page access command.

[0231] S2902: the server can, in response to the received page access command, send a data request command to the server, and the data request command is used to obtain page loading data.

[0232] S2903 : the server can send the page loading data to the processor.

[0233] S2904: the processor can send the page loading data to the hardware decoding module.

[0234] S2905 : the hardware decoding module can perform hardware decoding on the page loading data to obtain the first video decoding data.

[0235] S2906: the hardware decoding module can send the first video decoding data to the hardware display module.

[0236] S2907, the hardware display module can display the media image.

[0237] In some embodiments, when the user selects a predefined control on the user interface via an input device such as a remote control or touchpad, the processor 250 can receive the user’s page access command and can control the communicating device 220 to send a data request command to the server, thereby displaying the media image corresponding to the page loading data to the user via the page loading data returned by the server.

[0238] It should be understood that when the processor 250 receives the page loading data returned by the server, it needs to decode the page loading data to obtain decoding data, and then the media image is displayed based on the decoding data. Therefore, the processor 250 can receive page loading data through the native browser of the display apparatus and perform decoding processing on the page loading data.

[0239] It should be noted that, as shown in FIG. 31, in some embodiments of the present application, a hardware decoding preprocessing module is added based on the browser’s basic architecture, which allows the browser to transmit the page loading data to the hardware decoding interface via the hardware decoding preprocessing module when receiving the page loading data, thereby processing part of the data based on theVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P hardware decoding mode of the display apparatus 200 to improve page loading efficiency and maintain the operational performance of the display apparatus 200.

[0240] In this way, when the browser receives page loading data transmitted back from the server, the processor 250 can first determine the decoding data format supported by the page loading data, and then decide which method to use for decoding the page loading data.

[0241] When the communicating device receives the page loading data from the server, the decoding data format of the page loading data and the hardware decoding format range supported by the display apparatus are obtained.

[0242] In some embodiments, when the processor 250 receives the page loading data, it can obtain the decoding data format of the page loading data and the hardware decoding format range supported by the display apparatus 200. By comparing the decoding data format with the hardware decoding format range, it can be determined whether the display apparatus 200 can decode the page loading data using the hardware decoding mode.

[0243] If the decoding data format is within the hardware decoding format range, the page loading data is decoded using the hardware decoding mode to obtain the first video decoding data.

[0244] In some embodiments, if the decoding data format of the page loading data is within the hardware decoding format range supported by display apparatus 200, the processor 250 can send the page loading data to the hardware decoding interface via the hardware decoding preprocessing module to decode the page loading data through the hardware decoding interface, thereby obtaining the first video decoding data.

[0245] It should be noted that the hardware decoding function integrated into the display apparatus 200 is designed for page loading data within the hardware decoding format range. It can process the page loading data using pre-integrated hardware decoding circuits, thus avoiding the need to occupy additional system resources of the display apparatus 200 and achieving high decoding efficiency. Therefore, the efficiency of decoding page loading data using the hardware decoding mode is higher than that of decoding page loading data using the software decoding mode.

[0246] The display is controlled to display the media image corresponding to the first video decoding data.

[0247] After receiving the first video decoding data, the processor 250 can control the display to display the media image corresponding to the first video decoding data. In some embodiments, in the application scenario where the user starts a cloud game, the utilization rate of CPU (Central Processing Unit) of the display apparatus 200 needs to reach 75%, the page loading latency is 1-2 seconds, and the screen frequently can stutter before the cloud gaming scenario is fully loaded, requiring 3-5 seconds to respond to input commands from devices such as remote controls or handles.

[0248] In other embodiments, as shown in FIG. 31, by adding the hardware decoding preprocessing module to the decoding process of the display apparatus 200, and adding a step to determine whether the decoding data format supported by the page loading data is within the hardware decoding format range supported by the display apparatus 200, the page loading data can be decoded using a software -hardware collaborative approach.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946PIn the application scenario where the user starts a cloud game, the utilization rate of CPU of the display apparatus 200 is reduced to 40%, the page loading latency is reduced to 120ms, and the device can respond to user input commands via input devices without latency.

[0249] It should be understood that in some embodiments of the present application, the method of decoding page loading data via the browser using the software decoding method is retained, as shown in FIG. 30. In some embodiments, after the processor performs the comparison between the decoding data format and the decoding format range, it can further include: if the decoding data format is outside the decoding format range, the page loading data is decoded based on the software decoding mode to obtain the second video decoding data.

[0250] In FIG. 30, the following steps are executed.

[0251] S3001: he processor can receive a page access command.

[0252] S3002: the server can, in response to the received page access command, send the data request command to the server, and the data request command is used to obtain page loading data.

[0253] S3003: the server can send the page loading data to the processor.

[0254] S3004: if the decoding data format is outside the decoding format range, the processor can send the page loading data to the software decoding module.

[0255] S3005: the software decoding module can decode the page loading data based on the software decoding mode to obtain the second video decoding data.

[0256] S3006: the software decoding module can send the second video decoding data to the software display module.

[0257] S3007: the media image is displayed at the graphics layer.

[0258] The display is controlled to display the media image corresponding to the second video decoding data.

[0259] As shown in FIG. 31, the processor 250 can determine whether the page loading data can support hardware decoding mode through a predefined determination interface, i.e., whether the data decoding format of the page loading data is within the decoding format range. When the determination interface determines that the data decoding format of the page loading data is outside the decoding format range, the processor 250 can decode the page loading data based on the software decoding mode provided by the browser to obtain the second video decoding data.

[0260] In FIG. 31, the following steps are executed.

[0261] S3101 : the processor 250 can obtain the page loading data.

[0262] S3102: the processor 250 can determine whether the page loading data can support hardware decoding mode through the predefined determination interface.

[0263] S3103 : the processor 250 can determine whether the data decoding format of the page loading data is within the decoding format range. If the decoding data format is within the decoding format range, S3104 is performed; and if the decoding data format is outside the decoding format range, S3107 is performed.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0264] S3104: the processor 250 can send the page loading data to the hardware decoding module via the hardware decoding preprocessing module.

[0265] S3105: the page loading data is decoded using the hardware decoding module to obtain the first video decoding data.

[0266] S3106: the media is displayed at the video layer.

[0267] S3107: the processor 250 can decode the page loading data using the software decoding mode provided by the browser to obtain the second video decoding data.

[0268] S3108: the media is displayed at the graphics layer.

[0269] It should be noted that the media images corresponding to the first video decoding data and the second video decoding data are the same, both being the media images corresponding to the page loading data returned from the server. Here, the terms “first video decoding data” and “second video decoding data” are used solely to distinguish between decoding data obtained via hardware decoding mode and decoding data obtained via software decoding mode, and do not impose any restrictions on the type or content represented by the decoding data.

[0270] After obtaining the second video decoding data, the processor 250 can control the display 260 to display the media image corresponding to the second video decoding data. In this way, the display apparatus 200 can flexibly adjust the decoding method based on the format of the decoding data in the page loading data, thereby releasing system resources and improving the stability of the operation of the display apparatus 200. Additionally, when the page loading data can support hardware decoding, it can also improve page loading efficiency.

[0271] It should be noted that when the display apparatus 200 decodes the page loading data using the hardware decoding mode, as the hardware decoding process proceeds, some first video decoding data is generated, and partial media image corresponding to some first video decoding data can be displayed at the video layer. However, this partial media image is unclear, and from the user’s perspective, it can manifest as stuttering, frozen frames, or other issues, thereby affecting the user experience when viewed by the user.

[0272] Therefore, in some embodiments, the processor can perform decoding of the page load data based on the hardware decoding mode to obtain the first video decoding data, and can also be configured to execute computer instructions to cause the display apparatus to set the video layer and the graphics layer.

[0273] When the hardware decoding interface performs hardware decoding on the page loading data, the display apparatus is controlled to overlay the video layer and graphics layer, and display partial media image with an area smaller than the video layer in the video layer, and display an blocked image in the graphics layer. The partial media image can correspond to the first video decoding data obtained through the hardware decoding mode.

[0274] It should be noted that the hardware decoding interface is a hardware module integrated into the CPU. The hardware module corresponding to the hardware decoding interface can directly output the decoding data to the video layer for display after decoding. Through customized design of the hardware module, the pageVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P loading data can not only be decoded but also adaptively optimized during the decoding process, enabling direct display in the video layer and thereby improving page loading efficiency.

[0275] In some embodiments, the video layer is the display layer corresponding to the hardware decoding interface, and the graphics layer is the display layer used to display the blocked image. Therefore, as shown in FIG. 32, the display level of the graphics layer can be set higher than that of the video layer. Additionally, the display mode of the video layer and graphics layer can be set to overlay display. In this way, when displaying partial media image in the video layer, the blocked image in the graphics layer can cover the partial media image displayed in the video layer.

[0276] It should be noted that in some embodiments of the present application, by adding the hardware decoding preprocessing module, the page loading data can be migrated to the hardware decoding interface for decoding. To ensure that the overall runtime architecture of the browser is not affected by the hardware decoding preprocessing module, as shown in FIG. 33, some embodiments of the present application further include a data consumption module.

[0277] In some embodiments, the processor executes decoding of the page loading data based on the hardware decoding mode to obtain the first video decoding data, and can also be configured to execute computer instructions to cause the display apparatus to perform the following steps.

[0278] S3301, the page loading data is copied to obtain page copy data identical to the page loading data.

[0279] S3302, the data consumption module can perform packet loss processing on the page copy data received.

[0280] The page copy data is sent to the data consumption interface via the first process to obtain the size parameters contained in the page copy data through the data consumption interface.

[0281] The display size of the blocked image is set based on the size parameters; and the display size of the blocked image is the same as the display size of the media image corresponding to the first video decoding data.

[0282] In some embodiments, the first process can be a render process created by the browser. When the display apparatus 200 requests page loading data from the server via the communicating device 220, the processor 250 can create the first process via the browser. Furthermore, a WebRTC instance is created via the first process, so the WebRTC can receive the page loading data returned by the server via the communicating device 220. The WebRTC instance then processes the page loading data, such as by forwarding it.

[0283] When processing the page loading data, the WebRTC instance can copy the page loading data to obtain the page copy data, then send the page copy data to the data consumption interface, and obtain the size parameters contained in the page copy data through the data consumption interface.

[0284] The data consumption interface can refer to the data consumption module created by the browser. The data consumption module can obtain information such as the data format and size parameters of the page copy data from the page copy data. The data consumption module can output data to the graphics layer, but the graphics layer is primarily used to display a blocked image to block partial media image in the video layer.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946PTherefore, the data consumption module only needs to perform packet discard processing on the received page loading data. This allows the graphics layer to continuously display a blank screen as the blocked image to block partial media image in the video layer.

[0285] In this way, during the hardware decoding process on display apparatus 200, the data consumption module can maintain the browser’s operational architecture, allowing the browser to continue using the initial software decoding control logic. This does not affect the browser’s operational architecture and thus does not impact other functions such as interface display or data decoding, thereby helping to maintain the operational stability of display apparatus 200.

[0286] Furthermore, the page copy data and page loading data are identical. Therefore, the media display sizes, data formats, and other information contained in the page copy data are the same as those in the page loading data. Consequently, the WebRTC instance can use the size parameters obtained from the data consumption module to control the display sizes of the blank screen in the graphics layer. This ensures that the display sizes of the blank screen identical to those of the media image in the video layer, thereby achieving the blocking for the video layer.

[0287] In some embodiments, the cloud gaming page accessed by the user is a small window page. Therefore, when the display apparatus 200 requests the page loading data from the server via the communicating device 220, it can request the page loading data that matches the size of the small window page. As a result, the page loading data returned by the server will also include display size. Therefore, when the data consumption module receives the page copy data, it can obtain the display size contained in the page copy data, and the WebRTC instance can obtain the display size information contained in the page copy data from the data consumption module. In this way, the WebRTC instance can control the display size of the blank screen in the graphics layer using the obtained display size information to achieve blocking for the video layer.

[0288] By controlling the display size of the blank screen in the graphics layer, issues such as excessive blank screen size causing additional blocking for other user interface elements in various playback scenarios (e.g., small window playback, picture -in-picture playback) can be avoided, thereby enhancing the user experience.

[0289] It should be understood that the page copy data is sent by the WebRTC instance to the data consumption module to control the display size of the blank screen in the graphics layer. The page loading data must be sent to the hardware decoding interface to obtain the first video decoding data by decoding using the hardware decoding interface.

[0290] In some embodiments, when the processor executes decoding of the page loading data based on the hardware decoding mode to obtain the first decoding data, it can be configured to execute computer instructions to cause the display apparatus to perform the following steps.

[0291] S3303, the page loading data is sent to the hardware decoding interface via the first process.

[0292] S3304, the hardware decoding interface can perform hardware decoding on the page loading data to obtain the first video decoding data.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0293] The display apparatus is controlled to display the media image corresponding to the first video decoding data in the video layer.

[0294] It should be understood that the render process provided by the browser can send the page loading data to the hardware decoding interface, which then perform hardware decoding on the page loading data. When the hardware decoding interface receives the page loading data, it can immediately perform hardware decoding on the page loading data to obtain the first video decoding data. While obtaining the first video decoding data, the hardware decoding interface can output the first video decoding data to the video layer. After the hardware decoding interface has fully decoded the page loading data, the processor 250 can control the display 260 to display the complete media image corresponding to the first video decoding data in the video layer.

[0295] It should be noted that the render process provided by the browser has a lower permission scope, while the hardware decoding interface belongs to a hardware module, which must be called by the browser process provided by the browser. Therefore, in some scenarios, it is difficult for the render process to directly transmit the page loading data to the hardware decoding interface, and the browser process must migrate the page loading data to the hardware decoding interface.

[0296] As shown in FIG. 34, in some embodiments, the processor executes the first process to send the page loading data to the hardware decoding interface. It can be configured to execute computer instructions to cause the display apparatus to perform the following steps.

[0297] The first communication identifier of the first process and the second communication identifier of the second process are obtained.

[0298] Based on the communication connection relationship determined by the first communication identifier and the second communication identifier, the page loading data is sent to the second process through the first process.

[0299] The page load data is sent to the hardware decoding interface via the second process to enable the hardware decoding interface to decode the page load data.

[0300] In some embodiments, the first process refers to the render process provided by the browser, and the second process refers to the browser process provided by the browser. The render process can support module calls and data transmission related to software decoding function. The browser process has higher permissions than the render process and can support hardware module calls and data transmission. Therefore, when calling the hardware decoding interface, it is necessary to regulate the page loading data and the hardware decoding interface based on the browser process.

[0301] It should be noted that the page loading data is received by the first process through the WebRTC instance, so it is necessary to migrate the page loading data from the first process to the second process. In some embodiments of the present application, the hardware decoding preprocessing module is added to achieve the process of migrating the page loading data from the first process to the second process.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0302] In some embodiments, when the WebRTC instance inputs the page copy data into the data consumption module, it can also input page loading data into the hardware decoding preprocessing module. The hardware decoding preprocessing module can migrate the page loading data from the render process to the browser process using mojo inter-process communication.

[0303] The hardware decoding preprocessing module can determine the communication connection relationship between the browser process and the render process based on their communication identifiers, thereby enabling the migration of page loading data from the render process to the browser process.

[0304] In this way, the processor 250 can further invoke the hardware decoding interface through the browser process to perform hardware decoding processing on the page loading data.

[0305] It should be noted that the specific method of inter-process communication based on mojo is not the focus of the description in the embodiments of the present application, and other methods of inter-process communication can also be applied to the embodiments of the present application. The focus of the description in the embodiments of the present application is on migrating the page loading data from a process with lower permission levels to a process with higher permission levels corresponding to the hardware decoding module, thereby enabling the hardware decoding module to perform decoding processing on the page loading data.

[0306] After the processor 250 migrates the page loading data from the render process to the browser process, the hardware decoding interface can perform hardware decoding processing on the page loading data in the browser process and output the first video decoding data to the video layer. During the decoding process, the video layer is blocked by the blank screen provided by the graphics layer. Upon completion of decoding, the media image in the video layer can be displayed by applying transparency processing to the graphics layer.

[0307] In FIG. 34, the following steps are executed.

[0308] S3401 : the first process can send the page loading data to the hardware decoding preprocessing module.

[0309] S3402: the hardware decoding preprocessing module can enable mojo inter-process communication.

[0310] S3403: the hardware decoding preprocessing module can send the page loading data to the second process via the mojo inter-process communication.

[0311] S3404: the second process can send the page loading data to the hardware decoding module.

[0312] As shown in FIG. 35, in some embodiments, the processor can be configured to execute control commands to cause the display apparatus to perform the following when displaying the media image corresponding to the first video decoding data.

[0313] When the hardware decoding interface completes hardware decoding on the page loading data, the region of the video layer overlapping with the graphics layer is determined based on the display sizes of the video layer and the graphics layer.

[0314] The transparency processing is performed on the portion of the graphics layer corresponding to the overlapping region to display the media image displayed in the video layer.VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P

[0315] It should be understood that determining the region of the video layer overlapping with the graphics layer based on their display sizes means determining the overlapping region based on the display size of the blank screen in the graphics layer and the display size of the media image in the video layer.

[0316] After the WebRTC instance inputs the page copy data into the data consumption module, the WebRTC instance can obtain the display size of the blank screen from the data consumption module.

[0317] Since the display level of the graphic layer is higher than that of the video layer, only the graphic layer needs to be made transparent to display the media image corresponding to the first video decoding data in the video layer.

[0318] It should be noted that the video layer also can correspond to the hardware display module integrated into processor 250. Therefore, the first video decoding data obtained by the hardware decoding module can be directly input into the hardware display module, and the media image corresponding to the first video decoding data can be displayed in the video layer. Based on the display method of the video layer, this can also alleviate the occupancy rate of the graphics layer, thereby saving system resources and improving display efficiency.

[0319] In application scenarios such as cloud gaming, the target page accessed by the user can include both the media image and the corresponding audio playback associated with the media image. Therefore, the page loading data can include not only video data but also audio data. In this way, the processor 250 can display the media images based on the video data and play the corresponding audio based on the audio data.

[0320] In some embodiments, the display apparatus 200 can further include an audio player coupled to the processor, and the page loading data can include audio encoding data. The processor can also be configured to execute computer instructions to cause the display apparatus to perform the following.

[0321] The audio encoding data is sent to the audio decoding interface via the first process to obtain audio decoding data.

[0322] Based on the time stamps of the audio decoding data and the video decoding data, when controlling the display to display the media image corresponding to the first video decoding data, the audio player is controlled to play the audio data corresponding to the time stamp of the media image from all the audio data corresponding to the audio decoding data.

[0323] It should be noted that the audio encoding data is the audio data included in the page loading data, which must be decoded by the display apparatus 200 before it can be played through the music player. The resources consumed during the decoding of audio data are minimal, so software decoding can be used to decode the audio data. However, when ultimately playing the media image corresponding to the first video decoding data, it is necessary to ensure the correspondence between the audio data and the media image.

[0324] In some embodiments, both the audio data and video data are time-stamped, and the decoded audio data and video data obtained through decoding are also time-stamped. Thus, when the processor controls the video layer to play the media data, it can pre-align the decoded audio data and video data based on the time stamps before playing the media data. This allows the media image to be played back simultaneously with theVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P corresponding audio data, thereby preventing the problem such as degraded user experience caused by audiovisual desynchronization.

[0325] It should be noted that since the hardware decoding module is a circuit designed for specific data formats, its decoding process can include optimized data processing, and the hardware decoding module can be directly connected to the hardware display module. Therefore, the hardware display module can display the media image corresponding to the decoding data in the video layer based on the decoding order of the hardware decoding module. However, when the display apparatus 200 decodes data using the software decoding function, it needs to optimize the graphics and render the page based on the optimized data to obtain the media image corresponding to the decoding data.

[0326] In some embodiments, when the processor renders the page based on the second video decoding data, it is configured to execute computer instructions to cause the display apparatus to perform the following.

[0327] When decoding the page loading data based on the software decoding mode, the media image is rendered based on the second video decoding data through the first process.

[0328] The display apparatus is controlled to play the media image rendered from the second video decoding data.

[0329] It should be understood that the processor 250 can execute the software decoding function through the first process, i.e., through the render process. That is, when the WebRTC instance created by the render process receives the page loading data, it can invoke the software decoding function to decode the page loading data, thereby obtaining the second video decoding data.

[0330] After obtaining the second video decoding data, the render process creates a graphics rendering module and can use the graphics rendering module to render and play the second video decoding data. Thus, both the software decoding and playing the media images corresponding to the page loading data can be executed by the render process. Even if the display apparatus 200 cannot perform hardware decoding on the page loading data, the data can be decoded using the software decoding function provided by the browser to promptly display the media image to the user when the user inputs the page access command.

[0331] Some embodiments of the present application also provide a page loading method, including: in response to the received page access command, sending a data request command to the server, where the data request command is used to obtain the page loading data; upon receiving the page loading data from the server, obtaining the decoding data format of the page loading data and the hardware decoding format range of the display apparatus; if the decoding data format is within the hardware decoding format range, decoding the page loading data using the hardware decoding mode to obtain the first video decoding data; and displaying the media image corresponding to the first video decoding data.

[0332] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present application can be implemented using software in conjunction with necessary general -purpose hardware platforms. Based on this understanding, the implementation methods in the embodiments of the present application, or the portions that contribute to the prior art, can be embodied in the form of a softwareVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P product. This computer software product can be stored on a storage medium, such as ROM / RAM, disk, or optical disc, and can include a series of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present application or certain portions thereof. It should be noted that the above embodiments are provided solely to illustrate the implementation methods of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art will understand that modifications can be made to the implementation methods described in the aforementioned embodiments, or equivalent replacements can be made to some or all of the technical features; such modifications or replacements do not cause the corresponding implementation methods to deviate from the scope of the implementation methods of the embodiments of the present application.

[0333] For the sake of clarity, the above description has been provided in conjunction with specific embodiments. However, the exemplary discussion above is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be derived. The selection and description of the above embodiments are intended to better explain the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of the embodiments suitable for specific applications.

Claims

VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946PWhat is claimed is:

1. A display apparatus, comprising: a display, configured to display images and / or a user interface; at least one processor, configured to execute computer instructions to cause the display apparatus to: in response to a background operation command for a media playback application, hide a target page of the media playback application and detect video elements of the target page, wherein the video elements are used to embed a playback image of first media data; pause playback of the first media data of the video elements; record a playback progress of the first media data; release redundant data from a fust buffer queue, wherein the first buffer queue is used to cache the first media data of the media playback application, the redundant data is the fust media data in the first buffer queue except for reference data, and the reference data is segment data corresponding to the playback progress; record target information based on the redundant data; receive a startup command for the media playback application; and in response to the startup command, start the media playback application based on the target information and the playback progress, and control the display to display the playback image of the first media data based on the video elements of the target page.

2. The display apparatus according to claim 1, wherein the at least one processor is further configured to execute the computer instructions to cause the display apparatus to pause playback of the first media data of the video elements by: iterating through video objects of the video elements; calling a predefined pause function to stop the video objects from playing the first media data; and releasing a playback resource used for playing the first media data.

3. The display apparatus according to claim 2, wherein the at least one processor is further configured to execute the computer instructions to cause the display apparatus to, before recording the playback progress of the first media data: query a current time attribute of the video objects; and parse a current playback position of the first media data based on the current time attribute to generate the playback progress of the first media data.

4. The display apparatus according to claim 1, wherein the at least one processor is further configured to execute the computer instructions to cause the display apparatus to release the redundant data from the first buffer queue by: querying a target key frame based on the playback progress, wherein the target key frame is a key frame, that is closest to a key point, of a second media data in the first buffer queue; querying target segment data comprising the target key frame;VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P marking third media data, a playback position of which is ahead of the target segment data, in the first buffer queue as the reference data; and deleting the fust media data in the first buffer queue except for the reference data.

5. The display apparatus according to claim 1, wherein the at least one processor is further configured to execute the computer instructions to cause the display apparatus to record the target information based on the redundant data by: detecting a cache time and a segment address of the redundant data; and adding the cache time and the segment address to the first buffer queue to record the target information.

6. The display apparatus according to claim 1, wherein the at least one processor is further configured to execute the computer instructions to cause the display apparatus to start the media playback application based on the target information and the playback progress by: revoking a hidden state of the target page; locating the reference data in the first buffer queue based on the playback progress; decoding the reference data in the first buffer queue starting from the playback progress; and rendering the decoded reference data to the video elements to continue displaying the playback image of the first media data through the video elements.

7. The display apparatus according to claim 1, wherein the at least one processor is further configured to execute the computer instructions to cause the display apparatus to, after receiving the startup command for the media playback application: in response to the startup command, read the target information recorded in the first buffer queue; download the redundant data based on the target information; cache the downloaded redundant data in the first buffer queue; and play the first media data based on the first buffer queue.

8. The display apparatus according to claim 7, wherein the at least one processor is further configured to execute the computer instructions to cause the display apparatus to, after receiving the startup command for the media playback application: in response to the startup command, download fourth media data based on the media playback application, wherein the fourth media data is media data a playback position of which is located behind the redundant data; cache the fourth media data in a second buffer queue; and play the first media data based on the second buffer queue and the first buffer queue.

9. The display apparatus according to claim 8, wherein the at least one processor is further configured to execute the computer instructions to cause the display apparatus to: after caching the downloaded redundant data in the first buffer queue, merge the first buffer queue and the second buffer queue to generate a composite buffer queue; read a target playback position of the first media data in the video elements; decode fifth media data in the composite buffer queue based on the target playback position; andVI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P render the decoded fifth media data to the video elements to display a playback image of the fifth media data through the video elements.

10. The display apparatus according to claim 1, wherein the at least one processor is further configured to execute the computer instructions to cause the display apparatus to: in response to an abnormal event being detected, exit playback of the video elements and close the target page based on the media playback application.

11. A resource control method, comprising: in response to a background operation command for a media playback application, hiding a target page of the media playback application and detecting video elements of the target page, wherein the video elements are used to embed a playback image of first media data; pausing playback of the first media data of the video elements; recording a playback progress of the first media data; releasing redundant data from a first buffer queue, wherein the first buffer queue is used to cache the first media data of the media playback application, the redundant data is the first media data in the first buffer queue except for reference data, and the reference data is segment data corresponding to the playback progress; recording target information based on the redundant data; receiving a startup command for the media playback application; and in response to the startup command, starting the media playback application based on the target information and the playback progress, and controlling a display to display the playback image of the first media data based on the video elements of the target page.

12. The resource control method according to claim 11, wherein the pausing the playback of the first media data of the video elements comprises: iterating through video objects of the video elements; calling a predefined pause function to stop the video objects from playing the first media data; and releasing a playback resource used for playing the first media data.

13. The resource control method according to claim 12, further comprising: before the recording the playback progress of the first media data: querying a current time attribute of the video objects; and parsing a current playback position of the first media data based on the current time attribute to generate the playback progress of the first media data.

14. The resource control method according to claim 11, wherein the releasing the redundant data from the first buffer queue comprises: querying a target key frame based on the playback progress, wherein the target key frame is a key frame, that is closest to a key point, of a second media data in the first buffer queue; querying target segment data comprising the target key frame;VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P marking third media data, a playback position of which is ahead of the target segment data, in the first buffer queue as the reference data; and deleting the first media data in the first buffer queue except for the reference data.

15. The resource control method according to claim 1, wherein the recording the target information based on the redundant data comprises: detecting a cache time and a segment address of the redundant data; and adding the cache time and the segment address to the first buffer queue to record the target information.

16. The resource control method according to claim 11, wherein the starting the media playback application based on the target information and the playback progress comprises: revoking a hidden state of the target page; locating the reference data in the first buffer queue based on the playback progress; decoding the reference data in the first buffer queue starting from the playback progress; and rendering the decoded reference data to the video elements to continue displaying the playback image of the first media data through the video elements.

17. The resource control method according to claim 11, further comprising, after the receiving the startup command for the media playback application: in response to the startup command, reading the target information recorded in the first buffer queue; downloading the redundant data based on the target information; caching the downloaded redundant data in the first buffer queue; and playing the first media data based on the first buffer queue.

18. The resource control method according to claim 17, further comprising: after the receiving the startup command for the media playback application: in response to the startup command, downloading fourth media data based on the media playback application, wherein the fourth media data is media data a playback position of which is located behind the redundant data; caching the fourth media data in a second buffer queue; and playing the first media data based on the second buffer queue and the first buffer queue.

19. The resource control method according to claim 18, further comprising, after the caching the downloaded redundant data in the first buffer queue: merging the first buffer queue and the second buffer queue to generate a composite buffer queue; reading a target playback position of the first media data in the video elements; decoding fifth media data in the composite buffer queue based on the target playback position; and rendering the decoded fifth media data to the video elements to display a playback image of the fifth media data through the video elements.

20. The resource control method according to claim 1, further comprising:VI 0418P-WO-0004 PCT-application - TextDocket No. TD250503946P in response to an abnormal event being detected, exiting playback of the video elements and closing the target page based on the media playback application.

Images