Display device and resource control method of background mode

Abstract

The embodiment of the invention provides a display device and a background mode resource control method, and the method can respond to a background operation instruction of a media resource playing application, hide a target page of the media resource playing application, and detect video elements of the target page. And pausing the playing of the media resource data of the video element, and recording the playing progress of the media resource data. And releasing the redundant data in the first buffer queue, and recording the target information according to the redundant data. Wherein the redundant data is the media asset data except the reference data in the buffer queue, and the reference data is the fragmented data corresponding to the playing progress. The method can also respond to a starting instruction of the media resource playing application, and the media resource playing application is started according to the target information and the playing progress, so that the playing picture of the media resource data is continuously displayed based on the target page. According to the method, the memory occupation of the media resource playing application in the background can be reduced, the playing can be quickly recovered at a fixed point when the media resource playing application is restarted, and the use experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a resource control method for a background mode. Background Art

[0002] Display devices are intelligent devices that can present user interfaces and support user interaction. Taking smart TV as an example, smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and integrates multiple functions such as audio and video, entertainment, and data to meet the diverse and personalized needs of users.

[0003] The display device can be configured with a variety of different types of applications to achieve different functions. For example, the display device can have a built-in media player application. The media player application can download media data based on the browser, and decode and play the media data, so that the display device presents the playback screen corresponding to the media data. When the display device plays media through the media player application, the page of the media player application is displayed in the foreground. For ease of control, the display device can also retain the browser process of the media player application according to the user's specific operations and switch the media player application to the background. In this way, when the user re-enters the media player application, the media player application can be quickly switched back to the foreground through the retained browser process to achieve rapid display of the media player application page.

[0004] Then, after the media player application is switched to the background, although the display device no longer displays the page of the media player application, the browser process of the media player application will continue to download and decode the media data, resulting in a large memory usage and affecting the operating efficiency of the display device. Summary of the invention

[0005] The present application provides a display device and a resource control method for a background mode to solve the problem of large memory usage of an application in the background mode.

[0006] In a first aspect, some embodiments of the present application provide a display device, including a display and a controller. The display is configured to display a user interface, and the controller is configured to execute the following program steps:

[0007] In response to a background running instruction of a media asset playback application, hiding a target page of the media asset playback application, and detecting a video element of the target page, the video element being used to embed a playback screen of media asset data;

[0008] Pause playing the media data of the video element;

[0009] Recording the playback progress of the media asset data;

[0010] Releasing redundant data in a first buffer queue, where the first buffer queue is used to cache media data of the media asset playback application, where the redundant data is media asset data in the first buffer queue other than the reference data, where the reference data is the fragment data corresponding to the playback progress;

[0011] Record target information according to the redundant data;

[0012] Receiving a start instruction of the media resource playing application;

[0013] In response to the start instruction, the media asset playback application is started according to the target information and the playback progress, and the display is controlled to display a playback screen of the media asset data based on the video element of the target page.

[0014] In some embodiments of the present application, the controller executes pausing the playback of the media data of the video element, and is specifically configured to: traverse the video objects of the video element; call a preset pause function to stop the video object from playing the media data; and release the playback resources used to play the media data.

[0015] In some embodiments of the present application, before the controller executes the step of recording the playback progress of the media data, it is also configured to: query the current time attribute of the video object; parse the current playback position of the media data through the current time attribute to generate the playback progress of the media data.

[0016] In some embodiments of the present application, the controller executes the release of redundant data in the first buffer queue, and is specifically configured to: query the target key frame according to the playback progress, the target key frame being the key frame of the media data in the first buffer queue that is closest to the key point; query the target fragment data containing the target key frame; mark the media data in the first buffer queue whose playback position is before the target fragment data as the baseline data; and delete the media data in the first buffer queue except the baseline data.

[0017] In some embodiments of the present application, the controller executes recording of target information based on the redundant data, and is specifically configured to: detect the cache time and the shard address of the redundant data; add the cache time and the shard address to the first buffer queue to record the target information.

[0018] In some embodiments of the present application, the controller executes to start the media asset playback application according to the target information and the playback progress, and is specifically configured to: cancel the hidden state of the target page; locate the benchmark data in the first buffer queue according to the playback progress; decode the benchmark data in the first buffer queue starting from the playback progress; render the decoded benchmark data to the video element to continue to display the playback screen of the media asset data through the video element.

[0019] In some embodiments of the present application, after the controller executes the step of receiving the startup instruction of the media playback application, it is also configured to: read the target information recorded in the first buffer queue in response to the startup instruction; download the redundant data according to the target information; cache the downloaded redundant data to the first buffer queue; and play the media data based on the first buffer queue.

[0020] In some embodiments of the present application, after the controller executes the step of receiving the startup instruction of the media playback application, it is also configured to: in response to the startup instruction, download supplementary media data based on the media playback application, the supplementary media data is media data whose playback position is located after the redundant data; cache the supplementary media data to a second buffer queue; and play the media data according to the second buffer queue and the first buffer queue.

[0021] In some embodiments of the present application, after the controller executes the step of caching the downloaded redundant data into the first buffer queue, it is also configured to: merge the first buffer queue and the second buffer queue to generate a composite buffer queue; read the target playback position of the media data in the video element; decode the media data in the composite buffer queue according to the target playback position; render the decoded media data to the video element to display the playback screen of the media data through the video element.

[0022] In a second aspect, some embodiments of the present application further provide a resource control method in a background mode, comprising the following steps:

[0023] In response to a background running instruction of a media asset playback application, hiding a target page of the media asset playback application, and detecting a video element of the target page, the video element being used to embed a playback screen of media asset data;

[0024] Pause playing the media data of the video element;

[0025] Recording the playback progress of the media asset data;

[0026] Releasing redundant data in a first buffer queue, where the first buffer queue is used to cache media data of the media asset playback application, where the redundant data is media asset data in the first buffer queue other than the reference data, where the reference data is the fragment data corresponding to the playback progress;

[0027] Record target information according to the redundant data;

[0028] Receiving a start instruction of the media resource playing application;

[0029] In response to the start instruction, the media asset playback application is started according to the target information and the playback progress, and the display is controlled to display the playback screen of the media asset data based on the video element of the target page.

[0030] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and a resource control method for the background mode, and the method can respond to the background running instruction of the media player application, hide the target page of the media player application, and detect the video element of the target page. Pause the media data of the video element, and record the playback progress of the media data. Then release the redundant data in the first buffer queue, and record the target information according to the redundant data. Among them, the redundant data is the media data in the first buffer queue except the benchmark data, and the benchmark data is the fragmented data corresponding to the playback progress. Receive the start instruction of the media player application, and respond to the start instruction, start the media player application according to the target information and the playback progress, and control the display to continue to display the playback screen of the media data based on the video element of the target page. The method can reduce the memory usage of the media player application in the background, and can quickly resume the playback at a fixed point when the media player application is restarted, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0033] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0034] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0035] Figure 4 A schematic diagram of the process of downloading and parsing media asset data provided in some embodiments of the present application;

[0036] Figure 5 A schematic diagram of the process of decoding and playing media data provided in some embodiments of the present application;

[0037] Figure 6 A schematic diagram of the effect of switching a media playback application to the background provided in some embodiments of the present application;

[0038] Figure 7 A timing diagram of a background mode resource control method provided in some embodiments of the present application;

[0039] Figure 8 A flowchart of a background mode resource control method provided in some embodiments of the present application;

[0040] Fig. 9 A schematic diagram of releasing redundant data provided in some embodiments of the present application;

[0041] Fig.10 A schematic diagram of background recording information provided by some embodiments of the present application;

[0042] Fig.11 A schematic diagram of the effect of the background tab provided in some embodiments of the present application;

[0043] Fig.12 A schematic diagram of the effect of switching a media resource playback application from the background to the foreground provided in some embodiments of the present application;

[0044] Fig.13 A schematic diagram of a process for downloading redundant data provided in some embodiments of the present application;

[0045] Fig.14 A schematic diagram of a process for downloading supplementary media asset data provided in some embodiments of the present application;

[0046] Fig.15 A schematic diagram of a process for merging buffer queues provided in some embodiments of the present application;

[0047] Fig.16 A schematic diagram of the effect of a media asset playback application switching from the background to the foreground under abnormal circumstances provided in some embodiments of the present application;

[0048] Fig.17 A schematic diagram of the calculation principle of the live media playback time provided in some embodiments of the present application. DETAILED DESCRIPTION

[0049] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0050] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0051] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0052] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0053] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0054] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0055] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0056] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0057] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0058] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0059] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0060] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0061] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0062] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0063] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0064] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0065] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0066] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0067] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0068] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0069] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0070] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0071] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0072] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0073] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, Figure 3 A software configuration diagram of a display device provided for some embodiments of the present application. In some embodiments, the system of the display device 200 can be divided into three layers, namely, an application layer, a middleware layer, and a hardware layer from top to bottom.

[0074] The application layer mainly includes applications on the TV and the application framework. Among them, applications are mainly applications developed based on the browser, such as HTML5 APPs and native applications.

[0075] The Application Framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces of these functions (toolbars, status bars, menus, dialog boxes).

[0076] Native apps can support online or offline, message push or local resource access.

[0077] The middleware layer includes various TV protocols, multimedia protocols, system components and other middleware. The middleware can use the basic services (functions) provided by the system software to connect various parts of the application system or different applications on the network, and can achieve the purpose of resource sharing and function sharing.

[0078] The hardware layer mainly includes HAL interface, hardware and driver. Among them, HAL interface is the unified interface for all TV chips to connect, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0079] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0080] In order to expand the functions of the display device 200, in some embodiments, such as Figure 3 As shown, the application layer of the display device 200 may be configured with applications for implementing different functions, such as media asset playback applications, game applications, etc. For example, the media asset playback application is an application developed based on a browser, and media asset data may be played using MSE (Media Source Extensions) playback, where the browser downloads the media asset data and actively pushes the media asset data through Js (Javascript) to achieve playback of the media asset data.

[0081] In some embodiments, the media player application uses the video element to display the playback screen of the media data. The video element is a HTML5 <video>The tag can provide native support for video playback. The video element can directly embed media data in HTML documents. The media data can support multiple formats, such as MP4, WebM, Ogg, etc.

[0082] In some embodiments, the display device 200 can communicate with the server 400 based on the browser to achieve the interaction of media asset data. That is, during the communication between the display device 200 and the server 400, the display device 200 can send a request including a network link to the server 400 to obtain the media asset data corresponding to the media asset. After receiving the acquisition request sent by the display device 200, the server 400 extracts the corresponding media asset data according to the acquisition request, and returns the extracted media asset data to the display device 200, thereby downloading the media asset data.

[0083] In order to realize data interaction between the display device 200 and the server 400, in some embodiments, the display device 200 establishes a communication connection with the server 400 through the browser of the media asset playback application to establish a transmission channel for the media asset data. For example, the display device 200 and the server 400 can both be connected to the Internet, and the interactive data is transmitted between the display device 200 and the server 400 according to the Internet transmission protocol.

[0084] It should be noted that the display device 200 and the server 400 may also establish a communication connection relationship by other connection methods, such as wired broadband, wireless local area network, cellular network, Bluetooth, infrared, radio frequency communication, etc.

[0085] In some embodiments, the display device 200 and the server 400 are connected in a "many-to-one" relationship, that is, multiple display devices 200 can establish a communication connection with the same server 400, so that the server 400 can provide services for multiple display devices 200. Alternatively, the display device 200 and the server 400 can also be connected in a "many-to-many" relationship, that is, multiple display devices 200 can establish a communication connection with multiple servers 400, so that the multiple servers 400 can provide different services for the display device 200. Obviously, in some application scenarios, the display device 200 and the server 400 can also be connected in a "one-to-one" relationship, that is, one server 400 specifically provides services for one display device 200.

[0086] In order to improve the smoothness of media playback, in some embodiments, the media playback application of the display device 200 pre-downloads a certain amount of media data to the memory through the browser, parses the downloaded media data, and caches the parsed media data in the buffer queue. Figure 4 As shown, the display device 200 can download the fragmented data of the media asset data through the browser, parse the fragmented data, and then send the parsed fragmented data to the BufferQueue (buffer queue) for caching.

[0087] In some embodiments, after the display device 200 downloads the media data through the browser, it decodes the media data through the browser's decoder, and then renders the decoded data frame into the corresponding video element, so that the media playback application can display the playback screen of the media data through the browser page. Figure 5 As shown, the display device 200 can read the media data cached in the BufferQueue through the browser, decode the read media data through the decoder, and play the decoded media data to display the playback screen of the media data through the browser page.

[0088] In some embodiments, the data frames of the media data include I-frames (intra-frame coded frames) and P-frames (forward-predictive coded frames). Among them, I-frames appear in different forms in different codecs, such as IDR, CRA or BLA. I-frames are composed only of intra-frame predicted macroblocks. Each macroblock in the frame can only match other macroblocks in the same frame and can only be compressed through intra-frame spatial redundancy, that is, through the similarity between pixels of a single frame. Different types of I-frames are essentially the same, and there is no time domain prediction. Therefore, I-frames can be encoded and decoded independently. P-frames represent predicted frames. In addition to spatial domain prediction, P-frames can also be compressed by time domain prediction. P-frames need to refer to the previous frame for motion estimation, and each macroblock in the P frame can be predicted by time domain, spatial domain and skipped. Among them, skipping means that the decoder copies the macroblocks located in the same position in the previous frame, that is, 0 motion direction.

[0089] In order to facilitate user control, in some embodiments, after the display device 200 starts the media player application, the target page of the media player application is displayed in the foreground. At this time, the user can input a background operation instruction through a specific interactive action, so that the display device 200 switches the media player application to the background in response to the background operation instruction and hides the display of the target page. Figure 6 As shown, the user can input background running instructions through the "HOME" button in the remote control of the display device 200, so that the display device 200 switches to the home page and hides the playback page of the media playback application in the background; or, when the display device 200 supports touch operation, the user can input background running instructions through a swiping up touch gesture.

[0090] In some embodiments, when the display device 200 switches the media asset playback application to the background in response to the background running instruction, the browser process of the media asset playback application is retained. In this way, when the user re-enters the media asset playback application, the display device 200 can quickly switch the media asset playback application to the foreground through the retained browser process, thereby restoring the playback effect of the browser application.

[0091] In some embodiments, when the display device 200 switches the media player application to the background, the display device 200 freezes the Js of the media player application so that Js no longer performs any playback operations. However, the browser of the media player application continues to execute Figure 4-Figure 5 That is, the display device 200 only pauses the media playback operation in the Js layer of the media playback application and hides the target page of the media playback application, but the browser is still continuously downloading and playing the media data, resulting in a large memory usage, thereby affecting the operating efficiency of the display device 200.

[0092] Furthermore, since the browser continues to download and play the media data, the playing progress of the media data in the media playing application will be pushed back and the playing channel of the display device 200 will be occupied. In this way, when the display device 200 re-runs the media playing application, the playing progress of the media data and the playing progress when the media playing application is switched to the background will not match, and the media data of other channels cannot be played normally, which not only affects the user's viewing experience, but also easily causes the media playing application to report errors.

[0093] For example, a media player application is playing media data. When the playback progress of the media data reaches playback progress A, the user switches the media player application to the background. As time goes by, the browser continues to download, decode, and play the media data. The user enters the media player application again. At this time, the playback progress of the media data in the browser has reached playback progress B. The media player application determines that the playback status is inconsistent and reports an error.

[0094] To this end, some embodiments of the present application provide a display device 200, which can release most of the memory when the media playback application enters the background, thereby reducing the resource occupation of the media playback application; and the display device 200 can also quickly seek to the playback progress when the media playback application enters the background when re-entering the display device 200, and resume the playback of the media data, thereby achieving a non-perceptible switching effect and improving the user experience.

[0095] like Figure 7-Figure 8 As shown, in some embodiments, the display device 200 may include a display 260 and a controller 250 .

[0096] The display 260 is configured to display a user interface, and the controller 250 is configured to execute the following program steps:

[0097] S801: In response to a background running instruction of a media player application, hiding a target page of the media player application, and detecting a video element of the target page.

[0098] The display device 200 can play the media data through the media playback application, and the playback mode of the media data can include live broadcast or on-demand. In some embodiments, the display device 200 can respond to the media playback instruction input by the user, establish a connection with the server 400 through the browser of the media playback application, so as to download, parse, decode and play the media data in real time, so as to display the playback screen of the media data through the target page of the media playback application, that is, the browser page. For example, the display device 200 can Figure 4-Figure 5 The media data is downloaded and played in the manner shown.

[0099] In some embodiments, the display device 200 can render the decoded media data to a video element, that is, the video element can be used to embed the playback screen of the media data. Each video element can be set with a corresponding video object, that is, an HTMLVideo Element object. When the video element is loaded into the browser, it will be parsed by the browser as an HTMLVideoElement object, and the HTMLVideoElement object can provide methods and properties for operating the playback of media data. For example, pausing the playback of media data, currentTime attribute, etc.

[0100] During the process of the media asset playback application playing the media asset data, the user can input the background operation instruction of the media asset playback application to the display device 200 through a specific interactive operation, switch the media asset playback application to the background, and perform other operations or temporarily pause the playback of the media asset data through the display device 200. Therefore, the display device 200 will hide the target page of the media asset playback application in response to the background operation instruction of the media asset playback application.

[0101] In some embodiments, when the display device 200 switches the media player application to the background, it freezes the JS of the media player application and hides the target page. At this time, the JS of the media player application will no longer perform any operations, and the user cannot perceive the display effect of the target page. However, since JS does not affect the browser and the hardware platform of the display device 200, the browser and the hardware platform will still execute Figure 4-Figure 5 Therefore, the display device 200 also detects the video element in the target page to process the video element in the target page.

[0102] S802: Pause playing the media asset data of the video element.

[0103] After detecting the video element of the target page, the display device 200 pauses playing the media data in the video element. At this time, the browser of the media playback application will no longer continue to download the media data from the server 400, which can reduce a certain amount of memory usage and release system resources.

[0104] In some embodiments, after the display device 200 detects the video element of the target page, it can traverse the video objects of the video element. Then, it calls a preset pause function to stop the video object from playing the media data, and releases the playback resources of the media data to reduce the memory usage of the media playback application in the background. For example, the display device 200 can call the pause() method to perform a pause operation on the video object.

[0105] In some embodiments, the playback resources released by the display device 200 may include at least one of decoder resources, memory resources, network resources, and GPU resources. Among them, the memory resources may include memory space for storing video frames, decoded data, and playback status information, and the network resources may include a connection established with a video streaming server and related network buffers, etc.

[0106] S803: Record the playback progress of the media asset data.

[0107] The display device 200 pauses the playing of the media asset data by the video element and also records the playing progress of the media asset data in the video element, so as to record the playing progress of the media asset data in the current media asset playing application.

[0108] In some embodiments, the display device 200 can query the currentTime attribute of the video object and parse the current playback position of the media asset data through the currentTime attribute to generate the playback progress of the media asset data. In this embodiment, the currentTime attribute is recorded as the playback progress of the media asset data, which can facilitate the display device 200 to quickly locate the playback progress of the media asset playback data.

[0109] In some embodiments, the display device 200 can also record the playback progress according to the frame number or timestamp of the media data, that is, according to the playback position of the current media data, determine the frame number or timestamp of the playback position data frame, and record the determined frame number or timestamp to record the playback progress of the media data.

[0110] S804: Release the redundant data in the first buffer queue.

[0111] Since the display device 200 will preload some media data, and this part of the media data is useless for the media playback application in the background and will also occupy a large amount of memory. Therefore, after the display device 200 pauses the playback of the video element and records the playback progress of the media data, the display device 200 can also release the redundant data in the buffer queue. For the sake of distinction, the embodiment of the present application represents the buffer queue that caches the media data when the media playback application is running in the foreground as the first buffer queue.

[0112] Among them, the redundant data is the media data in the first buffer queue except the benchmark data, and the benchmark data is the slice data corresponding to the playback progress. That is, in some embodiments, the display device 200 can query the target key frame according to the playback progress. Among them, the target key frame is the key frame of the media data closest to the key point in the first buffer queue. Then query the target slice data containing the target key frame, and mark the media data in the first buffer queue whose playback position is before the target slice data as benchmark data. Then delete the media data in the first buffer queue except the benchmark data to release the redundant data and reduce the memory occupied by the media playback application in the background.

[0113] For example, Fig. 9 As shown, the display device 200 can determine the fragmented data containing the key frame according to the playback progress of the media data, retain the fragmented data, and delete the remaining fragmented data. In this way, only the media data corresponding to the playback progress, that is, the reference data, is retained in the first buffer queue, which can effectively reduce the memory usage of the cached data. In addition, the retained reference data can also support fast playback of re-entering the media playback application, so as to realize the seamless switching of the media playback application from the background to the foreground.

[0114] S805: Record target information according to the redundant data.

[0115] Since part of the media data pre-downloaded by the display device 200 is released when the media playback application switches to the background, if the user starts the media playback application again, the media playback application unfreezes Js and pauses the video element, and the browser will continue to download the media data. The media data that the browser continues to download is based on the media data pre-loaded last time, and the integrity of the media data will be affected, resulting in playback errors.

[0116] Therefore, in order for the media playback application to continue playing the media data when re-entering the media playback application, the display device 200 releases the redundant information while also recording the cache time and segment address of the redundant data, and generates target information to re-download and cache the released media data according to the target information.

[0117] In some embodiments, when the display device 200 records the target information according to the redundant data, it can detect the cache time and the fragment address of the redundant data, and then add the cache time and the fragment address to the first buffer queue to record the target information. That is, the bufferRange and the fragment address of each fragment data in the redundant data are recorded, so that the display device 200 can re-cache the redundant data later. For example, Fig.10 As shown, the playback progress, target information, and retained benchmark data recorded by the display device 200 only occupy a small amount of memory and will not affect the playback of other channels in the display device 200, such as HDMI, etc., thereby reducing the occupancy of system resources and ensuring the operating efficiency of the display device 200.

[0118] S806: Receive a start instruction of the media asset playback application.

[0119] After the display device 200 switches the media player application to the background, the user can also switch the media player application back to the foreground based on a specific interactive operation. For example, the user can select the icon of the media player application and input a start instruction of the media player application to the display device 200; or Fig.11 As shown, the user can also control the display device to display the tab of the background application by long pressing the "HOME" button of the remote control of the display device 200, and then input the start instruction of the media playback application to the display device 200 by selecting the tab 1101 of the media playback application.

[0120] S807: In response to the start instruction, start the media asset playback application according to the target information and the playback progress, and control the display to display the playback screen of the media asset data based on the video element of the target page.

[0121] After the display device 200 receives the start instruction of the media asset playback application again, it can respond to the start instruction, start the media asset playback application according to the target information and the playback progress, and control the display 260 to continue to display the playback screen of the media asset data based on the video element of the target page, that is, cancel the pause of the video element. In this way, the progress of the media asset data can be quickly located through the playback progress, and then the media asset data can be continuously downloaded and cached according to the target information, which can improve the playback continuity and smoothness of the media asset data, and achieve the following: Fig.12 The media resource playback application shown switches seamlessly from the background to the foreground, improving the user experience.

[0122] In some embodiments, when the display device 200 starts the media asset playback application according to the target information and the playback progress, the hidden state of the target page can be cancelled, so that the display 260 of the display device 200 can redisplay the display effect of the target page, and the target page of the media asset playback application is switched to the foreground. Then, the reference data is located in the first buffer queue according to the playback progress, and the reference data in the first buffer queue is decoded starting from the playback progress. Then, the decoded reference data is rendered to the video element, so as to continue to display the playback screen of the media asset data through the video element.

[0123] That is, the display device 200 can seek to the playback time of the media asset data when the media asset playback application switches to the background according to the recorded playback progress, and then continue to decode and play the media asset data at that time. Since the display device 200 has retained the segment data corresponding to the playback progress when releasing the data, the media asset playback application can quickly present the screen of the media asset data at this time, realizing the seamless switching of the media asset playback application.

[0124] In some embodiments, after receiving the start instruction of the media asset playback application, the display device 200 also responds to the start instruction, reads the target information recorded in the first buffer queue, downloads the redundant data according to the target information, and caches the downloaded redundant data to the first buffer queue, and then plays the media asset data based on the first buffer queue.

[0125] That is, after the display device 200 switches the media asset playback application to the background, if the media asset playback application is started again, the display device 200 will unfreeze the application Js of the media asset playback application and continue to download subsequent media asset data, that is, the media asset data whose playback position is after the redundant data. In order to improve the continuity of media asset playback, the display device 200 needs to re-download the released redundant data to the first buffer queue according to the recorded target information for decoding and playback.

[0126] In some embodiments, when the display device 200 downloads redundant data according to the target information, the redundant data is downloaded according to the recorded cache time and the shard address. Fig.13 As shown, the display device 200 can download each segment data in sequence according to the recorded target information 1301, and cache the downloaded data in the first buffer queue, so as to decode and play the media asset data through the first buffer queue.

[0127] In some embodiments, the display device 200 further downloads supplementary media data based on the media playback application in response to the start instruction. The supplementary media data is media data whose playback position is located after the redundant data. The supplementary media data is then cached in the second buffer queue, and the media data is played according to the second buffer queue and the first buffer queue.

[0128] For example, Fig.14 As shown, after the display device 200 unfreezes the application Js, the media asset playback application will continue to download subsequent supplementary media asset data and cache the supplementary media asset data in the second buffer queue, so that the display device 200 can decode and play the media asset data according to the first buffer queue and the second buffer queue.

[0129] In order to improve the continuity and accuracy of the playback process, in some embodiments, after the display device 200 caches the downloaded redundant data into the first buffer queue, it also merges the first buffer queue and the second buffer queue to generate a composite buffer queue. Then, the target playback position of the media data in the video element is read, and the media data in the composite buffer queue is decoded according to the target playback position. Then, the decoded media data is rendered to the video element of the target page, so as to display the playback screen of the media data through the video element.

[0130] For example, after the display device 200 downloads and caches the redundant data to the first buffer queue according to the target information, the second buffer queue for downloading the supplementary media asset data can be merged into the end of the first buffer queue to form a buffer queue such as Fig.15 In this way, the display device 200 can decode and play the media data according to the composition buffer queue, so as to continuously render the playing screen of the media data in the video element of the target page.

[0131] Among them, the first buffer queue is the fragment data downloaded by the browser of the media asset playback application after switching to the background compared with the application Js, and the second buffer queue is the fragment data newly downloaded after the application Js is unfrozen when the media asset playback application is restarted. The two can be merged into an overall buffer queue.

[0132] For some abnormal events, such as invalid segment address, abnormal seek of media asset playback application, etc., in some embodiments, the display device 200 can exit the playback of video elements and close the target page based on the media asset playback application when the abnormal event is detected. At this time, the user can see the application page located at the first level of the playback page in the media asset playback application in the display device 200. Since the browser process of the media asset playback application is not closed when the media asset playback application enters the background, there is no need to reload the page resources of the previous level page. Therefore, the display device 200 can respond to the startup instruction and quickly display the display screen of the previous level page to achieve a quick response to the startup instruction from the background to the foreground.

[0133] For example, Fig.16 As shown, during the process of the display device 200 playing media through the media player application, the user inputs a background operation command to switch the media player application to the background. After a period of time, the user re-enters the media player application. At this time, the display device 200 detects an abnormal event of invalid fragment address, then exits the playback of the video element in the target page, closes the target page, and returns to the upper menu page of the media player application, so as to achieve a quick response effect to the start command.

[0134] For live broadcast type media data, since the live broadcast type media needs to push the playback progress backward according to the real-time time, when the display device 200 enters the foreground from the background, the playback progress of the media data needs to be updated according to the real-time time. Therefore, in some embodiments, when the display device 200 detects that the playback type of the media data is a live broadcast type, it can respond to the background running instruction of the media playback instruction and execute the above steps S801 and S802. Then record the current first playback time and the first network time corresponding to the network UTC (Coordinated Universal Time). Then, release all the media data in the first buffer queue to reduce the memory usage of the media playback application in the background.

[0135] For live media data, in some embodiments, after the display device 200 switches the media playback application to the background, it receives a startup instruction of the media playback application, and then obtains the second network time corresponding to the current network UTC in response to the startup instruction. Then, the difference between the second network time and the first network time is calculated, and the current second playback time is calculated based on the difference and the first playback time, so that the media playback application continues to play the media data according to the second playback time.

[0136] For example, Fig.17 As shown, the display device 200 plays live media through the media player application, and the user inputs the background running instruction of the media player application to the display device 200, switches the media player application to the background, and does not display. At this time, the playback time recorded by the display device 200 is 1719973655, and its corresponding UTC time is 1719944855. After a period of time, the user starts the media player application again, and the display device 200 switches the media player application from the background to the foreground. At this time, the UTC time obtained by the display device 200 is 1719948455, then the current latest playback time calculated by the two UTC times is 1719977255, and the media player application is controlled to continue to play the live media according to the playback time of 1719977255, so as to ensure the correct playback of the live media and avoid playback errors due to inconsistent time.

[0137] Based on the above display device 200, some embodiments of the present application further provide a resource control method in a background mode, which can be applied to the display device 200 provided in the above embodiment, such as Figure 8 As shown, the method comprises the following steps:

[0138] S801: In response to a background running instruction of a media asset playback application, a target page of the media asset playback application is hidden, and a video element of the target page is detected, where the video element is used to embed a playback screen of media asset data.

[0139] S802: Pause playing the media asset data of the video element;

[0140] S803: Recording the playback progress of the media asset data;

[0141] S804: releasing redundant data in a first buffer queue, where the first buffer queue is used to cache media data of the media asset playback application, where the redundant data is media data in the first buffer queue except for reference data, where the reference data is segmented data corresponding to the playback progress;

[0142] S805: Record target information according to the redundant data;

[0143] S806: Receive a start instruction of the media resource playback application;

[0144] S807: In response to the start instruction, start the media asset playback application according to the target information and the playback progress, and control the display to display the playback screen of the media asset data based on the video element of the target page.

[0145] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and a resource control method for the background mode, and the method can respond to the background running instruction of the media player application, hide the target page of the media player application, and detect the video element of the target page. Pause the media data of the video element, and record the playback progress of the media data. Then release the redundant data in the first buffer queue, and record the target information according to the redundant data. Among them, the redundant data is the media data in the first buffer queue except the benchmark data, and the benchmark data is the fragmented data corresponding to the playback progress. Receive the start instruction of the media player application, and respond to the start instruction, start the media player application according to the target information and the playback progress, and control the display 260 to continue to display the playback screen of the media data based on the video element of the target page. The method can reduce the memory usage of the media player application in the background, and can quickly resume the playback at a fixed point when the media player application is restarted, thereby improving the user experience.

[0146] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0147] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0149] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.< / video>

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; The controller is configured as: In response to a background running instruction of a media asset playback application, hiding a target page of the media asset playback application, and detecting a video element of the target page, the video element being used to embed a playback screen of media asset data; Pause playing the media data of the video element; Recording the playback progress of the media asset data; Releasing redundant data in a first buffer queue, where the first buffer queue is used to cache media data of the media asset playback application, where the redundant data is media asset data in the first buffer queue other than the reference data, where the reference data is the fragment data corresponding to the playback progress; Record target information according to the redundant data; Receiving a start instruction of the media resource playing application; In response to the start instruction, the media asset playback application is started according to the target information and the playback progress, and the display is controlled to display a playback screen of the media asset data based on the video element of the target page.

2. The display device according to claim 1, characterized in that The controller executes pausing the playing of the media asset data of the video element, and is specifically configured as follows: Traversing the video objects of the video element; Calling a preset pause function to stop the video object from playing the media asset data; Release the playback resources used to play the media asset data.

3. The display device according to claim 2, characterized in that Before the controller executes the step of recording the playback progress of the media asset data, the controller is further configured to: Querying the current time attribute of the video object; The current playback position of the media asset data is parsed through the current time attribute to generate the playback progress of the media asset data.

4. The display device according to claim 1, characterized in that The controller executes releasing the redundant data in the first buffer queue, and is specifically configured as follows: Querying a target key frame according to the playback progress, the target key frame being a key frame of the media asset data in the first buffer queue that is closest to the key point; Query the target slice data containing the target key frame; Marking the media asset data in the first buffer queue whose playing position is before the target segment data as the reference data; The media asset data except the reference data in the first buffer queue is deleted.

5. The display device according to claim 1, characterized in that The controller executes recording target information according to the redundant data, and is specifically configured to: Detecting the cache time and shard address of the redundant data; The cache time and the slice address are added to the first buffer queue to record the target information.

6. The display device according to claim 1, characterized in that The controller starts the media asset playback application according to the target information and the playback progress, and is specifically configured as follows: Revoking the hidden state of the target page; Locating the reference data in the first buffer queue according to the playback progress; Taking the playback progress as a starting point, decoding the reference data in the first buffer queue; The decoded reference data is rendered to the video element, so as to continue to display the playback screen of the media asset data through the video element.

7. The display device according to claim 1, characterized in that After the controller executes the step of receiving the start instruction of the media asset playback application, the controller is further configured to: In response to the start instruction, reading the target information recorded in the first buffer queue; Downloading the redundant data according to the target information; Cache the downloaded redundant data into the first buffer queue; The media asset data is played based on the first buffer queue.

8. The display device according to claim 7, characterized in that After the controller executes the step of receiving the start instruction of the media asset playback application, the controller is further configured to: In response to the start instruction, downloading supplementary media asset data based on the media asset playback application, the supplementary media asset data being media asset data whose playback position is located after the redundant data; Cache the supplementary media asset data in a second buffer queue; The media asset data is played according to the second buffer queue and the first buffer queue.

9. The display device according to claim 8, characterized in that After the controller executes the step of caching the downloaded redundant data into the first buffer queue, the controller is further configured to: Merging the first buffer queue and the second buffer queue to generate a composite buffer queue; Read the target playback position of the media asset data in the video element; Decoding the media asset data in the synthesis buffer queue according to the target playback position; The decoded media data is rendered to the video element, so as to display a playback screen of the media data through the video element.

10. A resource control method in background mode, characterized in that: include: In response to a background running instruction of a media asset playback application, hiding a target page of the media asset playback application, and detecting a video element of the target page, the video element being used to embed a playback screen of media asset data; Pause playing the media data of the video element; Recording the playback progress of the media asset data; Releasing redundant data in a first buffer queue, where the first buffer queue is used to cache media data of the media asset playback application, where the redundant data is media asset data in the first buffer queue other than the reference data, where the reference data is the fragment data corresponding to the playback progress; Record target information according to the redundant data; Receiving a start instruction of the media resource playing application; In response to the start instruction, start the media asset playback application according to the target information and the playback progress, and The display is controlled to display a playback screen of the media asset data based on the video element of the target page.

Images