Display device and display method of media asset picture

Abstract

The embodiment of the invention provides a display device and a media asset picture display method, and the method can respond to a media asset playing instruction, control a display to display a playing window used for displaying a browser page, and calculate an overlapping area of each display layer in the browser page and the media asset picture. And then drawing a display layer, and setting the overlapping region determined in the display layer as a transparent region. And decoding the media resource picture according to the return coordinate, so that the media resource picture is displayed in a display area corresponding to the return coordinate. Wherein the return coordinate is determined according to the display state of the playing window, and the display state comprises a full-screen state and a non-full-screen state. According to the method, the layer hole digging operation and the browser can be synthesized and decoupled, the return coordinates can be converted according to the display state of the window, the playing limitation of the browser is reduced, meanwhile, correct display of media asset pictures is ensured, and therefore the user experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a method for displaying a media resource screen. Background Art

[0002] Display devices are intelligent devices that can present user interfaces and support user interaction. Taking smart TV as an example, smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and integrates multiple functions such as audio and video, entertainment, and data to meet the diverse and personalized needs of users.

[0003] The user interface presented by the display device includes browser pages generated based on the browser, which can include elements such as text, pictures and videos. Taking the scenario of a browser playing a video as an example, when the display device plays a video, the web page elements are layered into a PictureLayer (picture layer), and the video elements are layered into a VideoLayer (video layer). The PictureLayer generates a texture through rasterization, while the VideoLayer generates a video frame texture through decoding by the media module. The two are combined into a composite frame in the renderer of the browser kernel, and then displayed on the screen after synthesis. At the same time, the VideoLayer can feed back the position information of the video screen it carries to the back end of the display device, so that the back end can use its own hardware decoding capabilities to decode and display the video. In this process, the display device can use the layer digging technology to set the corresponding area of ​​the layer below the VideoLayer to transparent, thereby ensuring that the video screen played by the back end can be directly displayed.

[0004] However, the position information that VideoLayer feeds back to the backend is based on the coordinate system of the browser page. If the browser page is not displayed in full screen on the display device, the video image may not be correctly displayed. In addition, the above-mentioned layer digging technology relies on the browser's synthesis mechanism. When the browser's synthesis method does not match the layer digging technology, the digging effect will fail, thereby affecting the video playback effect of the display device and reducing the user experience. Summary of the invention

[0005] The present application provides a display device and a method for displaying a media resource screen to solve the problem of poor video playback effect in the display device.

[0006] In a first aspect, some embodiments of the present application provide a display device, comprising a display and a controller, wherein the display and the controller are coupled. The display is configured to display a user interface, and the controller is configured to execute the following program steps:

[0007] In response to a media playback instruction, controlling the display to display a playback window, the playback window being used to display a browser page, the browser page including a media screen and at least two display layers;

[0008] Calculating a target area of ​​the display layer, where the target area is an overlapping area between the media asset screen and the display layer;

[0009] Drawing the display layer, and setting the target area in the display layer as a transparent area;

[0010] The media resource picture is decoded according to the returned coordinates, so that the media resource picture is displayed in the display area corresponding to the returned coordinates; the returned coordinates are determined according to the display state of the playback window, and the display state includes a full-screen state and a non-full-screen state.

[0011] The above technical solution has the following advantages or beneficial effects: after receiving the media playback instruction, the display device can respond to the media playback instruction, determine the overlapping area between each display layer and the media screen in the browser page according to the position of the media screen, and set the calculated overlapping area to transparent when drawing each display layer, so that the layer digging technology can be decoupled from the browser synthesis, thereby reducing the limitations of the browser media playback; at the same time, the display device also decodes the media screen according to the return coordinates determined by the corresponding display state of the playback window, so that the media screen can be correctly displayed from the display layer, so as to improve the display effect of the media screen during the media playback process.

[0012] In some embodiments of the present application, the display layer includes a first layer and a second layer, the first layer includes a target layer for displaying the media asset screen and a display layer with a display level higher than the target layer, and the second layer includes a display layer with a display level lower than the target level; the controller executes calculation of the target area of ​​the display layer, and is specifically configured to: calculate a first position area of ​​the media asset screen according to the target layer, the first position area being the position area of ​​the media asset screen in the screen coordinate system; calculate a second position area of ​​the media asset screen according to the first position area and the first layer, the second position area being the position area of ​​the media asset screen in the content coordinate system of the first layer; detect an intersection area of ​​the second position area with a boundary box of the first layer to obtain a target area of ​​the first layer.

[0013] The above technical solution has the following advantages or beneficial effects: the display device can convert the coordinates of the media picture according to the position information of the media picture, based on the screen coordinate system and the content coordinate system of each display layer, and judge the position of the media picture in each display layer according to the coordinates obtained by the conversion, so that the display device can perform subsequent digging operations according to the calculated position (that is, perform transparent processing on specific areas in the display layer).

[0014] In some embodiments of the present application, the controller executes calculation of the first position area of ​​the media asset screen based on the target layer, and is specifically configured to: obtain a first transformation matrix of the target layer and a third position area of ​​the media asset screen, the first transformation matrix is ​​used to convert coordinates in the content coordinate system of the target layer into coordinates in the screen coordinate system, and the third position area is the position area of ​​the media asset screen in the content coordinate system of the target layer; extract the coordinate matrix of the third position area; convert the coordinate matrix of the third position area through the first transformation matrix, so as to obtain the first position area through the converted coordinate matrix.

[0015] The above technical solution has the following advantages or beneficial effects: according to the full-screen transformation matrix (i.e., the first transformation matrix) of the target layer used to display the video (i.e., the media screen), the position and size of the video in the content coordinate system corresponding to the target layer are converted to the screen coordinate system, thereby determining the exact position and size of the video on the screen based on the screen coordinate system of the display device.

[0016] In some embodiments of the present application, the controller calculates the second position area of ​​the media asset screen based on the first position area and the first layer, and is specifically configured to: obtain the inverse matrix of the second transformation matrix of the first layer, the second transformation matrix is ​​used to convert the coordinates in the content coordinate system of the first layer into coordinates in the screen coordinate system; extract the coordinate matrix of the first position area; convert the coordinate matrix of the first position area through the inverse matrix of the second transformation matrix, so as to obtain the second position area through the converted coordinate matrix.

[0017] The above technical solution has the following advantages or beneficial effects: through the inverse matrix of the full-screen transformation matrix (i.e., the second transformation matrix) corresponding to the current display layer, the position and size of the video in the screen coordinate system are converted into the position and size in the content coordinate system corresponding to the display layer, so that the display device can accurately determine the position and size of the video in the current display layer.

[0018] In some embodiments of the present application, the controller executes drawing the display layer and setting the target area in the display layer as a transparent area, and is specifically configured as follows: determining a drawing order according to a display level of the display layer, the drawing order of the second layer being earlier than the drawing order of the first layer; drawing the display layer by a renderer of a browser thread according to the drawing order; when the drawn display layer is the second layer, drawing the second layer by the renderer; when the drawn display layer is the first layer, drawing the first layer by the renderer, and setting the target area of ​​the first layer to a transparent area.

[0019] The above technical solution has the following advantages or beneficial effects: the display device can draw each display layer according to the display level of each display layer in the browser page. Among them, the lower the display level of the display layer (that is, the lower the level of the display layer when displayed on the screen), the earlier the corresponding drawing order (that is, the lower the display level, the earlier the display layer is drawn). When the display device draws each display layer, it does not perform a hole digging operation on the display level lower than the video layer (that is, the target layer) to save system resources; the display device only performs a hole digging operation on the video layer and the display layer with a display level higher than the video layer according to the calculated target area. In this way, the display device can complete the hole digging operation in the rendering stage, while decoupling the hole digging operation and the browser synthesis mechanism to ensure the correct display of the video.

[0020] In some embodiments of the present application, the controller executes setting the target area of ​​the first layer as a transparent area, and is specifically configured as follows: generating a unit drawing instruction according to the target area associated with the first layer, the unit drawing instruction carrying the location information of the target area; sending the unit drawing instruction to the first layer; detecting the target area of ​​the first layer through the renderer in response to the unit drawing instruction; setting the transparency value of the target area in the first layer to a target value, the target value being less than the transparency value of the media asset screen.

[0021] The above technical solution has the following advantages or beneficial effects: the display device performs a hole operation on the video layer and the display layer with a display level higher than the video layer, and sets the transparency of the target area corresponding to each display layer to transparent or nearly transparent, so that the video can be displayed from each display layer, thereby improving the video display effect played by the browser.

[0022] In some embodiments of the present application, after the controller executes the step of controlling the display to display the playback window, it is also configured to: monitor the display status of the playback window; if the display status of the playback window is full-screen, mark the coordinate matrix of the media resource screen in the page coordinate system as the return coordinates, and send the return coordinates to the back end; the page coordinate system is the logical coordinate system corresponding to the browser page; if the display status of the playback window is non-full-screen, convert the coordinate matrix of the media resource screen in the page coordinate system into a target matrix, mark the target matrix as the return coordinates, and send the return coordinates to the back end.

[0023] The above technical solution has the following advantages or beneficial effects: the display device can monitor the display status of the playback window in real time, and determine the video coordinates to be sent back according to the display status of the playback window. If the playback window is in full-screen display, the browser page and the screen are completely mapped, and the video coordinates can be directly sent back to the back end for display; if the playback window is not in full-screen display, the browser page and the screen are not completely mapped, and the display device needs to convert the video coordinates according to the page coordinate system, and send the converted coordinates back to the back end for display, so that the video can be correctly displayed at the hole position of the playback window.

[0024] In some embodiments of the present application, if the display state of the playback window is a non-full-screen state, the controller is further configured to: obtain first size information of the media asset screen in the page coordinate system, and obtain second size information of the playback window in the screen coordinate system; parse the mapping relationship between the first size information and the second size information; generate a first conversion formula based on the mapping relationship, the first conversion formula is used to characterize the process of converting a first coordinate matrix into a second coordinate matrix, the first coordinate matrix is ​​the coordinate matrix of the media asset screen in the page coordinate system, and the second coordinate matrix is ​​the coordinate matrix of the media asset screen in the screen coordinate system; generate a second conversion formula through the second coordinate matrix and the mapping relationship, the second conversion formula is used to characterize the process of converting the second coordinate matrix into a third coordinate matrix, the third coordinate matrix is ​​the coordinate matrix in the page coordinate system corresponding to the second coordinate matrix in the screen coordinate system.

[0025] The above technical solution has the following advantages or beneficial effects: for a playback window that is not displayed in full screen, when the display device converts the video coordinates, it can determine the conversion relationship between the video coordinates and the returned coordinates based on the coordinates and size in the page coordinate system, the coordinates and size in the screen coordinate system, and the coordinate size of the window in the screen coordinate system.

[0026] In some embodiments of the present application, the controller executes the conversion of the coordinate matrix of the media asset screen in the page coordinate system into a target matrix, and is specifically configured as follows: generating a target conversion formula based on the first conversion formula and the second conversion formula, the target conversion formula being used to characterize the process of converting the first coordinate matrix into the third coordinate matrix; calculating the third coordinate matrix according to the first coordinate matrix and the target conversion formula; and marking the third coordinate matrix as the target matrix.

[0027] The above technical solution has the following advantages or beneficial effects: when the playback window is not displayed in full screen, the video coordinates are converted into corresponding return coordinates through the conversion relationship between the video coordinates and the return coordinates, so that the back end of the display device can decode and display the video according to the return coordinates, so that the video can be correctly displayed from the transparent area of ​​the browser page.

[0028] In a second aspect, some embodiments of the present application further disclose a method for displaying a media resource screen, the method comprising the following steps:

[0029] In response to a media playback instruction, controlling the display to display a playback window, wherein the playback window is used to display a browser page, wherein the browser page includes a media screen and at least two display layers;

[0030] Calculating a target area of ​​the display layer, where the target area is an overlapping area between the media asset screen and the display layer;

[0031] Drawing the display layer, and setting the target area in the display layer as a transparent area;

[0032] The media resource picture is decoded according to the returned coordinates, so that the media resource picture is displayed in the display area corresponding to the returned coordinates; the returned coordinates are determined according to the display state of the playback window, and the display state includes a full-screen state and a non-full-screen state.

[0033] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and a method for displaying a media screen. The method can respond to a media playback instruction, control the display to display a playback window for displaying a browser page, and calculate the overlapping area between each display layer and the media screen in the browser page. Then draw the display layer, and set the overlapping area determined in the display layer as a transparent area. Then decode the media screen according to the returned coordinates so that the media screen is displayed in the display area corresponding to the returned coordinates. Among them, the returned coordinates are determined according to the display state of the playback window, and the display state includes a full-screen state and a non-full-screen state. The method can decouple the layer digging operation from the browser synthesis, and can convert the returned coordinates according to the display state of the window, while reducing the limitations of the browser playback, ensuring the correct display of the media screen, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0036] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0037] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0038] Figure 4 A schematic diagram of a media asset screen display process based on a hole-digging operation provided in some embodiments of the present application;

[0039] Figure 5 A flow chart of the display layer hole digging operation provided in some embodiments of the present application;

[0040] Figure 6 A flowchart of a method for displaying a media resource screen provided in some embodiments of the present application;

[0041] Figure 7 A timing diagram of a media resource screen display method provided in some embodiments of the present application;

[0042] Figure 8 A schematic diagram of the display hierarchy relationship of display layers in a browser page provided in some embodiments of the present application;

[0043] Fig. 9 A schematic diagram of a process for calculating a display layer corresponding to a target area provided in some embodiments of the present application;

[0044] Fig.10 A schematic diagram of a flow chart of a target area calculation principle provided in some embodiments of the present application;

[0045] Fig.11 A schematic diagram of the process of drawing and displaying an image and digging a hole provided in some embodiments of the present application;

[0046] Fig.12 A schematic diagram of the media resource screen effect in a non-full screen state without converting the returned coordinates provided in some embodiments of the present application;

[0047] Fig.13 A schematic diagram of a process for calculating and returning coordinates provided in some embodiments of the present application;

[0048] Fig.14 A schematic diagram of the media resource screen effect of converting and returning coordinates in a non-full screen state provided in some embodiments of the present application. DETAILED DESCRIPTION

[0049] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0050] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0051] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0052] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0053] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0054] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0055] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0056] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0057] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0058] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0059] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0060] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0061] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0062] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0063] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0064] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0065] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0066] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0067] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0068] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0069] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0070] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0071] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0072] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0073] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, Figure 3 A software configuration diagram of a display device provided for some embodiments of the present application. In some embodiments, the system of the display device 200 can be divided into three layers, namely, an application layer, a middleware layer, and a hardware layer from top to bottom.

[0074] The application layer mainly includes applications on the TV and the application framework. Among them, applications are mainly applications developed based on the browser, such as HTML5 APPs and native applications.

[0075] The Application Framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces of these functions (toolbars, status bars, menus, dialog boxes).

[0076] Native apps can support online or offline, message push or local resource access.

[0077] The middleware layer includes various TV protocols, multimedia protocols, system components and other middleware. The middleware can use the basic services (functions) provided by the system software to connect various parts of the application system or different applications on the network, and can achieve the purpose of resource sharing and function sharing.

[0078] The hardware layer mainly includes HAL interface, hardware and driver. Among them, HAL interface is the unified interface for all TV chips to connect, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0079] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0080] The display device 200 can implement different functions based on the applications or services configured in the above application layer. In some embodiments, the display device 200 can display different browser pages based on the browser application. The content presented by the browser page may include elements such as text, video, and pictures.

[0081] In some embodiments, when the display device 200 starts the browser application, the browser kernel first parses the browser page (web page) content and converts the content of the browser page into a parsing format supported by the browser. After the browser kernel parses the browser page, the browser will layer the web page elements in the CC (Compositing and Constant) renderer. For example, web page elements such as text and pictures will be layered into PictureLayer, and video elements will be layered into VideoLayer. After the PictureLayer is rasterized, a Tile Texture will be generated.

[0082] Regarding the media playback process of the browser application, in some embodiments, when the browser application plays the media (i.e., video), the browser layers the video element into a Videolayer and layers other elements into a PictureLayer. The Picturelayer generates a tile texture after the rasterization phase, and the Videolayer generates a video frame texture after the video decoding of the media module. Then, in the drawing phase of the CC Renderer, the above two textures are combined into a compositor frame and sent to the browser for synthesis. After synthesis, the browser page and the video can be displayed on the screen together.

[0083] like Figure 4 As shown, in some embodiments, the decoding display of the display device 200 is based on the chip decoding capability of the controller 250, so the media playback of the browser application can be operated through the browser layer digging operation. That is, Videolayer decodes through the media module, but directly hands it over to the backend (backend) of the display device 200 to decode and display the video frame. Videolayer returns the calculated display area of ​​the media asset screen to the backend. Then, the display device 200 uses the layer digging operation to make the corresponding area of ​​the layer below Videolayer (the display layer with a display level higher than Videolayer) transparent, thereby transparently displaying the video played by the backend.

[0084] like Figure 5As shown, in some embodiments, when the display device 200 performs a hole digging operation on the display layer, a transparent DrawQuad (drawing quadrilateral) is constructed to implement the hole digging of the RenderPassQuad (rendering pass quadrilateral) on the same layer, and the hole digging operation of the display layer is implemented by passing the bottom layer hole digging flag to the synthesis stage. Among them, the synthesis stage can use different synthesis methods, such as the glRenderer method, the skiaRenderer method, etc. The operation methods corresponding to different synthesis methods are also different. For example, the gl instruction is used in glRenderer, and the canvas method is used in skiaRenderer. In this way, the hole digging operation of the display layer needs to be set according to the synthesis method. If the synthesis method of the browser does not match, the hole digging operation of the display layer may fail, resulting in the media asset screen not being able to be correctly displayed.

[0085] and, Figure 4 The decoding display performed by the back end of the display device 200 shown is based on the media asset screen position determined by the coordinates returned by Videolayer. When the playback window of the browser application is displayed in full screen, the coordinate system of the browser page and the screen coordinate system are completely mapped, and the display device 200 can correctly display the media asset screen in the appropriate position according to the coordinates returned by Videolayer, so that the media asset screen can be revealed from the dug area; when the playback window of the browser application is not displayed in full screen, although the browser page can be correctly displayed in the playback window through the browser's zoom function, the coordinates returned to the back end of the display device 200 cannot be adapted to the playback window, resulting in problems such as offset and obstruction of the display position of the media asset screen, affecting the user's viewing experience.

[0086] To this end, some embodiments of the present application provide a display device 200, which can be applicable to playback windows in different states, so that the returned coordinates of the media screen can be more accurately adapted to windows in various display states to ensure the correct display of the media screen; and the display device 200 provided in the embodiment of the present application, the digging operation performed on the display layer does not rely on the synthesis of the browser. By decoupling the digging operation of the media playback process from the browser synthesis, the limitations of the digging operation can be improved.

[0087] like Figure 6-Figure 7 As shown, in some embodiments, the display device 200 may include a display 260 and a controller 250. The display 260 is coupled to the controller 250, and the display 260 is configured to display a user interface, and the controller 250 is configured to execute the following program steps:

[0088] S601: In response to a media resource playing instruction, controlling a display to display a playing window, wherein the playing window is used to display a browser page, wherein the browser page includes a media resource screen and at least two display layers.

[0089] The controller 250 of the display device 200 can receive control instructions input by the user based on different interaction methods, such as touch operation, instructions input through the control device 100 of the display device 200, interactive instructions input by voice, etc. In some embodiments, the instructions for playing the media assets are represented as media asset playing instructions, and the media asset playing instructions are used to instruct the display device 200 to decode and play specific media assets. For example, the media asset playing instruction can be input based on a specific playback control displayed by the browser application. When the user performs a selection operation on the playback control, the display device 200 automatically generates a media asset playing instruction for the media asset corresponding to the playback control.

[0090] After receiving the media asset playback instruction, the controller 250 of the display device 200 can respond to the media asset playback instruction and control the display 260 to display the playback window corresponding to the media asset. The playback window is used to display a browser page, and the browser page includes a media asset screen and at least two display layers. In some embodiments, the display layer of the browser page includes a target layer for displaying the media asset screen (i.e., a Videolayer for displaying a video element), and other display layers (i.e., a PictureLayer) that are not used to display the media asset screen.

[0091] In some embodiments, the display layer includes a first layer and a second layer, the first layer includes a target layer (i.e., Videolayer) for displaying a media asset screen and a display layer with a display level higher than the target layer, and the second layer includes a display layer with a display level lower than the target layer. The display level refers to the hierarchical relationship between the display layers when they are displayed in the foreground (i.e., the screen), and when there is an overlapping area, the display layer with a higher display level will cover the display layer with a lower display level.

[0092] For example, Figure 8 As shown, the target layer 801 is a display layer for displaying video elements, the target layer 801 itself and the display layers with a display level higher than the target layer 801 are the first layer 802, and the display layers with a display level lower than the target layer 801 are the second layer 803.

[0093] S602: Calculate a target area of ​​the display layer, where the target area is an overlapping area between the media asset screen and the display layer.

[0094] The controller 250 of the display device 200 displays the playback window in response to the media playback instruction, and calculates the overlapping area between each display layer and the media asset screen in the browser page according to the browser page to be displayed in the playback window. For ease of description, the embodiment of the present application represents the overlapping area between each display layer and the media asset screen as the target area.

[0095] like Fig. 9 As shown, due to the different sizes and display positions of different layers, in some embodiments, when the controller 250 of the display device 200 calculates the target area of ​​each display layer, it can calculate the first position area of ​​the media asset screen according to the target layer (S901), and then calculate the second position area of ​​the media asset screen according to the first position area and the first layer (S902). Among them, the first position area is the position area of ​​the media asset screen in the screen coordinate system, and the second position area is the position area of ​​the media asset screen in the content coordinate system of the first layer. Then, the intersection area of ​​the second position area and the bounding box corresponding to the first layer is detected to obtain the target area of ​​the first layer (S903).

[0096] In some embodiments, since the display level of the second layer is lower than the display level of the target layer (video layer), the second layer will not block the target layer. Therefore, when calculating the target area, the controller 250 of the display device 200 can only calculate the target area of ​​the first layer, without calculating the target area of ​​the second layer, thereby saving the consumption of system resources in the display device 200.

[0097] For the calculation of the first position area, in some embodiments, the controller 250 of the display device 200 may obtain the first transformation matrix of the target layer and the third position area of ​​the media asset screen. The first transformation matrix is ​​used to convert the coordinates in the content coordinate system of the target layer into the coordinates in the screen coordinate system, and the third position area is the position area of ​​the media asset screen in the content coordinate system of the target layer. Then, the display device 200 extracts the coordinate matrix of the third position area to obtain the first position area through the converted coordinate matrix.

[0098] For calculation of the second position area, in some embodiments, the controller 250 of the display device 200 may obtain the inverse matrix of the second transformation matrix of the first layer. The second transformation matrix is ​​used to convert the coordinates in the content coordinate system of the first layer into the coordinates in the screen coordinate system. Then, the display device 200 extracts the coordinate matrix of the first position area, and transforms the coordinate matrix of the first position area by the inverse matrix of the second transformation matrix, so as to obtain the second position area by the transformed coordinate matrix.

[0099] That is, the controller 250 of the display device 200 can determine the position and size of the media asset picture in the display device screen according to the coordinates of the media asset picture in the screen coordinate system, and then convert the position and size of the display device in the screen into the position and size in the coordinate system of the display layer. Then, the media asset picture is intersected with the bound (bounding box) of the display layer through the position and size in the coordinate system of the display layer to determine the target area of ​​the display layer that needs to be dug, so that the media asset picture can be correctly displayed from the target area of ​​the display layer.

[0100] For example, the display process of the display device 200 involves the content coordinate system, the drawing coordinate system and the screen coordinate system. Among them, the content coordinate system is mainly used for drawing the layer itself, the drawing coordinate system is used for the drawing relationship between layers, and the screen coordinate system is used for drawing and positioning the layer content on the screen. The screen coordinate system is the same for all layers, so the display device 200 needs to first obtain the coordinates and size of the video in the screen coordinate system, and then transfer it to the content coordinate system of the corresponding layer.

[0101] That is, Fig.10 As shown, the controller 250 of the display device 200 first obtains the position area of ​​the video in the screen coordinate system according to the full-screen transformation matrix (first transformation matrix) of the video layer; then obtains the position area corresponding to the video in the content coordinate system of the current first layer according to the inverse matrix of the full-screen transformation matrix of the first layer (a display layer with a display level lower than the video layer); then intersects the obtained position area with the bound of the first layer to obtain the digging area (i.e., the target area) in the first layer.

[0102] S603: Draw the display layer, and set the target area in the display layer as a transparent area.

[0103] After the controller 250 of the display device 200 calculates the target area of ​​each display layer, it can draw the display layer according to the target area, and set the target area corresponding to the display layer as a transparent area, so that the media image can be shown through the transparent area to complete the display and playback of the media image.

[0104] In some embodiments, when the controller 250 of the display device 200 draws the display layer, the drawing order is determined according to the display level of the display layer. Among them, the drawing order of the second layer is earlier than the drawing order of the first layer, that is, the higher the display level, the later the corresponding drawing order of the display layer, and the lower the display level, the earlier the corresponding drawing order of the display layer. For example, the display device 200 can recursively traverse the entire layer tree starting from the root layer through ActiveLayerTreeIml (layer tree). Therefore, the digging operation (i.e., DrawQuad operation) can be uniformly performed on the video layer and the first layer behind it according to the drawing order.

[0105] After the controller 250 of the display device 200 determines the drawing order of the display layers, the display layers are drawn by the renderer of the browser thread (such as the cc renderer) according to the drawing order. When the drawn display layer is the second layer, the second layer is drawn by the renderer; when the drawn display layer is the first layer, the first layer is drawn by the renderer, and the target area of ​​the first layer is set as a transparent area. For example, Fig.11 As shown, in the cc rendering stage, the display device 200 determines the drawing order according to the nodes of the layer tree, and draws the browser page according to the drawing order. During the drawing process, the DrawQuad operation is performed on the video layer and the display layer whose drawing order is after the video layer to set the corresponding target area as a transparent area.

[0106] It should be noted that Fig.11 The bottom layer mentioned in the above refers to the bottom layer of the layer tree, not the bottom layer of the display hierarchy. The display layers closer to the bottom layer in the layer tree correspond to a higher display hierarchy.

[0107] Therefore, in some embodiments, when the controller 250 of the display device 200 sets the target area as a transparent area, a unit drawing instruction (i.e., a DrawQuad instruction) can be generated according to the target area associated with the first layer, and the unit drawing instruction can be sent to the first layer. Then, the renderer responds to the unit drawing instruction, detects the target area of ​​the first layer, and sets the transparency value of the target area in the first layer to the target value. The target value is less than the transparency value of the media screen, and can be transparent or approximately transparent. For example, the Alpha value of the target area is set to 0 to set the target area to a transparent area.

[0108] It is understandable that there may be many ways to set the target area as a transparent area in the embodiment of the present application, and the present application does not limit this.

[0109] S604: Decode the media resource screen according to the returned coordinates, so that the media resource screen is displayed in the display area corresponding to the returned coordinates; the returned coordinates are determined according to the display state of the playback window, and the display state includes a full-screen state and a non-full-screen state.

[0110] The controller 250 of the display device 200 draws each display layer of the browser page, and sets the target area of ​​the display layer to be transparent when drawing. It can also decode the video according to the coordinates returned by the video layer to the back end, so that it is displayed in the appropriate position. The coordinates returned by the video layer are the corresponding coordinates when the playback window is in full-screen display state. If the playback window is not in full-screen display state, the display device 200 directly decodes the video according to the coordinates returned by the video layer, which will cause the media image to be offset, for example Fig.12 As shown in the play window 1201 and the media resource screen 1202 , there is a large offset between the media resource screen 1202 and the play window.

[0111] To this end, in some embodiments, the controller 250 of the display device 200 can monitor the display state of the play window and execute different processing strategies on the returned coordinates according to different display states. The display state of the play window can include a full screen state and a non-full screen state.

[0112] like Fig.13 As shown, in some embodiments, if the controller 250 of the display device 200 detects that the display state of the playback window is full screen, the coordinate matrix of the media asset screen in the page coordinate system is marked as the returned coordinates, and the returned coordinates are sent to the backend for processing (S1301), wherein the page coordinate system is the logical coordinate system (i.e., logical coordinate system) of the browser page. That is, when the playback window is in full screen state, the display device 200 does not need to process the coordinates returned by the video, and can directly decode and display the video according to the coordinates.

[0113] If the controller 250 of the display device 200 detects that the display state of the playback window is not full screen, the coordinate matrix of the media resource screen in the page coordinate system is converted into a target matrix (S1302), and the target matrix is ​​marked as the return coordinate, and the return coordinate is sent to the back end for processing (S1303). That is, when the playback window is not displayed in full screen, the display device 200 needs to convert and adapt the coordinates returned by the video so that the media resource screen can be correctly displayed in the playback window.

[0114] For a non-full-screen playback scenario, in some embodiments, the controller 250 of the display device 200 may obtain first size information of the media asset screen in the page coordinate system and second size information of the playback window in the screen coordinate system, and then parse the mapping relationship between the first size information and the second size information, and generate a first conversion formula according to the mapping relationship.

[0115] The first conversion formula is used to characterize the process of converting the first coordinate matrix into the second coordinate matrix, the first coordinate matrix is ​​the coordinate matrix of the media screen in the page coordinate system, and the second coordinate matrix is ​​the coordinate matrix of the media screen in the screen coordinate system. Then, the second conversion formula is generated by the second coordinate matrix and the mapping relationship. The second conversion formula is used to characterize the process of converting the second coordinate matrix into the third coordinate matrix, and the third coordinate matrix is ​​the coordinate matrix in the page coordinate system corresponding to the second coordinate matrix in the screen coordinate system.

[0116] In some embodiments, the first size information may be obtained based on a browser kernel object property in the display device 200 , and the second size information may be obtained based on mojom (inter-module communication protocol) communication of the display device 200 .

[0117] For example, suppose the coordinate size of the video currently returned is (x o ,y o ,w o ,h o ), the converted coordinate size is (x t ,y t ,w t ,h t ). Among them, x, y represent the coordinate position of the video in the logical coordinate system, and w, h represent the width and height of the video in the page logical size coordinate system. The logical size (W) of the browser page can be obtained through the browser kernel object properties (such as window..innerWidth / window.innerHeight). l ,H l The screen size (W s ,H s ). Through mojom communication, you can get the coordinate size (X w ,Y w ,W w ,H w Since the logical size of the browser page and the size of the playback window are completely mapped, the display device 200 can first obtain the coordinate size (x s ,y s ,ws ,h s ), the first conversion formula is as follows:

[0118]

[0119] Then, the display device 200 converts it into a coordinate size (x t ,y t ,w t ,h t ), the second transformation is as follows:

[0120]

[0121] According to the first and second conversion formulas above, we can get:

[0122]

[0123] (x t ,y t ,w t ,h t ) mark as the returned coordinates, the video can be displayed correctly at the appropriate position.

[0124] Therefore, in some embodiments, the controller 250 of the display device 200 may generate a target conversion formula according to the first conversion formula and the second conversion formula, wherein the target conversion formula is used to represent the process of converting the first coordinate matrix into the third coordinate matrix.

[0125] For example, based on the above example, the target conversion formula may be the following conversion formula:

[0126]

[0127] This transformation is used to characterize the o ,y o ,w o ,h o ) to (x t ,y t ,w t ,h t )’s relationship.

[0128] After the display device 200 generates the target conversion formula, the third coordinate matrix can be calculated according to the first coordinate matrix and the target conversion formula, and then the third coordinate matrix can be marked as the target matrix as the final returned coordinates. In this way, the display device 200 converts the coordinates returned by the video layer according to the display state of the playback window, and then decodes and displays the video according to the converted coordinates, so that the video can be displayed at the appropriate position in the video layer, and the video picture can be correctly displayed from the transparent area of ​​each display layer to improve the display effect of the video.

[0129] For example, Fig.14 The media resource screen shown, the display device 200 decodes and displays the media resource according to the converted returned coordinates, and the media resource screen 1402 can also be displayed at a suitable position when the playback window 1401 is not in full screen state. Fig.12 The display effect of the media resource screen shown in the embodiment of the present application is better (such as Fig.14 shown).

[0130] In some embodiments, after the controller 250 of the display device 200 controls the display 260 to display the playback window, the calculation program of the returned coordinates and the calculation program of the target area can be processed in parallel, or the calculation program of the returned coordinates and the calculation program of the target area can be processed in series. When processing in series, the returned coordinates can be calculated first and then the target area, or the target area can be calculated first and then the returned coordinates. This application does not limit this.

[0131] Based on the above display device 200, some embodiments of the present application also provide a method for displaying a media asset screen, and the display method can be applied to the display device 200 provided in the above embodiment. Figure 6 As shown, the display method includes the following program steps:

[0132] S601: In response to a media playback instruction, controlling a display to display a playback window, wherein the playback window is used to display a browser page, wherein the browser page includes a media screen and at least two display layers;

[0133] S602: Calculate a target area of ​​the display layer, where the target area is an overlapping area between the media asset screen and the display layer;

[0134] S603: drawing the display layer, and setting the target area in the display layer as a transparent area;

[0135] S604: Decode the media resource screen according to the returned coordinates, so that the media resource screen is displayed in the display area corresponding to the returned coordinates; the returned coordinates are determined according to the display state of the playback window, and the display state includes a full-screen state and a non-full-screen state.

[0136] It can be seen from the above technical solutions that the display device and the display method of the media asset screen provided by some embodiments of the present application can respond to the media asset playback instruction, control the display 260 to display the playback window for displaying the browser page, and calculate the overlapping area between each display layer and the media asset screen in the browser page. Then draw the display layer, and set the overlapping area determined in the display layer as a transparent area. Then decode the media asset screen according to the returned coordinates so that the media asset screen is displayed in the display area corresponding to the returned coordinates. Among them, the returned coordinates are determined according to the display state of the playback window, and the display state includes a full-screen state and a non-full-screen state. The method can decouple the layer digging operation from the browser synthesis, and can convert the returned coordinates according to the display state of the window, while reducing the limitations of the browser playback, ensuring the correct display of the media asset screen, thereby improving the user experience.

[0137] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0138] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0140] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; A controller is coupled to the display and is configured to: In response to a media playback instruction, controlling the display to display a playback window, the playback window being used to display a browser page, the browser page including a media screen and at least two display layers; Calculating a target area of ​​the display layer, where the target area is an overlapping area between the media asset screen and the display layer; Drawing the display layer, and setting the target area in the display layer as a transparent area; The media resource picture is decoded according to the returned coordinates, so that the media resource picture is displayed in the display area corresponding to the returned coordinates; the returned coordinates are determined according to the display state of the playback window, and the display state includes a full-screen state and a non-full-screen state.

2. The display device according to claim 1, characterized in that The display layer includes a first layer and a second layer, the first layer includes a target layer for displaying the media asset screen and a display layer with a display level higher than the target layer, and the second layer includes a display layer with a display level lower than the target layer; the controller performs calculation of the target area of ​​the display layer, and is specifically configured as follows: Calculate a first position area of ​​the media resource picture according to the target layer, where the first position area is a position area of ​​the media resource picture in a screen coordinate system; Calculate a second position area of ​​the media asset screen according to the first position area and the first layer, where the second position area is a position area of ​​the media asset screen in the content coordinate system of the first layer; An intersection area between the second position area and a bounding box corresponding to the first layer is detected to obtain a target area of ​​the first layer.

3. The display device according to claim 2, characterized in that The controller calculates the first position area of ​​the media asset screen according to the target layer, and is specifically configured as follows: Obtaining a first transformation matrix of the target layer and a third position area of ​​the media asset screen, wherein the first transformation matrix is ​​used to convert coordinates of the target layer in a content coordinate system into coordinates of the screen coordinate system, and the third position area is a position area of ​​the media asset screen in the content coordinate system of the target layer; Extracting a coordinate matrix of the third position area; The coordinate matrix of the third position region is transformed by the first transformation matrix to obtain the first position region by the transformed coordinate matrix.

4. The display device according to claim 2, characterized in that The controller calculates the second position area of ​​the media asset screen according to the first position area and the first layer, and is specifically configured as follows: Obtain an inverse matrix of a second transformation matrix of the first layer, where the second transformation matrix is ​​used to convert coordinates in a content coordinate system of the first layer into coordinates in the screen coordinate system; Extracting a coordinate matrix of the first location area; The coordinate matrix of the first position region is transformed by the inverse matrix of the second transformation matrix to obtain the second position region by the transformed coordinate matrix.

5. The display device according to claim 2, characterized in that The controller executes drawing the display layer and setting the target area in the display layer as a transparent area, and is specifically configured as follows: Determining a drawing order according to the display level of the display layer, the drawing order of the second layer is earlier than the drawing order of the first layer; Drawing the display layer through the renderer of the browser thread according to the drawing order; When the displayed layer to be drawn is the second layer, drawing the second layer by the renderer; When the displayed layer to be drawn is the first layer, the first layer is drawn by the renderer, and the target area of ​​the first layer is set as a transparent area.

6. The display device according to claim 5, characterized in that The controller executes setting the target area of ​​the first layer as a transparent area, and is specifically configured as follows: generating a unit drawing instruction according to the target area associated with the first layer, wherein the unit drawing instruction carries the location information of the target area; Sending the unit drawing instruction to the first layer; Detecting, by the renderer, a target area of ​​the first layer in response to the unit drawing instruction; The transparency value of the target area in the first layer is set as a target value, and the target value is smaller than the transparency value of the media asset screen.

7. The display device according to claim 1, characterized in that After the controller executes the step of controlling the display to display the play window, the controller is further configured to: Monitoring the display status of the playback window; If the display state of the playback window is full screen, the coordinate matrix of the media resource screen in the page coordinate system is marked as the returned coordinates, and the returned coordinates are sent to the back end; the page coordinate system is the logical coordinate system corresponding to the browser page; If the display state of the playback window is not full screen, the coordinate matrix of the media resource screen in the page coordinate system is converted into a target matrix, the target matrix is ​​marked as the return coordinates, and the return coordinates are sent to the back end.

8. The display device according to claim 7, characterized in that If the display state of the play window is a non-full screen state, the controller is further configured to: Acquire first size information of the media resource screen in the page coordinate system, and acquire second size information of the playback window in the screen coordinate system; parsing a mapping relationship between the first size information and the second size information; Generate a first conversion formula according to the mapping relationship, the first conversion formula is used to represent the process of converting a first coordinate matrix into a second coordinate matrix, the first coordinate matrix is ​​the coordinate matrix of the media asset screen in the page coordinate system, and the second coordinate matrix is ​​the coordinate matrix of the media asset screen in the screen coordinate system; A second transformation formula is generated by the second coordinate matrix and the mapping relationship. The second transformation formula is used to characterize the process of converting the second coordinate matrix into a third coordinate matrix. The third coordinate matrix is ​​a coordinate matrix in the page coordinate system corresponding to the second coordinate matrix in the screen coordinate system.

9. The display device according to claim 8, characterized in that The controller executes conversion of the coordinate matrix of the media resource screen in the page coordinate system into a target matrix, and is specifically configured as follows: Generate a target transformation formula according to the first transformation formula and the second transformation formula, wherein the target transformation formula is used to represent a process of transforming the first coordinate matrix into the third coordinate matrix; Calculate the third coordinate matrix according to the first coordinate matrix and the target transformation formula; The third coordinate matrix is ​​labeled as the target matrix.

10. A method for displaying a media resource screen, characterized in that: include: In response to a media playback instruction, controlling the display to display a playback window, wherein the playback window is used to display a browser page, wherein the browser page includes a media screen and at least two display layers; Calculating a target area of ​​the display layer, where the target area is an overlapping area between the media asset screen and the display layer; Drawing the display layer, and setting the target area in the display layer as a transparent area; The media resource picture is decoded according to the returned coordinates, so that the media resource picture is displayed in the display area corresponding to the returned coordinates; the returned coordinates are determined according to the display state of the playback window, and the display state includes a full-screen state and a non-full-screen state.

Images