Display device and screen projection continuous playing method

Abstract

The embodiment of the invention provides a display device and a screen projection continuous playing method, and the method can respond to a playing instruction of a screen projection picture to obtain continuous playing information of the screen projection picture. Wherein the continuous playing information is information recorded when the screen projection picture is quitted from playing, and the continuous playing information comprises a screen projection protocol type, a media resource playing address and a media resource playing progress. And creating a virtual session object based on a screen projection protocol type in the continuous playing information, acquiring screen projection data from the media asset playing address through the virtual session object, and playing the screen projection data according to the media asset playing progress. According to the method, the continuous playing information can be recorded when the display device is switched out of the screen projection interface, and when the display device returns to the screen projection interface, the display device is supported to acquire the previously stored continuous playing information and jump to the corresponding media resource playing progress according to the continuous playing information, so that continuous playing of the screen projection picture is realized, and the problem that the continuous playing cannot be realized in the screen projection playing process is solved.

Description

Technical Field

[0001] This application relates to the technical field of display devices, and particularly to a display device and a screen mirroring continuous playback method. Background Art

[0002] A display device is an intelligent device that can present a user interface and support user interaction. Taking a smart TV as an example, a smart TV is based on Internet application technology, has an open operating system and a chip, and has an open application platform. It can realize two-way human-computer interaction functions and integrates various functions such as audio-visual, entertainment, and data. It is a TV product used to meet the diverse and personalized needs of users. The display device can establish a communication connection with a terminal device such as a mobile phone, and based on a specific screen mirroring protocol, obtain the screen mirroring data of the terminal device to display the screen mirroring of the terminal device through the display device.

[0003] Since the data source of the screen mirroring is provided or pushed by the terminal device during the process of the display device displaying the screen mirroring, and the display device and the terminal device are two independent individuals, they cannot be played and controlled in the conventional video playback manner. For example, during the process of the display device displaying the screen mirroring through the screen mirroring playback interface, when the user presses the return key on the remote control to control the display device to jump to another interface, the display device will disconnect the screen mirroring connection during the jump and no longer receive the screen mirroring data transmitted by the terminal device. Therefore, when the user controls the display device to return to the state of displaying the screen mirroring, the display device needs to re-push the screen mirroring through the terminal device, and the user needs to manually fast forward to the previous playback progress to complete the continuous playback of the screen mirroring, which reduces the user experience. Summary of the Invention

[0004] Some embodiments of this application provide a display device and a screen mirroring continuous playback method to solve the problem that continuous playback cannot be achieved during the screen mirroring playback process.

[0005] On the one hand, some embodiments of this application provide a display device, including: a display, a communicator, and a controller. Among them, the display is configured to display a screen mirroring interface; the communicator is configured to establish a screen mirroring connection with a terminal device; the controller is configured to perform the following steps:

[0006] In response to a playback instruction for playing the screen mirroring, obtain the continuous playback information of the screen mirroring, where the continuous playback information is the information recorded when exiting the playback of the screen mirroring, and the continuous playback information includes the screen mirroring protocol type, the media playback address, and the media playback progress;

[0007] Create a virtual session object based on the continuous playback information, where the virtual session object is used to establish a virtual screen mirroring connection with the display device on behalf of the terminal device according to the screen mirroring protocol type;

[0008] Obtain the screen mirroring data from the media asset playback address through the virtual session object;

[0009] Play the screen mirroring data according to the media asset playback progress.

[0010] On the other hand, some embodiments of the present application also provide a screen mirroring resume playback method, which is applied to the above display device. The screen mirroring resume playback method includes:

[0011] In response to a playback instruction for playing the screen mirroring picture, obtain the resume playback information of the screen mirroring picture. The resume playback information is the information recorded when exiting the playback of the screen mirroring picture, and the resume playback information includes the screen mirroring protocol type, the media asset playback address, and the media asset playback progress;

[0012] Create a virtual session object based on the resume playback information. The virtual session object is used to establish a virtual screen mirroring connection with the display device on behalf of the terminal device according to the screen mirroring protocol type;

[0013] Obtain the screen mirroring data from the media asset playback address through the virtual session object;

[0014] Play the screen mirroring data according to the media asset playback progress.

[0015] As can be seen from the above technical solutions, some embodiments of the present application provide a display device and a screen mirroring resume playback method. The method can, in response to a playback instruction for the screen mirroring picture, obtain the resume playback information of the screen mirroring picture. Among them, the resume playback information is the information recorded when exiting the playback of the screen mirroring picture, and the resume playback information includes the screen mirroring protocol type, the media asset playback address, and the media asset playback progress. Then, create a virtual session object based on the screen mirroring protocol type in the resume playback information, obtain the screen mirroring data from the media asset playback address through the virtual session object, and play the screen mirroring data according to the media asset playback progress. The method can record the resume playback information when the display device cuts out the screen mirroring interface, and support the display device to obtain the previously saved resume playback information when returning to the screen mirroring interface, and jump to the corresponding media asset playback progress according to the resume playback information, so as to realize the resume playback of the screen mirroring picture, and solve the problem that the screen mirroring playback process cannot be resumed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings.

[0017] Figure 1 It is a schematic diagram of the operation scenario between the display device and the control device provided by some embodiments of the present application;

[0018] Figure 2 Schematic diagram of the hardware configuration of the display device provided by some embodiments of the present application;

[0019] Figure 3 Schematic diagram of the hardware configuration of the control device provided by some embodiments of the present application;

[0020] Figure 4 Schematic diagram of the software configuration of the display device provided by some embodiments of the present application;

[0021] Figure 5 Schematic diagram of mirror screen projection provided by some embodiments of the present application;

[0022] Figure 6 Schematic diagram of push screen projection provided by some embodiments of the present application;

[0023] Figure 7 Schematic diagram of the process of recording continuous playback information provided by some embodiments of the present application;

[0024] Figure 8 Schematic diagram of the process of generating continuous playback information according to the remaining duration provided by some embodiments of the present application;

[0025] Figure 9 Schematic diagram of the screen projection start interface provided by some embodiments of the present application;

[0026] Figure 10 Schematic diagram of the process of generating thumbnail images provided by some embodiments of the present application;

[0027] Figure 11 Schematic diagram of the playback function module provided by some embodiments of the present application;

[0028] Figure 12 Schematic diagram of the process of deleting continuous playback information provided by some embodiments of the present application;

[0029] Figure 13 Schematic diagram of the screen projection continuous playback process provided by some embodiments of the present application;

[0030] Figure 14 Schematic diagram of the process of continuous screen projection playback through a virtual session object provided by some embodiments of the present application;

[0031] Figure 15 Schematic diagram of the process of performing screen projection continuous playback according to the application type provided by some embodiments of the present application;

[0032] Figure 16 Schematic diagram of the continuous screen projection playback process of the second type of application provided by some embodiments of the present application. Detailed implementation manners

[0033] Embodiments will be described in detail below, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with the present application. They are merely examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0034] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described next, rather than intending to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood in their ordinary and usual meanings.

[0035] The terms "first", "second", "third", etc. in the specification, claims and above-mentioned drawings of this application are used to distinguish similar or like objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchanged under appropriate circumstances.

[0036] The terms "comprising" and "having" and any variations thereof are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components does not necessarily have to be limited to all the components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0037] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or a combination of hardware or / and software code that can perform functions related to that element.

[0038] Figure 1 It is a schematic diagram of an operation scenario between a display device and a control device provided for some embodiments of this application. As Figure 1 shown, the user can operate the display device 200 through touch operations, a mobile terminal 300 and a control device 100. Among them, the control device 100 can be a remote control, a stylus, etc.

[0039] In some embodiments, the mobile terminal 300 can install a software application with the display device 200 and achieve connection communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. It is also possible to transmit the audio and video content displayed on the mobile terminal 300 to the display device 200 to achieve a synchronous display function.

[0040] As Figure 1 also shown, the display device 200 also conducts data communication with the server 400 through various communication methods. The display device 200 is allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks.

[0041] In addition to providing the function of receiving broadcast television, the display device 200 can additionally provide an intelligent network television function with computer support functions, including but not limited to, Internet Protocol Television (IPTV), smart TV, etc.

[0042] Figure 2 For some embodiments of this application Figure 1 The hardware configuration block diagram of the display device 200 in

[0043] In some embodiments, the display device 200 includes at least one of a tuner demodulator 210, a communicator 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0044] In some embodiments, the detector 230 is used to collect signals from the external environment or interact with the outside. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures. Or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0045] In some embodiments, the display 260 includes a display screen component for presenting a picture, and a driving component for driving image display, which is used to receive an image signal output from the controller and display video content, image content, a menu control interface, and a user control UI interface, etc.

[0046] In some embodiments, the communicator 220 is a component for communicating with external devices or a server 400 according to various communication protocol types. The display device 200 can be provided with multiple communicators 220 according to different supported communication methods. For example, when the display device 200 supports wireless network communication, the display device 200 can be provided with a communicator 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communicator 220 including a Bluetooth function.

[0047] The communicator 220 can communicate and connect the display device 200 with external devices or the server 400 in a wireless or wired connection manner. Among them, the wired connection can connect the display device 200 with external devices through components such as data lines and interfaces. The wireless connection can connect the display device 200 with external devices through wireless signals or wireless networks. The display device 200 can directly establish a connection relationship with external devices or indirectly establish a connection relationship through gateways, routers, connection devices, etc.

[0048] In some embodiments, the display device 200 may also support establishing communication connection relationships with multiple external devices simultaneously. Multiple external devices may use the same connection method. For example, the external devices are terminal devices 500 such as mobile phones and tablets. The first terminal device 510 and the second terminal device 520 may establish a communication connection with the display device 200 by accessing the wireless local area network where the display device 200 is located. Multiple external devices may also connect to the display device 200 through different types of connection methods. For example, the first terminal device 510 connects to the display device 200 through a wireless local area network; the second terminal device 520 connects to the display device 200 through Bluetooth.

[0049] To implement the communication connection between the display device 200 and the terminal device 500, communicators that support the same communication type are respectively provided on the display device 200 and the terminal device 500. For example, when the display device 200 is provided with a communicator 220 including a WiFi module, the terminal device 500 should also be provided with a communicator including a WiFi module. Moreover, while the same type of communicators are provided, a specific communication transmission protocol is also required between the display device 200 and the terminal device 500 for data transfer. For example, WiFi protocol, Bluetooth connection protocol, ZigBee protocol, NFC protocol, etc.

[0050] In some embodiments, the controller 250 includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0051] In some embodiments, the controller 250 and the tuner demodulator 210 may be located in different split devices, that is, the tuner demodulator 210 may also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0052] In some embodiments, the user may input a user command on the graphical user interface (GUI) displayed on the display 260, and then the user input interface receives the user input command through the graphical user interface (GUI).

[0053] In some embodiments, the user interface 280 is an interface that can be used to receive control inputs.

[0054] Figure 3 For some embodiments provided in this application Figure 1 The hardware configuration block diagram of the control device. As Figure 3As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0055] The control device 100 is configured to control the display device 200, and can receive user input operation instructions, and convert the operation instructions into instructions recognizable and responsive by the display device 200, acting as an interaction intermediary between the user and the display device 200.

[0056] In some embodiments, the control device 100 can be an intelligent device. For example, the control device 100 can install various applications for controlling the display device 200 according to user needs.

[0057] In some embodiments, as Figure 1 shown, after installing an application for controlling the display device 200, the mobile terminal 300 or other intelligent electronic devices can perform functions similar to those of the control device 100.

[0058] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between internal components and the data processing functions between the external and internal.

[0059] Under the control of the controller 110, the communication interface 130 realizes the communication of control signals and data signals with the display device 200. The communication interface 130 can include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, and other near-field communication modules.

[0060] The user input / output interface 140, where the input interface includes at least one of a microphone 141, a touchpad 142, a sensor 143, a button 144, and other input interfaces.

[0061] In some embodiments, the control device 100 includes at least one of the communication interface 130 and the input / output interface 140. The communication interface 130 is configured in the control device 100. For example, modules such as WiFi, Bluetooth, and NFC can encode user input instructions through the WiFi protocol, or the Bluetooth protocol, or the NFC protocol and send them to the display device 200.

[0062] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0063] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0064] As Figure 4 , in some embodiments, the system is divided into four layers, from top to bottom are the Applications layer (referred to as the "application layer" for short), the Application Framework layer (referred to as the "framework layer" for short), the Android runtime and the system libraries layer (referred to as the "system runtime layer" for short), and the kernel layer.

[0065] In some embodiments, at least one application runs in the application layer. These applications can be the window programs, system setting programs, or clock programs etc. that come with the operating system; or they can be applications developed by third-party developers. In specific implementation, the application packages in the application layer are not limited to the above examples.

[0066] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center, and this center determines the actions of the applications in the application layer. Applications can access the resources in the system and obtain the services of the system through the API interfaces during execution.

[0067] As Figure 4 shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc. Among them, the view system can design and implement the interfaces and interactions of applications. The view system includes lists, grids, text boxes, buttons, etc. The managers include at least one of the following modules: The Activity Manager is used to interact with all the activities running in the system; the Location Manager is used to provide access to the system location service for system services or applications; the Package Manager is used to retrieve various information related to the application packages currently installed on the device; the Notification Manager is used to control the display and clearing of notification messages; the Window Manager is used to manage the icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0068] In some embodiments, the activity manager is used to manage the life cycles of various applications and the general navigation back function, such as controlling the exit, opening, and backward movement of applications. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling the changes of the display window. For example, shrinking the display window, jittering the display, distorting the display, etc.

[0069] In some embodiments, the system runtime layer provides support for the upper layer, i.e., the framework layer. When the framework layer is used, the Android operating system will run the C / C++ libraries included in the system runtime layer to implement the functions to be achieved by the framework layer.

[0070] In some embodiments, the kernel layer is the layer between hardware and software. As Figure 4 shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0071] After the above display device 200 establishes a communication connection relationship with the terminal device 500, it can establish a screen mirroring connection channel and transmit screen mirroring data based on the screen mirroring connection, so that the display device 200 can display the screen mirroring of the terminal device 500. In some embodiments, in order to establish a screen mirroring connection relationship, the user can send a screen mirroring connection request through the terminal device 500 and send the screen mirroring connection request to the display device 200 through the WiFi network. The display device 200 then completes the transmission protocol configuration according to the screen mirroring connection request, thereby establishing a transmission channel for screen mirroring data with the terminal device 500.

[0072] The display device 200 can also achieve screen mirroring connection with the terminal device 500 through different connection methods. For example, when the display device 200 and the terminal device 500 are connected to the same wireless local area network, a screen mirroring connection can be established based on the WiFi network. Another example is that when both the display device 200 and the terminal device 500 are equipped with near field communication (NFC) components, a screen mirroring connection relationship can be established through the NFC components. Obviously, other wired or wireless connection methods can also be adopted between the display device 200 and the terminal device 500 to establish a screen mirroring connection relationship, such as RF radio frequency connection, infrared connection, cellular network, and other connection methods that those skilled in the art can think of based on the connection methods provided in the above embodiments.

[0073] After establishing the screen mirroring connection, the display device 200 can receive screen mirroring-related data from the terminal device 500 through the screen mirroring data channel, so as to play screen mirroring-related content on the display device 200. Under different screen mirroring data transmission protocols, the forms of screen mirroring data obtained by the display device 200 are different. For example, the display device 200 and the terminal device 500 can establish a screen mirroring connection through protocols such as the Digital Living Network Alliance (DLNA) protocol, the Miracast protocol, the AirPlay push protocol, WiDi screen mirroring, NFC mirroring screen mirroring, and LeEco screen mirroring.

[0074] Among them, the DLNA protocol is a protocol for realizing media resource sharing between devices. This protocol uses network technology (wired or wireless) to interconnect various devices and perform data interaction through standard protocols, enabling media resources to be shared between devices. The DLNA protocol is based on the UPnP protocol, enabling devices to discover and control each other and realizing seamless transmission and sharing of media resources. The DLNA protocol supports a variety of devices, including TVs, speakers, mobile phones, computers, etc., allowing users to seamlessly switch and share media content between different devices.

[0075] The Miracast protocol is a wireless display standard developed by the Wi-Fi Alliance and based on Wi-Fi Direct. The Miracast screen mirroring protocol is applicable to Android devices, that is, Android devices can share video images through the Miracast screen mirroring method. Smart devices above Android 5.0 have the Miracast protocol built in, so they can directly perform wireless screen mirroring through the Wi-Fi P2P function between the screen mirroring device and the receiving device's Wi-Fi network card without installing any software. In addition, Miracast screen mirroring also supports synchronous transmission of audio and video.

[0076] The Airplay protocol is a wireless playback technology launched by Apple and is a wireless screen mirroring protocol applicable to iOS and Mac systems. Apple series devices have the Airplay screen mirroring function built in. When used with a screen mirroring device that supports the AirPlay server service, the effect is clear and stable, and the protocol has a built-in screen mirroring code function. However, it is necessary to ensure that the Apple device and the screen mirroring device are used in the same local area network (same network segment) and cannot be used across network segments or VLANs.

[0077] WiDi screen mirroring is a wireless playback technology led by Intel and is also based on the Wi-Fi Direct technology. Windows 8 / 10 / 11 laptops natively support WiDi screen mirroring and can perform screen mirroring without installing any software, and also support the USB reverse control function.

[0078] In some embodiments, after the display device 200 establishes a screen mirroring connection with the terminal device 500, screen mirroring data can be transmitted through the mirroring mode and the push mode. Among them, the mirroring mode means that the display device 200 synchronously displays the screen content displayed by the terminal device 500. For example, as Figure 5 shown, the display device 200 displays the main interface of the terminal device 500, and when the user performs an interaction operation on the terminal device 500, resulting in a change in the main interface content, the content displayed by the display device 200 also changes accordingly. To this end, the terminal device 500 can generate a screen mirroring data stream in the form of a video based on the displayed content, and send the screen mirroring data stream to the display device 200 for display.

[0079] The push mode means that the display device 200 accesses the network link pushed by the terminal device 500 and obtains the corresponding screen mirroring screen. For example, as Figure 6 shown, when the terminal device 500 plays video media assets, a screen mirroring control is provided in the upper right corner of the playback interface. After the user clicks on the screen mirroring control and selects the display device 200 as the device for displaying the screen mirroring screen, the terminal device 500 can send the URL address of the currently played video media asset to the display device 200. After receiving the URL address, the display device 200 accesses the URL address to obtain the video media asset data and plays the video media asset.

[0080] Among the screen mirroring protocols supported by the display device 200 and the terminal device 500, some protocols are based on the mirroring mode, and some protocols are based on the push mode. For example, under the DLNA protocol, the display device 200 can obtain the link address of a certain resource in the terminal device 500 through the screen mirroring data channel, and obtain the media asset content data by accessing the link address, so as to play the media asset data. At the same time, when playing the media asset data, the display device 200 can also obtain the control instructions sent by the terminal device 500 through the screen mirroring data channel, such as fast forward, pause, stop screen mirroring, etc. For example, after establishing a connection through the communication protocol, the terminal device 500 such as a mobile phone and the display device 200 will negotiate the device functions and network conditions to select a suitable audio and video transmission format. After establishing the connection protocol (Session) for the audio and video stream, a series of Real Time Streaming Protocol (RTSP) control commands are used to control the playback and termination of the audio and video stream.

[0081] Under other screen mirroring connection protocols such as Miracast screen mirroring, Airplay screen mirroring, NFC mirroring screen mirroring, and LeEco screen mirroring, the display device 200 can directly obtain the video data stream from the terminal device 500. The video data stream can be the video data stream corresponding to the final display picture of the display unit of the terminal device 500, that is, the video data stream includes the media content picture, the UI interface, and other video picture contents that can be displayed in the display unit in the terminal device 500. The video data stream can also be only the video data that can be obtained in the terminal device 500, such as the media video data being played, etc. At this time, the terminal device 500 can be used as a signal source of the display device 200, and the display device 200 can continuously obtain video data through the screen mirroring data channel and display it in real time on the display 260, achieving the effect of synchronous display with the terminal device 500.

[0082] It should be noted that the above screen mirroring protocol can not only obtain the video data stream in the terminal device 500, but also obtain the audio data stream in the terminal device 500, so that the video data and audio data can be sent to the display device 200 synchronously, enabling the display device 200 to play video and audio synchronously.

[0083] The terminal device 500 can obtain different screen mirroring data according to different screen mirroring protocols. For example, for system applications or direct screen mirroring at the system level, the terminal device 500 can use Miracast screen mirroring, Airplay screen mirroring, or WiDi screen mirroring protocol to establish a screen mirroring data channel. Since the above Miracast screen mirroring protocol and Airplay screen mirroring protocol require system-level application permissions, the Miracast screen mirroring protocol and Airplay screen mirroring protocol are only applicable to system applications, or the display device 200 and the terminal device 500 of the same manufacturer directly establish a screen mirroring connection through the built-in rules of the operating system.

[0084] After the display device 200 establishes a screen mirroring connection with the terminal device 500 based on any one of the above screen mirroring protocols, it can form a screen mirroring picture according to the screen mirroring data sent by the terminal device 500. The screen mirroring picture can be displayed through a specific user interface. In the embodiment of the present application, the user interface for displaying the screen mirroring picture can be called a screen mirroring interface. It should be understood that the screen mirroring interface can be an independent interface. For example, the screen mirroring interface is a floating screen mirroring window. The screen mirroring interface can also be the playback interface of the display device 200. For example, after establishing a screen mirroring connection, when the display device 200 displays the screen mirroring content full screen, the entire full-screen display picture content is the screen mirroring interface.

[0085] Due to the special nature of the data source of the screen mirroring display, that is, after the display device 200 obtains the screen mirroring data from the terminal device 500, it plays the data to form the screen mirroring display. Therefore, the playback control of the screen mirroring data by the display device 200 is restricted by conditions such as the screen mirroring protocol and the terminal device 500, making the display device 200 in a passive playback mode and difficult to complete playback recording, adjustment, switching and other control processes according to the media asset playback process.

[0086] For example, in the mirror screen mirroring mode, the display device 200 can only display the mirror screen mirroring data stream sent by the terminal device 500 in real time, and the control of the screen mirroring process is limited to basic operations such as exiting. In the push screen mirroring mode, the display device 200 accesses the media asset playback address pushed by the terminal device 500 to obtain the screen mirroring data. However, since the screen mirroring data is pushed by the terminal device 500 and the data source is still defaulted to the terminal device 500, when the display device 200 exits the screen mirroring interface, it will directly disconnect the screen mirroring connection relationship and will not record any playback data. If the user wants to resume playing the screen mirroring display, they need to re-establish the screen mirroring connection relationship between the display device 200 and the terminal device 500, and manually adjust to the playback progress at the time of exit through the fast forward operation to resume playing the screen mirroring display. It can be seen that such a screen mirroring resume playback method is cumbersome to operate and has a long resume playback waiting time, which will reduce the user experience.

[0087] To solve the problem that the screen mirroring process cannot be resumed, some embodiments of the present application provide a display device 200. The display device 200 can record the resume playback information when exiting the screen mirroring, and when the user selects to resume playing the screen mirroring display, automatically jump to the playback progress at the time of exit according to the resume playback information, so as to achieve the purpose of automatically resuming the screen mirroring display. To implement the above functions, the display device 200 should at least include a display 260, a communicator 220, and a controller 250. Among them, the display 260 is configured to display the screen mirroring interface. The communicator 220 is configured to establish a screen mirroring connection with the terminal device 500, and the controller 250 is configured to execute the steps corresponding to the screen mirroring resume playback method, such as Figure 7 shown, including the following content:

[0088] S100: Obtain an exit instruction for exiting the screen mirroring interface.

[0089] In the embodiments of the present application, the exit instruction for exiting the screen mirroring interface can include two types. One type is an active exit instruction, that is, an exit instruction for the user to actively exit the screen mirroring interface through a specific interaction operation. For example, during the process of the display device 200 displaying the screen mirroring interface, the user presses the return key on the remote control supporting the display device 200 to control the display device 200 to exit the screen mirroring interface.

[0090] In some embodiments, the user can input an exit instruction based on different interaction operations. For example, for the display device 200 that supports touch interaction operations, when the display device 200 displays the screen mirroring interface, the user can control the display device 200 to exit the screen mirroring interface by means of an exit gesture of swiping upward from the bottom edge of the display 260, that is, the exit instruction can be input through touch interaction. Also, for example, for the display device 200 that supports voice interaction operations, when the display device 200 displays the screen mirroring interface, the user can input a voice such as "exit screen mirroring" to control the display device 200 to exit the screen mirroring interface, that is, the exit instruction can be input through voice interaction.

[0091] Another type is a passive exit instruction, that is, an exit instruction that is generated passively when the display device 200 switches the display content from the screen mirroring interface to another interface according to the running state or user interaction actions. For example, during the process of the display device 200 displaying the screen mirroring interface, the user inserts a game device into the HDMI interface. At this time, the display device 200 will automatically jump to the signal source setting interface, and at this time, the display device 200 can generate an exit instruction according to the plugging and unplugging of the signal.

[0092] According to the interface switching rules set by the operating system of the display device 200, the passive exit instruction can be generated in different running states, and the exit instruction can also be input or generated by the control device 100 or an external device. For example, when the user presses the home key on the remote control to control the display device 200 to switch from the screen mirroring interface to the home interface, the display device 200 can generate and execute an exit instruction. Also, for example, for the display device 200 that supports video (or voice) calls, if a video call request is received when the display device 200 displays the screen mirroring interface, the display device 200 can also switch from the screen mirroring interface to the video call interface.

[0093] S200: In response to the exit instruction, record the screen mirroring protocol type and the media resource playback address of the screen mirroring interface, and record the media resource playback progress at the moment when the exit instruction is input.

[0094] After obtaining the exit instruction, the display device 200 can record the information related to screen mirroring in response to the exit instruction. Among them, the information related to screen mirroring can be used to resume playing the screen mirroring video. Therefore, the information related to screen mirroring should include the information that can determine the content of the screen mirroring video and the information used to achieve the resume of screen mirroring, that is, the information related to screen mirroring can include the screen mirroring protocol type, the media resource playback address corresponding to the screen mirroring video, and the media resource playback progress.

[0095] To obtain the screen mirroring protocol type, the display device 200 can read the configuration file of the screen mirroring interface. The configuration file may record the screen mirroring protocol used by the display device 200 when establishing a screen mirroring connection with the terminal device 500. For example, when the user controls the display device 200 to establish a screen mirroring connection with the terminal device 500 through the system default DLNA protocol, when the display device 200 renders the screen mirroring interface, it will first create a configuration file and record the DLNA protocol related fields in the configuration file. When recording the screen mirroring protocol type, the display device 200 can call the configuration file again and read the DLNA protocol related fields from the configuration file, so as to record that the current screen mirroring protocol type is the DLNA protocol.

[0096] In some embodiments, the display device 200 can also determine the screen mirroring protocol type according to the screen mirroring data. Since different screen mirroring protocols can use different data formats and compression algorithms for screen mirroring data, the screen mirroring protocol type can be determined by reading the data format and compression algorithm of the screen mirroring data. For example, the Miracast protocol uses the WiFi Direct technology for data transmission, while the AirPlay protocol uses Apple's proprietary data format for transmission. Then, when the data transmission format conforms to the WiFi Direct technology, the screen mirroring protocol type can be determined to be the Miracast protocol.

[0097] In some embodiments, the display device 200 can also obtain and record the screen mirroring protocol type according to the screen mirroring application. For example, when the A video application is installed and running on both the display device 200 and the terminal device 500, the user can click the screen mirroring control at the upper right corner of the A video application interface on the terminal device 500, and then the terminal device 500 can establish a screen mirroring connection with the display device 200 through the screen mirroring protocol supported by the A video application. Therefore, when recording the screen mirroring protocol type, the display device 200 can determine that the current screen mirroring application is the A video application by reading the package name information of the screen mirroring application, and then query the default or supported screen mirroring protocol of the A video application to obtain the screen mirroring protocol type.

[0098] When recording the screen mirroring protocol type, the display device 200 can also record the media asset playback address. Among them, the media asset playback address can be obtained by reading the signal source address of the player. For example, after the display device 200 and the terminal device 500 establish a screen mirroring connection, the terminal device 500 can send the URL address of the screen mirroring data to the display device 200 based on the push screen mirroring method, and the display device 200 sets the data source of the player according to the received URL address. Therefore, when recording the media asset playback address, the display device 200 can directly read the data source of the player to obtain the media asset playback address.

[0099] Since the configuration file of the screen mirroring interface is created when the screen mirroring connection relationship is established, when creating the configuration file, the display device 200 can also write the media resource playback address sent by the terminal device 500 into the configuration file. Therefore, in some embodiments, the display device 200 can also obtain and record the media resource playback address by reading the configuration file.

[0100] Based on the above embodiments, while recording the screen mirroring protocol type and the media resource playback address, the display device 200 can also record the media resource playback progress according to the input time of the exit instruction, that is, record the media resource playback progress at the input time of the exit instruction. Among them, the media resource playback progress can be represented by the playback time or by a percentage. For example, the display device 200 can obtain the start time T0 of the player playing the screen mirroring data and the total duration T1 of the media resource of the screen mirroring data through the decoding result of the screen mirroring data, and then combine the input time T of the exit instruction to calculate the media resource playback progress. When representing the media resource playback progress by time, the display device 200 can calculate the difference between the input time T and the start time T0, that is, T - T0 to obtain the media resource playback progress. When representing the media resource playback progress by a percentage, the display device 200 can, after calculating the difference between the input time T and the start time T0, calculate the percentage of the difference to the total duration T1 of the media resource, that is, the media resource playback progress is equal to (T - T0) / T1.

[0101] In some embodiments, in addition to recording the screen mirroring protocol type, the media resource playback address, and the media resource playback progress, the display device 200 can also record other relevant information that facilitates the user to perform interactive operations, including the user's setting information and the display content information. For example, the display device 200 can, in response to the exit instruction, also record the media resource ID (name, title, etc.), the thumbnail, and information such as the playback speed, audio track, subtitle, and zoom mode set by the user.

[0102] It should be noted that for the display device 200, the active exit instruction is actively input by the user and can include the user's intention to actively stop screen mirroring. The passive exit instruction is caused by events such as accidental operations and therefore does not include the user's intention to actively stop screen mirroring. In a state without the intention of actively stopping screen mirroring, the user is more in need of continuing the screen mirroring. Therefore, in some embodiments, the display device 200 can record the resume playback information only when receiving the passive exit instruction.

[0103] During the process of recording the playback progress of media assets, the display device 200 can also detect the playback progress of media assets and the type of media assets to determine whether the playback progress or the type of media assets supports continued playback via screen mirroring. For example, for video media assets such as movies, when the playback progress of the media asset is relatively large and the main feature film has been played, performing continued playback only plays the content of the ending song, which has little significance for user use. Therefore, continued playback can no longer be performed on this screen-mirrored image, that is, the continued playback information corresponding to this screen-mirrored image is not recorded. Another example is live video. Since it has no fixed length and cannot achieve continued playback, the continued playback information corresponding to this screen-mirrored image can also not be recorded.

[0104] In this regard, as Figure 8 shown, in some embodiments, after executing the recording of the media playback address and media playback progress at the moment when the exit instruction is input, the display device 200 can obtain the total duration of the media of the screen-mirrored image (S801), and calculate the remaining duration of the media based on the total duration of the media and the media playback progress (S802).

[0105] If the remaining duration of the media is greater than or equal to a preset duration threshold, it indicates that the remaining duration of the screen-mirrored media is relatively long and is convenient for continued playback via screen mirroring. Then, the display device 200 can generate the continued playback information based on the screen mirroring protocol type, media playback address, and media playback progress (S803). If the remaining duration of the media is less than the preset duration threshold, it indicates that the remaining duration of the screen-mirrored media is relatively short and is not convenient for continued playback via screen mirroring. Therefore, the display device 200 can directly close the screen mirroring connection channel with the terminal device 500 (S804).

[0106] For example, when the display device 200 exits the screen mirroring interface, it can determine whether the video corresponding to the screen-mirrored image has been played or is a live video. If the remaining duration of the video is greater than 1 minute, it is determined that the current screen-mirrored image supports continued playback. Therefore, the display device 200 can record the media playback address and media playback progress and generate continued playback information. If the remaining duration of the video is less than 1 minute, it is determined that the current screen-mirrored image does not support continued playback, so it can directly exit. When it is detected that the playback duration (or remaining duration) is 0, it can be determined that the current screen-mirrored image is a live video and also does not support continued playback, so it can also directly exit.

[0107] S300: Generate the continued playback information based on the screen mirroring protocol type, media playback address, and media playback progress.

[0108] After recording the screen mirroring protocol type, media asset playback address, and media asset playback progress, the display device 200 can convert the recorded information according to a preset data format to generate resume playback information. In some embodiments, the display device 200 can record and manage the resume playback information through a specific application. That is, there is a screen mirroring entry application in the display device 200, and the screen mirroring entry application is an application that provides a screen mirroring start interface. For example, the display device 200 can save the information recorded in the above embodiments in the screen mirroring entry application through the Setting.Global process for the screen mirroring entry application to display the screen mirroring start interface.

[0109] Since the screen mirroring process is equivalent to providing a screen mirroring signal source for the display device 200, the screen mirroring start interface can be built into the signal source selection interface or be the same interface as the signal source selection interface. As Figure 9 shown, in the screen mirroring start interface, there can be multiple signal source options, such as a live TV signal source, an HDMI interface signal source, a USB interface signal source, a screen mirroring signal source, etc. These signal source options can display the signal source type through view controls such as title text, icons, and thumbnails for the user to select.

[0110] Based on this, in some embodiments, after the display device 200 records information such as the screen mirroring protocol type, media asset playback address, and media asset playback progress, it can record the content of the screen mirroring screen at the moment when the exit instruction is input. That is, as Figure 10 shown, the display device 200 can generate a resume playback image according to the media asset playback progress (S1001), perform compression processing on the screen mirroring image to generate a thumbnail (S1002), and then add the thumbnail to the resume playback information (S1003).

[0111] Among them, the resume playback image is used to display the content of the screen mirroring screen at the moment when the exit instruction is input. According to different image acquisition rules, the display device 200 can adopt different ways to acquire the resume playback image. In some embodiments, the display device 200 can extract the resume playback image from the data stream of the screen mirroring screen, that is, the resume playback image is a frame extracted from the data stream of the screen mirroring screen according to the media asset playback progress. After the display device 200 establishes a screen mirroring connection relationship with the terminal device 500, the display device 200 can obtain screen mirroring data based on the URL address sent by the terminal device 500 and form a screen mirroring data stream. The screen mirroring data stream includes multiple consecutive frame images, and each frame image is provided with a timestamp for playback. The display device 200 can record the input time of the exit instruction when obtaining the exit instruction, read the timestamp of the frame image according to the input time, determine the frame image with the smallest time difference from the input time, and copy the frame image to generate the resume playback image.

[0112] In some embodiments, the display device 200 may execute a screenshot of the current screen mirroring interface by invoking a screenshot process to obtain a resume-playing image. That is, the resume-playing image is an image obtained by executing a screenshot of the screen mirroring screen by invoking a screenshot interface. For example, when the display device 200 receives an exit instruction, it may call the developer API of Android to invoke the system's screenshot process. That is, the display device 200 first invokes the screenshot process by the instruction "import android.graphics.Bitmap; import android.graphics.Canvas; import android.view.View; import android.widget.ImageView"; then obtains the View to be screenshot by the instruction "View viewToCapture = findViewById(R.id.view_to_capture)"; then creates a Bitmap object according to the instruction "Bitmap bitmap = Bitmap.createBitmap(viewToCapture.getWidth(), viewToCapture.getHeight(), Bitmap.Config.ARGB_8888)" for storing the screenshot image; then creates a Canvas object according to the instruction "Canvas canvas = new Canvas(bitmap)" and uses the Bitmap object as its bitmap; then draws the View to be screenshot onto the Canvas by the instruction viewToCapture.draw(canvas); finally, saves the screenshot to the photo album of the display device 200 according to the instruction "MediaStore.Images.Media.insertImage(getContentResolver(), bitmap, "Screenshot", null); catch (FileNotFoundException e) e.printStackTrace()".

[0113] In some embodiments, the display device 200 may also extract a resume-playing image from the frame buffer of the screen mirroring interface, that is, the resume-playing image is a frame extracted from the frame buffer of the screen mirroring interface according to the time corresponding to the media playback progress. For example, as Figure 11As shown, the screen mirroring interface of the display device 200 is an overlay of a video layer and an On Screen Display (OSD) layer. That is, the image content of the screen mirroring interface can be rendered by the GPU component, so as to form a frame for display in the composition module. Among them, in the composition module, the frame for display can be stored in a frame buffer and a video buffer respectively, and the video process module can form the picture content on the corresponding layers respectively. That is, the image frame corresponding to the screen mirroring picture during the screen mirroring process forms the picture content through the video plane, while the image frames of other image pictures in the screen mirroring interface (such as controls and progress bars of the UI interface, etc.) form the picture content through the OSD plane. Finally, the display device 200 forms the final picture through the blend unit of the video process module for display. Therefore, the display device 200 can extract the frame of the final picture at the moment when the exit instruction is input in the blend unit, or extract the frame in the video buffer corresponding to the video layer as the resumed playback image.

[0114] In some embodiments, the display device 200 can also directly obtain the resumed playback image from the data source of the screen mirroring data. For example, the display device 200 can access the website link corresponding to the media playback address through the multimedia framework service (such as ffmpeg) twice, and encapsulate it as getScaledFrameAtTime(time, option, width, height) at the upper layer to obtain a thumbnail picture of a frame corresponding to the media playback progress.

[0115] According to the method in the above embodiments, after generating the resumed playback information, the display device 200 can store the resumed playback information. During the process of storing the resumed playback information, the display device 200 can construct a specific storage file format to save the resumed playback information. For example, the display device 200 can construct a resumed playback information data table, and in the resumed playback information data table, there can be multiple table entries, that is, the current media playback address, media playback progress, screen mirroring protocol type, media title, speed ratio manually set by the user, audio track, subtitle, zoom mode, etc. Each table entry can set the corresponding specific value according to the content of the recorded resumed playback information.

[0116] The resumed playback information can be stored in a specific storage area in the memory of the display device 200, and the resumed playback information can be set to a state that can be called by the screen mirroring entry application or other applications. After storing the resumed playback information, the display device 200 can execute the exit instruction to exit the screen mirroring state, that is, close the screen mirroring connection channel with the terminal device 500. After closing the screen mirroring connection channel between the display device 200 and the terminal device 500, the display device 200 can switch from the screen mirroring interface to other interfaces.

[0117] Due to the resumed playback information, the user can perform a resumed playback operation on the screen mirroring screen later. And the resumed playback operation has a relatively high probability of occurring in the usage scenario where the user exits the screen mirroring and then re - establishes the screen mirroring connection within a short period of time. Therefore, in some embodiments, the resumed playback information can also be stored according to a set duration, that is, as Figure 12 shown, after storing the resumed playback information, the display device 200 can obtain the storage duration of the resumed playback information and a preset storage threshold (S1201); if the storage duration is greater than or equal to the preset storage threshold, delete the resumed playback information (S1202).

[0118] For example, the preset storage threshold can be set to 24h according to the average value of the interval duration for the display device 200 to re - establish the screen mirroring connection relationship after exiting the screen mirroring. Then, after storing the resumed playback information, the display device 200 can record the storage duration of the resumed playback information. When the storage duration exceeds 24h, the display device 200 can delete the resumed playback information. By setting the preset storage threshold, not only can the situation where the user can still see this entry on the screen mirroring startup interface for a relatively long time be reduced, but also the occupancy of the storage space by the resumed playback information can be reduced, saving the storage space. In addition, by actively clearing the resumed playback information regularly, the display device 200 can also reduce the matching amount of the resumed playback information during subsequent resumed playback, facilitating quick completion of the resumed playback.

[0119] If the storage duration is less than the preset storage threshold, the display device 200 can maintain the storage state of the resumed playback information. Obviously, when the display device 200 has completed the resumed playback, the previously stored resumed playback information has no meaning of continuing to be stored. That is, the display device 200 can also monitor the startup event of the screen mirroring corresponding to the resumed playback information (S1203), and after monitoring the startup event, delete the resumed playback information.

[0120] For example, the display device 200 can monitor the startup event of the screen mirroring during the existence period of the resumed playback information. When it is monitored that the current screen mirroring has been resumed according to the resumed playback information, the display device 200 can clear the resumed playback information so that the playback progress after the resumed playback can also be recorded and stored, achieving the effect of continuous resumed playback.

[0121] In some embodiments, the display device 200 can also monitor the screen mirroring connection operations of other terminal devices. After monitoring that the display device 200 establishes a screen mirroring connection with other terminal devices, the display device 200 clears the resume information of the corresponding screen mirroring image of the original terminal device. For example, after the display device 200 stores the resume information A of the screen mirroring image corresponding to terminal A, during the existence period of the resume information A, when the display device 200 monitors the screen mirroring requests of other terminal devices, and after monitoring that the display device 200 establishes a screen mirroring connection with terminal B, the display device 200 can clear the resume information to prevent the original screen mirroring image from still being able to resume playback when the new screen mirroring image is played, which affects the user experience.

[0122] As can be seen from the technical solutions provided by the above embodiments, in the steps of S100 - S300 above, it is shown that the display device 200 can record information such as the screen mirroring protocol type, media resource playback address, and media resource playback progress during the screen mirroring process when exiting the screen mirroring interface. The recorded information is saved in the form of resume information, so that the display device 200 can resume the playback of the screen mirroring image based on the resume information, as Figure 13 shown.

[0123] S400: In response to a playback instruction for playing the screen mirroring image, obtain the resume information of the screen mirroring image.

[0124] After the display device 200 stores the resume information in the manner provided in the above embodiments, when playing the screen mirroring image subsequently, the display device 200 can call the resume information and perform playback control based on the resume information. Among them, the resume information is the information recorded when exiting the playback of the screen mirroring image, and the resume information includes the screen mirroring protocol type, media resource playback address, and media resource playback progress.

[0125] That is, the display device 200 can receive a playback instruction for playing the screen mirroring image. The playback instruction can also be actively input by the user, or generated by the display device 200 through judging the running state. For the actively input playback instruction, the user can perform instruction input based on the interactive operation methods supported by the display device 200, such as the terminal device 500, remote control, touch, voice, etc. For example, the user can click the screen mirroring icon button in the upper right corner of the playback interface on the terminal device 500 to control the terminal device 500 to send a screen mirroring request to the display device 200, that is, input a playback instruction to the display device 200.

[0126] For the play instruction generated by the display device 200, the display device 200 can determine whether the current state meets the set play conditions by listening to the current running state, and generate a play instruction when the play conditions are met. For example, after the display device 200 exits the screen mirroring interface, it can record the exit time when exiting the screen mirroring interface to determine the duration experienced after the display device 200 exits the screen mirroring interface. If the screen mirroring interface is returned within the preset return time, the display device 200 can automatically generate a play instruction.

[0127] It should be noted that the play instruction generated by the display device 200 should be conducive to realizing the continuous playback of the screen mirroring and will not affect the normal screen mirroring operation of the user. For example, when returning to the screen mirroring interface in a short time, the display device 200 should detect the screen mirroring connection relationship after returning to the screen mirroring interface. If the terminal device 500 that establishes the screen mirroring connection relationship with the display device 200 after returning to the screen mirroring interface is still the terminal device when exiting the screen mirroring interface, the display device 200 can automatically generate a play instruction. If the terminal device 500 that establishes the screen mirroring connection relationship with the display device 200 after returning to the screen mirroring interface is not the terminal device when exiting the screen mirroring interface, the display device 200 can detect the URL address of the current screen mirroring to determine whether it is the original screen mirroring. When the URL address at the time of return is the same as the URL address before exit, a play instruction is automatically generated.

[0128] To facilitate the user to perform interactive operations, the display device 200 can also provide a screen mirroring interactive interface. That is, in some embodiments, the display device 200 can call the screen mirroring entry application and obtain the continuous playback information of the screen mirroring through the screen mirroring entry application. Then control the screen mirroring entry application to generate a screen mirroring start interface according to the continuous playback information. Among them, at least one screen mirroring control is included in the screen mirroring start interface, and the display content of the screen mirroring control is generated according to the continuous playback information. After the screen mirroring entry application generates the screen mirroring start interface, the display device 200 can control the display to display the screen mirroring start interface, and listen to the play instruction input by the user based on the screen mirroring control, and in response to the play instruction, read the screen mirroring protocol type, media resource play address, and media resource play progress from the continuous playback information.

[0129] For example, after the display device 200 receives the screen mirroring request sent by the terminal device 500, it obtains the continuous playback information of the current screen mirroring. If the continuous playback information of the current screen mirroring is stored in the display device 200, the screen mirroring entry application can be called to control the display device 200 to display the screen mirroring start interface through the screen mirroring entry application.

[0130] In the screen projection start interface, a screen projection option corresponding to the current screen projection may be included, and the screen projection option may be associated with a resume control for the user to select. When the user clicks the resume control, the display device 200 may play the screen projection according to the resume information. At the same time, the display device 200 may generate a play instruction, and in response to the play instruction, read the screen projection protocol type, media asset playback address, and media asset playback progress recorded when exiting the screen projection interface from the resume information.

[0131] For example, when the user launches the screen casting entry application through the remote control, if there is thumbnail information in ContentProvider and video title in Setting.Global, the action to start the resume playback is monitored, that is, whether the user clicks the resume playback control in the screen casting start interface. If there is no data, the screen casting entry interface is displayed, and after the user clicks the screen casting control, it jumps to the screen casting guide page to re-establish the screen casting connection relationship.

[0132] S500: Acquire screen projection data from the media resource playback address.

[0133] After obtaining the resume information in response to the play instruction, the display device 200 can parse the resume information, thereby reading data such as the projection protocol type, the media asset playback address, and the media asset playback progress in the resume information. The projection screen is then resumed based on the parsed data, that is, the display device 200 can obtain the projection data from the media asset playback address and form a projection data stream for the player to play.

[0134] When the display device 200 and the terminal device 500 establish a screen projection connection, the user needs to perform corresponding connection operation steps. For example, the user needs to first open the video playback interface through the terminal device 500, and then click the screen projection button in the video playback interface. And when the display device 200 displays the screen projection start interface, the user also needs to click the resume control in the screen projection start interface to complete the resume operation, and there are many operation steps.

[0135] Therefore, in order to simplify user operations, in some embodiments, the display device 200 can complete the screen projection connection through a virtual session instead of the terminal device 500. Figure 14 As shown, after obtaining the resume broadcast information, the display device 200 can create a virtual session object based on the resume broadcast information, wherein the virtual session object is used to replace the terminal device to establish a virtual screen projection connection with the display device according to the screen projection protocol type.

[0136] In some embodiments, in order to create a virtual session object based on the resume play information, the display device 200 may further obtain the target screen mirroring control affected by the resume play instruction, and query the resume play application associated with the screen mirroring image according to the target screen mirroring control. Wherein, the resume play application is an application for forming the screen mirroring image. Then run the resume play application, and send the resume play information to the resume play application to control the resume play application to create a virtual session object based on the resume play information.

[0137] For example, a fusion screen mirroring service may be configured in the display device 200. When the screen mirroring entry application sends an Action message in response to a user's click operation on the resume play control, the fusion screen mirroring service will monitor this Action message and notify the adaptation layer of the resume play application to create a virtual Session through the protocol type. Wherein, Session is a session control object. Session can be used to store the attributes and configuration information required for a specific user session. When the user jumps between application interfaces, the variables stored in the Session object will not be lost, but will exist throughout the user session. When a user requests from an application interface, if the user does not have a session yet, the display device 200 can automatically create a Session object. When the session expires or is abandoned, the display device 200 will terminate the session.

[0138] The virtual Session object created by the display device 200 can then establish a virtual screen mirroring connection with the display device 200 on behalf of the terminal device 500 according to the screen mirroring protocol type recorded in the resume play information, so that the display device 200 does not need to repeatedly establish a screen mirroring connection relationship with the terminal device 500, but directly uses the media asset play address recorded in the resume play information to obtain the screen mirroring data, that is, obtain the screen mirroring data from the media asset play address through the virtual session object. In addition, the virtual Session object can also be used to obtain the resume play information and use the resume play information as a control variable during the play process for the player of the display device 200. That is, the adaptation layer of the resume play application receives the resume play message, creates a virtual Session, and obtains the saved resume play information.

[0139] S600: Play the screen mirroring data according to the media asset play progress.

[0140] After obtaining the screen mirroring data, the display device 200 can input the screen mirroring data into the player, and the player then performs playback operations such as decoding, rendering, and outputting the picture on the input screen mirroring data to form a screen mirroring picture. At the same time, the display device 200 can set the media playback progress for the player, so that the player plays the screen mirroring data according to the media playback progress. For example, in the resumed playback information obtained by the display device 200, the playback progress of media A, i.e., 0:38:25, can be recorded. Then the display device 200 can start obtaining the data stream from the playback address of media A according to 0:38:25 and input the obtained data stream into the player. Then the player can decode and render the media data after 0:38:25 and output the picture to the display 260, so that the display 260 displays the screen mirroring picture after 0:38:25.

[0141] As can be seen from the above technical solutions, in the above embodiments, when performing screen mirroring in the push mode, by recording the resumed playback information and calling the recorded resumed playback information when playing the screen mirroring picture, the display device 200 can play the screen mirroring data according to the media playback progress in the resumed playback information, realizing the screen mirroring resumed playback effect.

[0142] Since different display devices 200 support different screen mirroring methods, and the functions supported after establishing the screen mirroring connection are also different. For example, when the display device 200 establishes a screen mirroring connection through a video application, different video applications have different support for the resumed playback function. That is, for the system video application of the display device 200 and the video applications developed by third-party application providers that have a cooperation relationship with the operator of the display device 200, the resumed playback information can be recorded and read according to the cooperation specifications. However, for third-party video applications without a cooperation relationship, because they do not have built-in corresponding recording and reading algorithms, they do not support the screen mirroring resumed playback function. Therefore, as Figure 15 shown, in some embodiments, the display device 200 also needs to detect the application type of the resumed playback application and complete the resumed playback function in different ways according to the application type. That is, the display device 200 can detect the application type of the resumed playback application (S1501) by reading the application package name information, etc., where the application type is the first type of application that supports screen mirroring resumed playback or the second type of application that does not support screen mirroring resumed playback.

[0143] For example, by reading the application package name information, description information such as the name and developer of the application can be obtained, and through matching with a preset matching table, the type of the application can be determined. If the developer of Application A is the operator of the display device 200, that is, Application A is a system application of the display device 200, at this time, it can be determined that Application A supports the function of continued casting and playing, that is, the application type of Application A is the first type of application. Similarly, if by reading the application package name information, it is determined that the developer of Application B is not the operator of the display device 200 and there is no cooperation relationship between the developer of Application B and the operator of the display device 200, it can be determined that Application B does not support the function of continued casting and playing, that is, the application type of Application B is the second type of application.

[0144] After detecting the application type, the display device 200 can implement the continued casting and playing function in different ways according to different application types. If the application type is the first type of application, the display device 200 can record the continued casting and playing information according to the continued casting and playing method provided in the above embodiment, and send the continued casting and playing information to the continued casting and playing application (S1502), so that the display device 200 plays the casting data according to the media asset playing progress in the continued casting and playing information.

[0145] If the application type is the second type of application, a casting connection request is sent to the terminal device 500 (S1503), and the casting data fed back by the terminal device 500 according to the casting connection request is received (S1504). That is, when the display device 200 determines that the continued casting and playing application does not support continued casting and playing by recording the continued casting and playing information, it can send a casting connection request to the terminal device 500 again to re - establish the casting connection relationship.

[0146] To implement the continued casting and playing function, as Figure 16 shown, the display device 200 can also listen for the switching event of the casting interface (S1601) when determining that the application type of the continued casting and playing application is the second type of application. The switching event refers to an event that triggers the display device 200 to switch from the display casting interface to display other interfaces, which can be triggered by an exit instruction or other conditional judgment methods.

[0147] After detecting a switching event, the display device 200 may generate a pause play instruction in response to the switching event (S1602). The pause play instruction is used to pause the play of the projection data corresponding to the projection screen. That is, the display device 200 may control the projection screen to pause through the pause play instruction and switch the projection interface to run in the background (S1603). For different projection methods, the display device 200 implements pause play in different ways. When establishing a projection connection through the push method, the playback process is executed by the player of the display device 200. Then, after generating the pause play instruction, the display device 200 may directly respond to the pause play instruction and pause the playback process of the projection data. When establishing a projection connection through the mirroring method, since the playback process is not executed by the player of the display device 200 but by the terminal device 500, after generating the pause play instruction, a pause play instruction may be sent to the terminal device 500 so that the terminal device 500 responds to the pause play instruction and pauses the playback process of the projection data.

[0148] After switching the projection interface to run in the background, the display device 200 may detect a return event of the projection interface within a preset time (S1604). When detecting the return event, the display device 200 then generates a resume play instruction in response to the return event (S1605), that is, resumes playing the projection data corresponding to the projection screen and controls the display 260 to display the projection interface (S1606).

[0149] It can be seen that in the above embodiment, the display device 200 can adopt different methods to implement projection resume according to the application type of the resume play application, which can make the display device 200 adapt to more types of resume play applications to solve the problem that the projection playback process cannot be resumed.

[0150] Based on the above display device 200, some embodiments of the present application also provide a projection resume method, and the projection resume method includes:

[0151] In response to a play instruction for playing a projection screen, obtain resume information of the projection screen. The resume information is information recorded when exiting the play of the projection screen, and the resume information includes a projection protocol type, a media resource play address, and a media resource play progress;

[0152] Create a virtual session object based on the resume information. The virtual session object is used to establish a virtual projection connection with the display device on behalf of the terminal device according to the projection protocol type;

[0153] Obtain projection data from the media resource play address through the virtual session object;

[0154] Play the projection data according to the media resource play progress.

[0155] As can be seen from the above technical solutions, some embodiments of the present application provide a display device and a screen mirroring continuous playback method. The method can obtain continuous playback information of the screen mirroring in response to a playback instruction of the screen mirroring. Among them, the continuous playback information is the information recorded when exiting the playback of the screen mirroring, and the continuous playback information includes the screen mirroring protocol type, the media playback address, and the media playback progress. Then, a virtual session object is created based on the screen mirroring protocol type in the continuous playback information, and the screen mirroring data is obtained from the media playback address through the virtual session object, and the screen mirroring data is played according to the media playback progress. The method can record the continuous playback information when the display device exits the screen mirroring interface, and support the display device to obtain the previously saved continuous playback information when returning to the screen mirroring interface, and jump to the corresponding media playback progress according to the continuous playback information, so as to realize the continuous playback of the screen mirroring, and solve the problem that the screen mirroring cannot be continuously played during the playback process.

[0156] The same or similar parts among the various embodiments in this specification can be referred to each other, and will not be repeated here.

[0157] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods in the various embodiments or some parts of the embodiments of the present invention.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the various embodiments of the present application.

[0159] For the sake of convenience of explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. According to the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are for better explaining the principles and actual applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, Including: A display configured to display a screen mirroring interface; A communicator configured to establish a screen mirroring connection with a terminal device; A controller configured to: In response to a play instruction for playing a screen mirroring video, obtain resume information of the screen mirroring video, where the resume information is information recorded when exiting the play of the screen mirroring video, and the resume information includes a screen mirroring protocol type, a media resource play address, and a media resource play progress; Create a virtual session object based on the resume information, where the virtual session object is used to establish a virtual screen mirroring connection with a display device on behalf of the terminal device according to the screen mirroring protocol type; Obtain screen mirroring data from the media resource play address through the virtual session object; Play the screen mirroring data according to the media resource play progress.

2. The display device according to claim 1, characterized in that, The controller is further configured to: Obtain an exit instruction for exiting the screen mirroring interface; In response to the exit instruction, record the screen mirroring protocol type and media resource play address of the screen mirroring interface, and record the media resource play progress at the moment when the exit instruction is input; Generate the resume information according to the screen mirroring protocol type, the media resource play address, and the media resource play progress; Store the resume information, and close the screen mirroring connection channel with the terminal device.

3. The display device according to claim 2, characterized in that, Before the controller executes storing the resume information, it is further configured to: Generate a resume image according to the media resource play progress, where the resume image is a frame extracted from the data stream of the screen mirroring video according to the media resource play progress; or, the resume image is an image obtained by taking a screenshot of the screen mirroring video by calling a screenshot interface; or, the resume image is a frame extracted from the frame buffer of the screen mirroring interface according to the time corresponding to the media resource play progress; Perform compression processing on the screen mirroring image to generate a thumbnail; Add the thumbnail to the resume information.

4. The display device according to claim 3, characterized in that, After the controller executes storing the resume information, it is further configured to: Obtain the storage duration of the resume information and a preset storage threshold; If the storage duration is greater than or equal to the preset storage threshold, delete the resume information; If the storage duration is less than the preset storage threshold, monitor a start event of the screen mirroring video corresponding to the resume information, and delete the resume information after monitoring the start event.

5. The display device according to claim 2, characterized in that, After the controller executes recording the media resource play address and media resource play progress at the moment when the exit instruction is input, it is further configured to: Obtain the total duration of the media resource of the screen mirroring video; Calculate the remaining duration of the media resource according to the total duration of the media resource and the media resource play progress; If the remaining duration of the media resource is greater than or equal to a preset duration threshold, execute the step of generating the resume information according to the screen mirroring protocol type, the media resource play address, and the media resource play progress; If the remaining duration of the media resource is less than the preset duration threshold, execute the step of closing the screen mirroring connection channel with the terminal device.

6. The display device according to claim 1, characterized in that, When the controller executes in response to a play instruction for playing a screen mirroring video and obtains the resume information of the screen mirroring video, it is further configured to: Call a screen mirroring entry application, where the screen mirroring entry application is an application that provides a screen mirroring start interface; Obtain the resume information of the screen mirroring video through the screen mirroring entry application; Control the screen mirroring entry application to generate a screen mirroring start interface according to the resume information, where at least one screen mirroring control is included in the screen mirroring start interface, and the display content of the screen mirroring control is generated according to the resume information; Control the display to display the screen mirroring start interface, and monitor the play command input by the user based on the screen mirroring control; In response to the play command, read the screen mirroring protocol type, media resource play address, and media resource play progress from the resume information.

7. The display device according to claim 6, characterized in that, The controller executes to create a virtual session object based on the resume information, and is further configured to: Obtain the target screen mirroring control affected by the resume command; Query the resume application associated with the screen mirroring image according to the target screen mirroring control, where the resume application is an application used to form the screen mirroring image; Run the resume application, and send the resume information to the resume application; Control the resume application to create a virtual session object based on the resume information.

8. The display device according to claim 7, characterized in that, The controller executes to send the resume information to the resume application, and is further configured to: Detect the application type of the resume application, where the application type is the first type of application that supports screen mirroring resume or the second type of application that does not support screen mirroring resume; If the application type is the first type of application, execute the step of sending the resume information to the resume application; If the application type is the second type of application, send a screen mirroring connection request to the terminal device, and receive the screen mirroring data fed back by the terminal device according to the screen mirroring connection request.

9. The display device according to claim 8, characterized in that, If the application type is the second type of application, the controller is further configured to: Monitor the switching event of the screen mirroring interface; In response to the switching event, generate a pause play command, where the pause play command is used to pause the play of the screen mirroring data corresponding to the screen mirroring image; Switch the screen mirroring interface to run in the background; Monitor the return event of the screen mirroring interface within a preset time; In response to the return event, generate a resume play command, and control the display to display the screen mirroring interface, where the resume play command is used to resume the play of the screen mirroring data corresponding to the screen mirroring image.

10. A screen mirroring and continuous playback method, characterized in that, Applied to the display device according to any one of claims 1-9, the screen mirroring resume method includes: In response to a play command for playing a screen mirroring image, obtain the resume information of the screen mirroring image, where the resume information is the information recorded when exiting the play of the screen mirroring image, and the resume information includes the screen mirroring protocol type, media resource play address, and media resource play progress; Create a virtual session object based on the resume information, where the virtual session object is used to establish a virtual screen mirroring connection with the display device on behalf of the terminal device according to the screen mirroring protocol type; Obtain the screen mirroring data from the media resource play address through the virtual session object; Play the screen mirroring data according to the media resource play progress.

Images