Display device and screen mirroring playback resuming method

Abstract

The present disclosure provides a display device and a screen mirroring playback resuming method. The method comprises: in response to a playback instruction used for playing a screen mirroring picture, obtaining resume playback information of the screen mirroring picture, wherein the resume playback information can comprise a screen mirroring protocol type, a media asset playing address, a media asset playback progress, and playback effect information set by a user; creating a virtual session object on the basis of the resume playback information, the virtual session object being used for replacing a terminal device to establish a virtual screen mirroring connection with a display device according to the screen mirroring protocol type; obtaining screen mirroring data from the media asset playing address by means of the virtual session object; and on the basis of the media asset playback progress, determining a playback starting moment for playing the screen mirroring data, and on the basis of the playback effect information, playing the screen mirroring data from the playback starting moment. The resume playback information is recorded when the display device exits a screen mirroring interface, the stored resume playback information is obtained when the display device returns to the screen mirroring interface, and then the playback jumps to a corresponding media asset playback progress, thereby resuming a playback of the screen mirroring picture; the problem of being unable to resume a playback in a screen mirroring playback process is solved.

Description

Display device and screen projection and continuous broadcasting method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese application No. 202311734403.4, filed on December 15, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the technical field of display devices, and in particular to a display device and a screen projection and continuous broadcasting method. Background Art

[0004] Display devices are intelligent devices that can present user interfaces and support user interaction. For example, smart TVs are based on Internet application technologies, feature open operating systems and chips, and boast an open application platform. They enable two-way human-computer interaction and integrate multiple functions, including audio, video, entertainment, and data, to meet diverse and personalized user needs. Display devices can establish communication connections with mobile phones and other terminal devices. Based on specific projection protocols, they can obtain projection data from terminal devices and display the projection image on the display device.

[0005] Since the data source of the projected screen is provided or pushed by the terminal device when the display device is displaying the projected screen, and the display device and the terminal device are two independent entities, the playback control cannot be performed according to the conventional video playback method. For example, when the display device is displaying the projected screen through the projected screen playback interface, the user controls the display device to jump to another interface by pressing the return button on the remote control. In this case, the display device will disconnect the projected screen connection during the jump and will no longer receive the projected screen data transmitted by the terminal device. Therefore, when the user controls the display device to return to the state of displaying the projected screen, the display device needs to push the projected screen again through the terminal device, and the user needs to manually fast forward to the previous playback progress in order to complete the continued playback of the projected screen, which reduces the user experience.

[0006] Summary of the Invention

[0007] In a first aspect, an embodiment of the present disclosure provides a display device, comprising: a display configured to display a projection interface; a user input interface configured to receive instructions from a user; a communication device configured to establish a projection connection with a terminal device; a memory configured to store computer instructions and data associated with the display device; at least one processor connected to the display, the user input interface, the communication device and the memory, and configured to execute computer instructions so that the display device performs: in response to a playback instruction for playing a projection screen, obtaining resume information of the projection screen, the resume information being information recorded when exiting playback of the projection screen, the resume information including a projection protocol type, a media playback address, a media playback progress, and playback effect information set by a user; creating a virtual session object based on the resume information, the virtual session object being used to replace the terminal device in establishing a virtual projection connection with the display device according to the projection protocol type; obtaining projection data from the media playback address through the virtual session object; playing the start time of the projection data according to the media playback progress, and playing the projection data from the start time based on the playback effect information.

[0008] In the second aspect, the embodiment of the present disclosure also provides a screen projection continuation method, including: in response to a playback instruction for playing a projection screen, obtaining the continuation information of the projection screen, the continuation information is the information recorded when exiting the playback of the projection screen, and the continuation information includes the projection protocol type, the media playback address, the media playback progress, and the playback effect information set by the user; creating a virtual session object based on the continuation information, the virtual session object is used to replace the terminal device to establish a virtual projection connection with the display device according to the projection protocol type; obtaining projection data from the media playback address through the virtual session object; playing the start time of the projection data according to the media playback progress, and playing the projection data from the start time based on the playback effect information. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG1 is a schematic diagram of an operation scenario between a display device and a control device according to some embodiments;

[0010] FIG2 is a schematic diagram of a hardware configuration of a display device according to some embodiments;

[0011] FIG3 is a schematic diagram of a hardware configuration of a control device according to some embodiments;

[0012] FIG4 is a schematic diagram of software configuration of a display device according to some embodiments;

[0013] FIG5 is a schematic diagram of mirror screen projection according to some embodiments;

[0014] FIG6 is a schematic diagram of push screen projection according to some embodiments;

[0015] FIG7 is a schematic diagram of a process for recording and resuming broadcast information according to some embodiments;

[0016] FIG8 is a schematic diagram of a process for generating broadcast continuation information based on the remaining duration according to some embodiments;

[0017] FIG9 is a schematic diagram of a screen projection start interface according to some embodiments;

[0018] FIG10 is a schematic diagram of a process for recording the resume broadcast information at the time of inputting the exit instruction according to some embodiments;

[0019] FIG11 is a schematic diagram of a playback function module according to some embodiments;

[0020] FIG12 is a schematic diagram of a process for deleting resume broadcast information according to some embodiments;

[0021] FIG13 is a schematic diagram of a screen projection and continuous broadcasting process according to some embodiments;

[0022] FIG14 is a schematic diagram of a process for continuing screen projection through a virtual conversation object according to some embodiments;

[0023] FIG15 is a schematic diagram of a process for executing screen casting and continuous broadcasting according to application type according to some embodiments;

[0024] FIG16 is a schematic diagram of the process of continuous screen projection of the second type of application according to some embodiments. DETAILED DESCRIPTION

[0025] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device according to some embodiments. As shown in Figure 1, a user can operate a display device 200 through touch operation, a mobile terminal 300, and a control device 100. The control device 100 may be a remote control, a stylus, or the like.

[0026] In some embodiments, the mobile terminal 300 can install software applications on the display device 200 and connect to the display device 200 through a network communication protocol to achieve one-to-one control operations and data communication. It is also possible to transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronous display.

[0027] 1 , the display device 200 also communicates data with the server 400 via various communication methods. The display device 200 may be connected to the server 400 via a local area network (LAN), a wireless local area network (WLAN), or other networks.

[0028] In addition to providing broadcast reception television functions, the display device 200 may also provide intelligent network television functions with computer support functions, including but not limited to network television, smart TV, Internet Protocol television (IPTV), etc.

[0029] FIG2 is a block diagram of a hardware configuration of a display device according to some embodiments. As shown in FIG2 , the display device 200 may include at least one of a tuner / demodulator 210, a communication device 220, a detector 230, a device interface 240, at least one processor 250, a display 260, an audio output interface 270, a memory, a power supply, and a user input interface.

[0030] In some embodiments, detector 230 can be used to collect signals from the external environment or external interactions. For example, detector 230 may include a light receiver, such as a sensor for collecting ambient light intensity; or detector 230 may include an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 may include a sound collector, such as a microphone, for receiving external sounds.

[0031] In some embodiments, the display 260 may include a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface, as well as a user control UI interface, etc.

[0032] In some embodiments, the communication device 220 is a component used to communicate with an external device or server 400 according to various communication protocol types. The display device 200 can be provided with multiple communication devices 220 depending on the supported communication methods. For example, if the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 including WiFi functionality. If the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including Bluetooth functionality.

[0033] The communication device 220 can establish a communication connection between the display device 200 and an external device or server 400 via a wireless or wired connection. A wired connection can connect the display device 200 to an external device via a data cable, an interface, or other components. A wireless connection can connect the display device 200 to an external device via a wireless signal or wireless network. The display device 200 can establish a connection with an external device directly or indirectly through a gateway, router, or connection device.

[0034] In some embodiments, the display device 200 may also support establishing communication connections with multiple external devices at the same time. Multiple external devices may be connected through the same. For example, the external device is a terminal device 500 such as a mobile phone or a tablet computer. The first terminal device 510 and the second terminal device 520 may establish a communication connection with the display device 200 by accessing the wireless local area network where the display device 200 is located. Multiple external devices may also be connected to the display device 200 through different types of connection methods. For example, the first terminal device 510 is connected to the display device 200 via a wireless local area network; the second terminal device 520 is connected to the display device 200 via Bluetooth.

[0035] To achieve communication between the display device 200 and the terminal device 500, the display device 200 and the terminal device 500 are each equipped with a communication device that supports the same type of communication. For example, if the display device 200 is equipped with a communication device 220 including a WiFi module, the terminal device 500 should also be equipped with a communication device including a WiFi module. Furthermore, while the same type of communication device is provided, a specific communication transmission protocol is required for data transmission between the display device 200 and the terminal device 500. Examples include the WiFi protocol, Bluetooth connection protocol, ZigBee protocol, and NFC protocol.

[0036] In some embodiments, the processor 250 may include a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output. The processor 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The processor 250 controls the overall operation of the display device 200.

[0037] In some embodiments, the processor 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the processor 250 is located, such as an external set-top box.

[0038] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0039] In some embodiments, the user interface 280 is an interface that can be used to receive control input.

[0040] Figure 3 is a block diagram of a hardware configuration of a control device according to some embodiments. As shown in Figure 3, the control device 100 may include a processor 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0041] The control device 100 can be configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, thereby acting as an interactive intermediary between the user and the display device 200 .

[0042] In some embodiments, the control device 100 may be a smart device. For example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0043] In some embodiments, as shown in FIG. 1 , the mobile terminal 300 or other intelligent electronic device can perform similar functions as the control device 100 after installing an application for controlling the display device 200 .

[0044] The processor 110 may include a RAM 113 and a ROM 114, a communication interface 130, and a communication bus. The processor 110 may be used to control the operation and operation of the control device 100, as well as communication and coordination between internal components and external and internal data processing functions.

[0045] Under the control of the processor 110, the communication interface 130 can communicate control signals and data signals with the display device 200. The communication interface 130 can include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, or other near field communication modules.

[0046] The user input / output interface 140 includes at least one of a microphone 141 , a touch panel 142 , a sensor 143 , a button 144 and other input interfaces.

[0047] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, to encode user input commands via the WiFi protocol, Bluetooth protocol, or NFC protocol and transmit them to the display device 200.

[0048] The memory 190 can be used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the processor. The memory 190 can also store various control signal instructions input by the user.

[0049] The power supply 180 can be used to provide operating power support for various components of the control device 100 under the control of the processor.

[0050] As shown in Figure 4, in some embodiments, the system is divided into four layers, from top to bottom: the application layer (referred to as the "application layer"), the application framework layer (referred to as the "framework layer"), the Android runtime and system library layer (referred to as the "system runtime library layer"), and the kernel layer.

[0051] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0052] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0053] As shown in FIG4 , the application framework layer in the embodiment of the present disclosure includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to the system location service; a package manager for retrieving various information related to the application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0054] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling application exit, opening, and back. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes, such as shrinking, shaking, or distorting the display window.

[0055] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0056] In some embodiments, the kernel layer is a layer between hardware and software. As shown in FIG4 , the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a Wi-Fi driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power driver.

[0057] After establishing a communication connection with the terminal device 500, the display device 200 can establish a screen projection connection channel and transmit projection data based on the projection connection so that the display device 200 can display the projection image of the terminal device 500. In some embodiments, to establish a screen projection connection relationship, the user can issue a screen projection connection request through the terminal device 500 and send the screen projection connection request to the display device 200 via the WiFi network. The display device 200 then completes the transmission protocol configuration based on the screen projection connection request, thereby establishing a transmission channel for projection data between the display device 200 and the terminal device 500.

[0058] The display device 200 can also achieve a screen projection connection with the terminal device 500 through different connection methods. For example, when the display device 200 and the terminal device 500 are connected to the same wireless local area network, a screen projection connection can be established based on the WiFi network. For another example, when both the display device 200 and the terminal device 500 are provided with a Near Field Communication (NFC) component, a screen projection connection relationship can be established through the NFC component. Obviously, other wired or wireless connection methods can also be used between the display device 200 and the terminal device 500 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network and other connection methods that those skilled in the art can associate with the connection methods provided in the above embodiments.

[0059] After establishing the screen projection connection, the display device 200 can receive the screen projection related data from the terminal device 500 through the screen projection data channel, thereby playing the screen projection related content on the display device 200. Under different screen projection data transmission protocols, the screen projection data obtained by the display device 200 is in different forms. For example, the display device 200 and the terminal device 500 can establish a screen projection connection through the Digital Living Network Alliance (DLNA) protocol, Miracast protocol, AirPlay push protocol, WiDi screen projection, NFC mirror screen projection, LeTV screen projection, and other protocols.

[0060] The DLNA protocol is a protocol used to share media resources between devices. This protocol uses network technology (wired or wireless) to interconnect various devices and exchange data through standard protocols, allowing media resources to be shared between devices. The DLNA protocol is based on the UPnP protocol, enabling devices to discover and control each other, achieving seamless transmission and sharing of media resources. The DLNA protocol supports a variety of devices, including TVs, speakers, mobile phones, computers, and more, allowing users to seamlessly switch and share media content between different devices.

[0061] The Miracast protocol is a wireless display standard developed by the Wi-Fi Alliance and based on Wi-Fi Direct. The Miracast screen projection protocol is compatible with Android devices, allowing Android devices to share video images through Miracast. Smart devices running Android 5.0 and higher have the Miracast protocol built-in, allowing direct wireless screen projection without installing any software, using the Wi-Fi P2P function between the projection device and the receiving device's Wi-Fi network card. Miracast also supports simultaneous audio and video transmission.

[0062] AirPlay is a wireless playback technology introduced by Apple, suitable for wireless screen projection on iOS and Mac systems. Apple devices have built-in AirPlay projection functionality. When used with a screen projector that supports the AirPlay server service, the display is clear and stable. The protocol also includes a built-in projection code function. However, the Apple device and screen projector must be on the same local area network (LAN segment) and cannot be used across different network segments or VLANs.

[0063] WiDi is an Intel-developed wireless streaming technology based on Wi-Fi Direct. Windows 8 / 10 / 11 laptops natively support WiDi, allowing you to cast your screen directly without installing any software. It also supports USB reverse control.

[0064] In some embodiments, after the display device 200 establishes a screen projection connection with the terminal device 500, the projection data can be transmitted through mirror mode and push mode. Among them, the mirror mode means that the display device 200 synchronously displays the screen content displayed by the terminal device 500. For example, as shown in Figure 5, the display device 200 displays the main interface of the terminal device 500, and when the user performs an interactive operation on the terminal device 500, causing the content of the main interface to change, the content displayed by the display device 200 also changes. To this end, the terminal device 500 can generate a projection data stream in the form of a video based on the display content, and send the projection data stream to the display device 200 for display.

[0065] Push mode means that the display device 200 accesses the network link pushed by the terminal device 500 and obtains the corresponding projection image. For example, as shown in Figure 6, when the terminal device 500 is playing video media, a projection control is provided in the upper right corner of the playback interface. After the user clicks the projection control and selects the display device 200 as the device to display the projection image, the terminal device 500 can send the URL address of the currently playing video media to the display device 200. After receiving the URL address, the display device 200 accesses the URL address to obtain the video media data and plays the video media.

[0066] Among the screen projection protocols supported by the display device 200 and the terminal device 500, some protocols are based on the mirror mode, and some protocols are based on the push mode. For example, under the DLNA protocol, the display device 200 can obtain the link address of a resource in the terminal device 500 through the screen projection data channel, and obtain the media content data by accessing the link address, thereby playing the media data. At the same time, the display device 200 can also obtain the control instructions sent by the terminal device 500 through the screen projection data channel when playing the media data, such as fast forward, pause, stop screen projection, etc. For example, after establishing a connection through the communication protocol, the terminal device 500 such as a mobile phone and the display device 200 will negotiate the device functions and network conditions to select a suitable audio and video transmission format. After establishing the online protocol (Session) of the audio and video streaming, a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the audio and video streaming.

[0067] Under other screen projection connection protocols such as Miracast, Airplay, NFC mirroring, and LeTV, the display device 200 can obtain the video data stream directly from the terminal device 500. The video data stream can be the video data stream corresponding to the final display screen of the display unit of the terminal device 500, that is, the video data stream includes the media content screen, UI interface, and other video screen content that can be displayed in the display unit of the terminal device 500. The video data stream can also be only the video data that can be obtained in the terminal device 500, such as the media video data being played. At this time, the terminal device 500 can serve as a signal source for the display device 200, and the display device 200 can continuously obtain video data through the screen projection data channel, and display it in real time on the display 260, so as to achieve the effect of synchronous display with the terminal device 500.

[0068] It should be noted that the above-mentioned screen projection protocol can not only obtain the video data stream in the terminal device 500, but also obtain the audio data stream in the terminal device 500, so that the video data and audio data can be sent to the display device 200 synchronously, so that the display device 200 can play the video and audio synchronously.

[0069] The terminal device 500 can obtain different projection data according to different projection protocols. For example, for system applications or direct projection at the system level, the terminal device 500 can use Miracast projection, Airplay projection or WiDi projection protocol to establish a projection data channel. Since the above-mentioned Miracast projection protocol and Airplay projection protocol require system-level application permissions, the Miracast projection protocol and Airplay projection protocol are only applicable to system applications, or the display device 200 and the terminal device 500 of the same manufacturer directly establish a projection connection through the built-in rules of the operating system.

[0070] After the display device 200 establishes a projection connection with the terminal device 500 based on any one of the above-mentioned projection protocols, a projection screen can be formed according to the projection data sent by the terminal device 500. The projection screen can be displayed through a specific user interface. In the embodiment of the present disclosure, the user interface for displaying the projection screen can be referred to as a projection interface. It should be understood that the projection interface can be an independent interface. For example, the projection interface is a floating projection window. The projection interface can also be the playback interface of the display device 200. For example, after the projection connection is established, when the display device 200 displays the projection content in full screen, the entire full-screen displayed screen content is the projection interface.

[0071] Due to the unique data source of the projected screen, namely, the display device 200 receives the projected screen data from the terminal device 500 and then plays it to form the projected screen. Therefore, the display device 200's control over the playback of the projected screen data is restricted by the projected screen protocol, the terminal device 500, and other conditions. This puts the display device 200 in a passive playback mode, making it difficult to complete playback recording, adjustment, switching, and other control processes according to the media playback process.

[0072] For example, in the mirror projection mode, the display device 200 can only display the mirror projection data stream sent by the terminal device 500 in real time, and the control of the projection process is limited to basic operations such as exit. In the push projection mode, the display device 200 obtains the projection data by accessing the media playback address pushed by the terminal device 500. However, since the projection data is pushed by the terminal device 500, the data source is still the terminal device 500 by default. Therefore, after exiting the projection interface, the display device 200 will directly disconnect the projection connection relationship and will not record any playback data. If the user wants to continue to play the projection screen, it is necessary to re-establish the projection connection relationship between the display device 200 and the terminal device 500, and manually adjust the playback progress to the time of exit through the fast-forward operation to continue the projection screen. It can be seen that this method of continuing to play the screen is cumbersome to operate, and the waiting time for continuing to play is long, which will reduce the user experience.

[0073] In order to solve the problem that the screen projection process cannot be resumed, the embodiment of the present disclosure provides a display device 200. The display device 200 can achieve the purpose of automatically resuming the projection screen by recording the resume information when exiting the projection screen, and when the user chooses to resume the projection screen, automatically jumping to the playback progress at the time of exit according to the resume information. In order to meet the implementation of the above functions, the display device 200 may include a display, a user input interface, a communication device, a memory and at least one processor. Among them, the display 260 is configured to display the projection interface, the user input interface is configured to receive instructions from the user, the communication device is configured to establish a projection connection with the terminal device, the memory is configured to store computer instructions and data associated with the display device, at least one processor is connected to the display, the user input interface, the communication device and the memory, and is configured to execute computer instructions so that the display device executes the steps corresponding to the projection and resume method, as shown in Figure 7, including the following steps:

[0074] S100: Obtain an exit instruction for exiting the screen projection interface.

[0075] In the embodiments of the present disclosure, the exit instructions for exiting the screen projection interface may include two types. One type is an active exit instruction, that is, an exit instruction in which the user actively exits the screen projection interface through a specific interactive operation. For example, while the display device 200 is displaying the screen projection interface, the user presses the return key on the remote control of the display device 200 to control the display device 200 to exit the screen projection interface.

[0076] In some embodiments, the user can input an exit command based on different interactive operations. For example, for a display device 200 that supports touch interactive operations, when the display device 200 displays the screen projection interface, the user can control the display device 200 to exit the screen projection interface by sliding upward from the bottom edge of the display 260. That is, the exit command can be input through touch interaction. For another example, for a display device 200 that supports voice interactive operations, when the display device 200 displays the screen projection interface, the user can input a voice message such as "exit screen projection" to control the display device 200 to exit the screen projection interface. That is, the exit command can be input through voice interaction.

[0077] The other type is a passive exit command, which is a passive exit command generated when the display device 200 switches the displayed content from the projection interface to another interface based on the operating status or user interaction. For example, when the display device 200 is displaying the projection interface, the user plugs in a gaming device through the HDMI port. At this time, the display device 200 automatically jumps to the signal source setting interface, at which point the display device 200 can generate an exit command based on the plug-in signal.

[0078] According to the interface switching rules set by the operating system of the display device 200, a passive exit instruction can be generated in different operating states, and the exit instruction can also be input or generated by the control device 100 or an external device. For example, when the user controls the display device 200 to switch from the projection interface to the home interface by pressing the home button on the remote control, the display device 200 can generate and execute the exit instruction. For example, for a display device 200 that supports video (or voice) calls, if a video call request is received when the display device 200 displays the projection interface, the display device 200 can also switch from the projection interface to the video call interface.

[0079] S200: In response to the exit instruction, record the projection protocol type and media playback address of the projection interface, and record the media playback progress and playback effect information at the time when the exit instruction is input.

[0080] In order to obtain the screen projection protocol type, the display device 200 can read the configuration file of the screen projection interface. The configuration file can record the screen projection protocol used by the display device 200 when establishing a screen projection connection with the terminal device 500. For example, when the user controls the display device 200 to establish a screen projection connection with the terminal device 500 through the system default DLNA protocol, the display device 200 will first create a configuration file when rendering the screen projection interface and record the DLNA protocol-related fields in the configuration file. When recording the screen projection protocol type, the display device 200 can call the configuration file again and read the DLNA protocol-related fields from the configuration file, thereby recording that the current screen projection protocol type is the DLNA protocol.

[0081] In some embodiments, the display device 200 can also determine the type of screen projection protocol based on the projection data. Since different screen projection protocols can use different data formats and compression algorithms for projection data, the type of screen projection protocol can be determined by reading the data format and compression algorithm of the projection data. For example, the Miracast protocol uses WiFi Direct technology for data transmission, while the AirPlay protocol uses Apple's proprietary data format for transmission. When the data transmission format conforms to WiFi Direct technology, the screen projection protocol type can be determined to be the Miracast protocol.

[0082] In some embodiments, the display device 200 can also obtain and record the screen projection protocol type based on the screen projection application. For example, when the A video application is installed and running on both the display device 200 and the terminal device 500, the user can click the screen projection control in the upper right corner of the A video application interface of the terminal device 500, and the terminal device 500 can establish a screen projection connection with the display device 200 through the screen projection protocol supported by the A video application. Therefore, when recording the screen projection protocol type, the display device 200 can determine that the current screen projection application is the A video application by reading the package name information of the screen projection application, and then query the default or supported screen projection protocol of the A video application to obtain the screen projection protocol type.

[0083] While recording the screen projection protocol type, the display device 200 can also record the media resource playback address. The media resource playback address can be obtained by reading the signal source address of the player. For example, after the display device 200 establishes a screen projection connection with the terminal device 500, the terminal device 500 can send the URL address of the screen projection data to the display device 200 based on the push screen projection method. The display device 200 then sets the data source of the player based on the received URL address. Therefore, when recording the media resource playback address, the display device 200 can directly read the data source of the player to obtain the media resource playback address.

[0084] Since the configuration file of the screen projection interface is created when the screen projection connection relationship is established, when creating the configuration file, the display device 200 can also write the media playback address sent by the terminal device 500 into the configuration file. Therefore, in some embodiments, the display device 200 can also obtain and record the media playback address by reading the configuration file.

[0085] While recording the projection protocol type and the media playback address based on the above embodiment, the display device 200 can also record the media playback progress according to the input time of the exit instruction, that is, record the media playback progress at the time the exit instruction is input. The media playback progress can be represented by the playback time or by a percentage. For example, the display device 200 can obtain the start time T0 of the player playing the projection data and the total media duration T1 of the projection data through the decoding result of the projection data, and then calculate the media playback progress in combination with the input time T of the exit instruction. When time is used to represent the media playback progress, the display device 200 can calculate the difference between the input time T and the start time T0, that is, T-T0 to obtain the media playback progress. When percentage is used to represent the media playback progress, the display device 200 can calculate the difference between the input time T and the start time T0 and then calculate the percentage of the difference and the total media duration T1, that is, the media playback progress is equal to (T-T0) / T1.

[0086] In some embodiments, in addition to recording the projection protocol type, media playback address, and media playback progress, the display device 200 may also record other relevant information that facilitates user interaction, including user settings and display content information. For example, in response to an exit instruction, the display device 200 may also record the media ID (name, title, etc.), thumbnail, and user-set playback effect information such as speed, audio track, subtitles, and zoom mode.

[0087] It should be noted that for the display device 200, the active exit instruction is actively input by the user and may include the user's intention to actively stop screen projection. The passive exit instruction is caused by events such as unexpected operations and therefore does not include the user's intention to actively stop screen projection. In the state where there is no intention to actively stop screen projection, the user needs to continue screen projection. Therefore, in some embodiments, the display device 200 may only record the resume information when receiving the passive exit instruction.

[0088] In the process of recording the progress of media playback, the display device 200 can also detect the progress of media playback and the type of media to determine whether the playback progress or the type of media supports screen projection and continuation. For example, for video media such as movies, when the progress of media playback is large and the main film has been played, it is not meaningful for users to continue to play only the ending song content, so it is not necessary to continue the playback of the projection screen, that is, the continuation information corresponding to the projection screen is not recorded. For another example, live broadcast videos do not have a fixed length and cannot be continued, so the continuation information corresponding to the projection screen may not be recorded. For this purpose, the continuation information can be generated according to the remaining duration through the process shown in Figure 8, as shown in Figure 8, including the following steps:

[0089] S801: Obtain the total duration of the media resources of the projection screen;

[0090] S802: Calculate the remaining duration of the media resource based on the total duration of the media resource and the progress of the media resource playback;

[0091] S803: Determine whether the remaining duration of the media resource is greater than or equal to a preset duration threshold;

[0092] S804: If yes, generate playback continuation information based on the projection protocol type, media resource playback address, media resource playback progress, and playback effect information;

[0093] S805: If not, close the screen projection connection channel with the terminal device.

[0094] If the remaining duration of the media resource is greater than or equal to the preset duration threshold, it indicates that the remaining duration of the projected media resource is long, making it convenient to resume the projected broadcast. The display device 200 can then generate resume information based on the projected broadcast protocol type, media resource playback address, media resource playback progress, and playback effect information. If the remaining duration of the media resource is less than the preset duration threshold, it indicates that the remaining duration of the projected media resource is short, making it difficult to resume the projected broadcast. Therefore, the display device 200 can directly close the projected broadcast connection channel with the terminal device 500.

[0095] For example, when the display device 200 exits the screen projection interface, it can determine whether the video corresponding to the screen projection has been played or is a live video. If the remaining time of the video is greater than 1 minute, it is determined that the current screen projection supports continued playback. Therefore, the display device 200 can record the media playback address and media playback progress, and generate continued playback information. If the remaining time of the video is less than 1 minute, it is determined that the current screen projection does not support continued playback, so it can be exited directly. When the playback time (or remaining time) is detected to be 0, it can be determined that the current screen projection is a live video and does not support continued playback, so it can also be exited directly.

[0096] S300: Generate the resume broadcast information according to the screen projection protocol type, the media resource playback address, the media resource playback progress, and the playback effect information;

[0097] After recording the screen projection protocol type, media resource playback address, media resource playback progress, and playback effect information, the display device 200 can convert the recorded information according to a preset data format to generate resume broadcast information.

[0098] In some embodiments, the display device 200 can record and manage resume information through a specific application. That is, the display device 200 is provided with a screen casting entry application, which is an application that provides a screen casting start interface. For example, the display device 200 can save the information recorded in the above embodiment in the screen casting entry application through the Setting.Global process, so that the screen casting entry application can display the screen casting start interface.

[0099] Since the screen casting process is equivalent to providing a screen casting signal source for the display device 200, the screen casting start interface can be built into the signal source selection interface or be the same interface as the signal source selection interface. As shown in Figure 9, in the screen casting start interface, multiple signal source options can be included, such as a live TV signal source, an HDMI interface signal source, a USB interface signal source, a screen casting signal source, etc. In these signal source options, the signal source type can be displayed through view controls such as title text, icons, and thumbnails for users to select.

[0100] In some embodiments, the display device 200 may record the projection protocol type, media playback address, media playback progress, and playback effect information, and then record the projection screen content at the time the exit command is input, i.e., the resume information. In implementation, as shown in FIG. 10 , the display device 200 may generate a resume image based on the media playback progress ( S1001 ), compress the projection image to generate a thumbnail ( S1002 ), and then add the thumbnail to the resume information ( S1003 ).

[0101] To help users quickly find the video they want to resume playing, they can also add the video name of the video corresponding to the signal source in the signal source option of the screen projection start interface, as well as the thumbnail corresponding to the resumed playback image generated according to the media playback progress of the video, such as the signal source option USB1 shown in Figure 9 above.

[0102] Furthermore, considering that the thumbnail is displayed to the user through the aforementioned signal source option, and is used to allow the user to quickly browse to the video image corresponding to the media playback progress through the thumbnail, rather than to instruct the display device how to perform the resume function, after the resume image is generated in S1001, the resume image can be directly added to the resume information (S1004). That is, when recording the resume information, the resume image is not compressed to generate a thumbnail.

[0103] Among them, the resumed broadcast image is used to display the content of the projection screen at the moment the exit instruction is input. Depending on the image acquisition rules, the display device 200 can obtain the resumed broadcast image in different ways. In some embodiments, the display device 200 can extract the resumed broadcast graphics from the data stream of the projection screen, that is, the resumed broadcast image is a picture frame extracted from the data stream of the projection screen according to the progress of media playback. After the display device 200 establishes a projection connection relationship with the terminal device 500, the display device 200 can obtain the projection data and form a projection data stream based on the URL address sent by the terminal device 500. The projection data stream includes multiple continuous picture frames, and each picture frame is provided with a timestamp for playback. When obtaining the exit instruction, the display device 200 can record the input time of the exit instruction, and read the timestamp of the picture frame according to the input time, determine the picture frame with the smallest time difference from the input time, and copy the picture frame to generate the resumed broadcast image.

[0104] In some embodiments, the display device 200 can take a screenshot of the current projection interface by calling a screenshot process to obtain a resumed image. That is, the resumed image is an image obtained by taking a screenshot of the projection screen by calling a screenshot interface. For example, when the display device 200 receives an exit instruction, it can call the Android developer API to call the system's screenshot process. That is, the display device 200 first calls the screenshot process through the "import android.graphics.Bitmap; import android.graphics.Canvas; import android.view.View; import android.widget.ImageView" instruction; and then obtains the View to be screenshoted through the "View viewToCapture=findViewById(R.id.view_to_capture)" instruction;

[0105] Next, create a Bitamp object and use the Bitmap object as its bitmap. During implementation, follow the "Bitmap bitmap = Bitmap.createBitmap(viewToCapture.getWidth(), viewToCapture.getHeight(), Bitmap.Config.ARGB_8888)" instruction to create a Bitmap object for storing the screenshot image. Then follow the "Canvas canvas = new Canvas(bitmap)" instruction to create a Canvas object and use the Bitmap object as its bitmap. Then, use the viewToCapture.draw(canvas) instruction to draw the View to be screenshot onto the Canvas. Finally, follow the "MediaStore.Images.Media.insertImage(getContentResolver(), bitmap,"Screenshot", null); catch(FileNotFoundException e) e.printStackTrace()" instruction to save the screenshot to the album of the display device 200.

[0106] In some embodiments, the display device 200 can also extract a continued broadcast image from the frame buffer of the projection interface, that is, the continued broadcast image is a picture frame extracted from the frame buffer of the projection interface according to the time corresponding to the media playback progress. For example, as shown in Figure 11, the projection interface of the display device 200 is an overlay of the video layer and the graphics (On Screen Display, OSD) layer, that is, the image content of the projection interface can be rendered by the graphics processor GPU component, thereby forming a picture frame for display in the synthesis module. Among them, the synthesis module can store picture frames through the frame buffer (frame buffer) and the video buffer (video buffer) respectively, and form picture content in the corresponding layers respectively through the video process module. That is, the image frame corresponding to the projection screen of the projection process forms the picture content through the video layer (video plane), and the image frame of other image screens in the projection interface (such as controls, progress bars and other graphics of the UI interface) forms the picture content through the OSD layer (OSD plane). Finally, the display device 200 forms the final picture for display through the mixing unit (blend) of the video process module. Therefore, the display device 200 can extract the picture frame of the final picture at the time of inputting the exit instruction in the mixing unit, or extract the picture frame from the video buffer corresponding to the video layer as the resumed broadcast image.

[0107] In some embodiments, the display device 200 can also directly obtain the continued broadcast image through the data source of the projection data. For example, the display device 200 can access the website link corresponding to the media playback address for the second time through the multimedia framework service (such as ffmpeg), and encapsulate it as getScaledFrameAtTime(time, option, width, height) to obtain a frame thumbnail image corresponding to the media playback progress.

[0108] According to the method in the above embodiment, after the resume broadcast information is generated, the display device 200 can store the resume broadcast information. In the process of storing the resume broadcast information, the display device 200 can construct a specific storage file format to save the resume broadcast information. For example, the display device 200 can construct a resume broadcast information data table. In the resume broadcast information data table, a plurality of table items can be included, namely the current media asset playback address, media asset playback progress, screen projection protocol type, media asset title, and playback effect information that can be manually set by the user. The playback effect information may include playback speed, audio track, subtitles, zoom mode, etc. Each table item can be set with a corresponding specific value according to the recorded resume broadcast information content.

[0109] The resume information can be stored in a specific storage area in the memory of the display device 200, and the resume information can be set to a state that can be called by the screen projection entry application or other applications. After storing the resume information, the display device 200 can execute an exit instruction to exit the screen projection state, that is, close the screen projection connection channel with the terminal device 500. After closing the screen projection connection channel between the display device 200 and the terminal device 500, the display device 200 can switch from displaying the screen projection interface to displaying other interfaces.

[0110] Since the user of the resume information subsequently performs a resume operation on the projection screen, and the resume operation has a high probability of occurring after the user exits the projection screen and reestablishes the projection connection within a short period of time, the resume information can also be stored according to the set duration. The implementation can be shown in Figure 12, including the following steps:

[0111] S1201: Obtain the storage duration and preset storage threshold of the resumed broadcast information;

[0112] S1202: Determine whether the storage duration is greater than or equal to a preset storage threshold;

[0113] S1203: If yes, delete the resume broadcast information;

[0114] For example, the preset storage threshold can be set to 24 hours based on the average value of the interval time for the display device 200 to re-establish the projection connection relationship after exiting the projection. Then, after storing the resume information, the display device 200 can record the storage time of the resume information. When the storage time exceeds 24 hours, the display device 200 can delete the resume information. By setting the preset storage threshold, not only can the situation where the user can still see this entrance for a long time in the projection start interface be reduced, but also the storage space occupied by the resume information can be reduced, saving storage space. In addition, by actively clearing the resume information regularly, the display device 200 can also reduce the matching amount of the resume information in the subsequent resume process, so as to facilitate the quick completion of the resume.

[0115] S1204: If not, monitor the start event of the projection screen corresponding to the resume broadcast information;

[0116] S1205: Determine whether a start event is monitored; if a start event is monitored, jump to the above step S1203, otherwise return to the above step S1204 to continue monitoring.

[0117] If the storage duration is less than the preset storage threshold, the display device 200 can maintain the storage state of the continuous broadcast information. Obviously, when the display device 200 has completed the continuous broadcast, the previously stored continuous broadcast information is meaningless to continue to be stored. That is, the display device 200 can also monitor the start event of the projection screen corresponding to the continuous broadcast information, and delete the continuous broadcast information after monitoring the start event.

[0118] For example, the display device 200 can monitor the start-up event of the projection screen during the period of the continuation information. When it is monitored that the current projection screen has been resumed according to the continuation information, the display device 200 can clear the continuation information so that the playback progress after the continuation can also be recorded and stored, thereby achieving the effect of continuous continuation.

[0119] In some embodiments, the display device 200 can also monitor the screen projection connection operations of other terminal devices, and after monitoring that the display device 200 establishes a screen projection connection with other terminal devices, the display device 200 clears the resume information of the corresponding projection screen of the original terminal device. For example, after the display device 200 stores the resume information A of the projection screen corresponding to terminal A, while the resume information A is stored, the display device 200 can monitor the projection request of other terminal devices. When it monitors that the display device 200 establishes a screen projection connection with terminal B, the display device 200 can clear the resume information to avoid the original projection screen from continuing to play when the new projection screen is playing, affecting the user experience.

[0120] In the above steps S100 to S300, it is shown that the display device 200 can record information such as the screen projection protocol type, media playback address, and media playback progress during the screen projection process when exiting the screen projection interface. The recorded information is saved in the form of resume information, so that the display device 200 can resume the projection screen based on the resume information, as shown in Figure 13.

[0121] S400: In response to a play instruction for playing the projected screen, obtain the resume play information of the projected screen.

[0122] After the display device 200 stores the resume information according to the method provided in the above embodiment, it can call the resume information when playing the projected screen in the subsequent process and control the playback based on the resume information. The resume information is the information recorded when exiting the playback of the projected screen, and includes the projection protocol type, media playback address, and media playback progress.

[0123] That is, the display device 200 can receive a playback instruction for playing the projected screen. The playback instruction can also be actively input by the user, or generated by the display device 200 through judgment of the operating status. For actively input playback instructions, the user can input instructions based on the interactive operation methods such as the terminal device 500, remote control, touch, voice, etc. supported by the display device 200. For example, the user can click the screen projection icon button in the upper right corner of the playback interface on the terminal device 500 to control the terminal device 500 to send a screen projection request to the display device 200, that is, input a playback instruction to the display device 200.

[0124] For the play command generated by the display device 200, the display device 200 can monitor the current operating state to determine whether the current state meets the set play conditions, and generate a play command when the play conditions are met. For example, after exiting the screen projection interface, the display device 200 can record the exit time of the screen projection interface to determine the length of time that has elapsed since the display device 200 exited the screen projection interface. If the screen projection interface is returned within the preset return time, the display device 200 can automatically generate a play command.

[0125] It should be noted that the playback instruction generated by the display device 200 should be conducive to the continued broadcast of the projection screen and will not affect the user's normal projection screen operation. For example, when returning to the projection screen interface in a short time, the display device 200 should detect the projection screen connection relationship after returning to the projection screen interface. If after returning to the projection screen interface, the terminal device 500 that establishes a projection screen connection relationship with the display device 200 is still the terminal device when exiting the projection screen interface, the display device 200 can automatically generate a playback instruction. If after returning to the projection screen interface, the terminal device 500 that establishes a projection screen connection relationship with the display device 200 is not the terminal device when exiting the projection screen interface, the display device 200 can detect the URL address of the current projection screen to determine whether it is the original projection screen, and automatically generate a playback instruction when the URL address at the time of return is the same as the URL address before exiting.

[0126] To facilitate user interaction, the display device 200 may also provide a screen projection interactive interface. For example, the display device 200 may call a screen projection entry application and obtain the resume information of the projection screen through the screen projection entry application. The screen projection entry application is then controlled to generate a screen projection start interface based on the resume information. The screen projection start interface includes at least one screen projection control, and the display content of the screen projection control is generated based on the resume information.

[0127] After the screen casting entry application generates the screen casting start interface, the display device 200 can control the display to display the screen casting start interface, and the user can input the play instruction through at least one screen casting control in the screen casting start interface.

[0128] For example, after receiving the screen projection request sent by the terminal device 500, the display device 200 obtains the resume information of the current screen projection. If the resume information of the current screen projection is stored in the display device 200, the screen projection entry application can be called to control the display device 200 to display the screen projection start interface through the screen projection entry application.

[0129] The projection start interface may include projection options corresponding to the current projection screen. The projection options may be associated with a resume control for the user to select. When the user clicks the resume control, the display device 200 may play the projection screen according to the resume information. At the same time, the display device 200 may also generate the above-mentioned playback instructions.

[0130] For example, when a user launches the screen casting portal app using the remote control, if the ContentProvider contains thumbnail information and the video title is in Setting.Global, the app listens for the resume action to detect whether the user clicks the resume control on the screen casting start screen. If no data is available, the screen casting portal interface is displayed, and after the user clicks the cast control, the app jumps to the screen casting guide page to re-establish the screen casting connection.

[0131] S500: Obtaining screen projection data from the media resource playback address through the virtual session object.

[0132] After obtaining the resume information in response to the play command, the display device 200 can parse the resume information to read data such as the projection protocol type, media resource playback address, and media resource playback progress. Based on the parsed data, the projection screen can be resumed. That is, the display device 200 can obtain the projection data from the media resource playback address and form a projection data stream for playback by the player.

[0133] When establishing a screen projection connection between the display device 200 and the terminal device 500, the user needs to perform the corresponding connection operation steps. For example, the user needs to first open the video playback interface through the terminal device 500 and then click the screen projection button in the video playback interface. And when the display device 200 displays the screen projection start interface, the user also needs to click the resume control in the screen projection start interface to complete the resume operation, which requires more operation steps.

[0134] Therefore, to simplify user operations, in some embodiments, the display device 200 can complete the screen projection connection through a virtual session instead of the terminal device 500. That is, as shown in Figure 14, after obtaining the resume information through the aforementioned process, the display device 200 creates a virtual session object based on the resume information. The virtual session object is used to replace the terminal device to establish a virtual screen projection connection with the display device according to the screen projection protocol type.

[0135] In some embodiments, in order to create a virtual session object based on the resume information, the display device 200 may also respond to a trigger indication of at least one of the screen casting controls after monitoring the user clicking on at least one of the screen casting controls in the above-mentioned screen casting start page, and determine the target screen casting control corresponding to the trigger indication. Then, the resume application associated with the cast screen is queried based on the target screen casting control. Next, the resume application is run, and the resume information is sent to the resume application to control the resume application to create a virtual session object based on the resume information. The resume application is an application used to form the cast screen.

[0136] For example, the display device 200 can be configured with a fusion screen projection service. When the screen projection entry application sends an Action message in response to the user's click operation on the target screen projection control, the fusion screen projection service will monitor this Action message and notify the adaptation layer of the resume application to create a virtual Session (i.e., a virtual session object) through the protocol type. Session can be used to store the properties and configuration information required for a specific user session. When the user jumps between application interfaces, the variables stored in the Session object will not be lost, but will continue to exist throughout the user session. When the user request comes from the application interface, if the user does not have a session yet, the display device 200 can automatically create a Session. When the session expires or is abandoned, the display device 200 will terminate the session.

[0137] In order to reduce the storage capacity of the display device for resume information, after the resume information is generated through steps S100-S300, the resume information may not be stored locally. Instead, part of the resume information is stored in the screen projection control, and the rest is stored in the resume application. For example, the media playback address, user-set playback effect information, etc. can be stored in the screen projection control, and the screen projection protocol type, media playback progress, etc. can be stored in the resume application. In this way, after determining the target screen projection control triggered by the user, the resume information can be obtained from the target screen projection control and the resume application corresponding to the target screen projection control.

[0138] The virtual session object created by the display device 200 can replace the terminal device 500 to establish a virtual screen projection connection with the display device 200 according to the screen projection protocol type recorded in the resume broadcast information, so that the display device 200 does not need to repeatedly re-establish the screen projection connection relationship with the terminal device 500, but directly uses the media resource playback address recorded in the resume broadcast information to obtain the screen projection data, that is, obtains the screen projection data from the media resource playback address through the virtual session object.

[0139] In some embodiments, the virtual session object can also be used to obtain resume information and use the resume information as a control variable during playback for use by the player of the display device 200. That is, upon receiving the resume message, the adaptation layer of the resume application creates a virtual session and obtains the saved resume information.

[0140] S600: Play the start time of the projection data according to the media resource playback progress, and play the projection data from the start time based on the playback effect information.

[0141] After acquiring the projection data, the display device 200 can input the projection data into the player, which then performs playback operations such as decoding, rendering, and outputting the screen on the input projection data to form a projection screen. Simultaneously, the display device 200 can send the media playback progress in the resume information and the playback effect information manually set by the user to the player, allowing the player to determine the start time of the projection data according to the media playback progress and start playing the projection data from the start time according to the effects set in the playback effect information (e.g., the speed, audio track, subtitles, etc. of the aforementioned video playback).

[0142] For example, the resume information obtained by display device 200 may include playback progress of media asset A at 0:38:25, speed multiplier 1.2, subtitles not displayed, and other playback effect information. Display device 200 can then retrieve the data stream from the playback address of media asset A starting at 0:38:25 and input the retrieved data stream into the player. The player can then decode and render the media asset data after 0:38:25 and output the image to display 260, causing display 260 to play the projection image after 0:38:25 at 1.2x speed.

[0143] The above embodiment can record the resume information when projecting the screen in a push manner, and call the recorded resume information when playing the projected screen, so that the display device 200 can play the projected screen data according to the media playback progress in the resume information, thereby achieving the projected screen resume effect.

[0144] Since different display devices 200 support different screen projection methods, the functions supported after establishing a corresponding screen projection connection are also different. For example, when the display device 200 establishes a screen projection connection through a video application, different video applications support the resume function differently. That is, for the display device 200 system video application and the video application developed by the third-party application vendor that has a cooperative relationship with the display device 200 operator, the resume information can be recorded and read according to the cooperation specifications. However, for third-party video applications without a cooperative relationship, because they do not have the corresponding recording and reading algorithms built in, they do not support the screen projection resume function. Based on this, the display device of the embodiment of the present disclosure can also perform screen projection resume according to the application type. When implemented, it can be shown in Figure 15, including the following steps:

[0145] S1501: Determine the application type of the resumed application;

[0146] The application type may include a first type of application that supports screen casting and resume, or a second type of application that does not support screen casting and resume. During implementation, the application package name information can be read to obtain the application name, developer and other description information, and then matched with a preset matching table to determine the application type.

[0147] For example, if the developer of Application A is the operator of display device 200, that is, Application A is a system application of display device 200, then it can be determined that Application A supports the screen casting and resume function, that is, the application type of Application A is a first-category application. Correspondingly, if it is determined by reading the application package name information that the developer of Application B is not the operator of display device 200, and that the developer of Application B does not have a cooperative relationship with the operator of display device 200, then it can be determined that Application B does not support the screen casting and resume function, that is, the application type of Application B is a second-category application.

[0148] S1502: If the application type is a first-category application, the resume broadcast information is sent to the resume broadcast application;

[0149] S1503: If the application type is a second type application, a screen projection connection request is sent to the terminal device;

[0150] S1504: Receive the projection data fed back by the terminal device according to the projection connection request.

[0151] If the application type is a second type of application, it means that the resumed broadcast application does not support the screen casting and resumed broadcasting function. In this case, a screen casting connection request can be sent to the terminal device 500, and the screen casting data fed back by the terminal device 500 according to the screen casting connection request can be received. That is, when the display device 200 determines that the resumed broadcast application does not support the screen casting and resumed broadcasting in the manner of recording the resumed broadcast information, it can resend the screen casting connection request to the terminal device 500 to re-establish the screen casting connection relationship.

[0152] In order to implement the resume function, as shown in FIG16 , the display device 200 may further monitor the switching event of the projection interface when determining that the application type of the resumed application is a second-category application (S1601). The switching event refers to an event that triggers the display device 200 to switch from displaying the projection interface to displaying another interface, which may be triggered by an exit command or by other conditional judgment methods.

[0153] After monitoring the switching event, the display device 200 can generate a pause play instruction in response to the switching event (S1602). The pause play instruction is used to pause the playback of the projection data corresponding to the projection screen. That is, the display device 200 can control the projection screen to pause by the pause play instruction, and switch the projection interface to background operation (S1603). For different projection modes, the display device 200 implements different ways of pausing playback. When the projection connection is established by push, the playback process is executed by the player of the display device 200. The display device 200 can directly respond to the pause play instruction after generating the pause play instruction to pause the playback process of the projection data. When the projection connection is established by mirroring, since the playback process is not executed by the player of the display device 200, but by the terminal device 500, it can be sent to the terminal device 500 after generating the pause play instruction, so that the terminal device 500 responds to the pause play instruction and pauses the playback process of the projection data.

[0154] After switching the projection interface to background operation, the display device 200 can monitor the return event of the projection interface within a preset time (S1604). When the return event is monitored, the display device 200 responds to the return event and generates a resume playback instruction (S1605), that is, resumes playing the projection data corresponding to the projection screen, and controls the display 260 to display the projection interface (S1606).

[0155] It can be seen that in the above embodiment, the display device 200 can adopt different methods to realize screen projection and continuation according to the application type of the resumed application, so that the display device 200 can adapt to more types of resumed applications to solve the problem that the screen projection playback process cannot be resumed.

[0156] Based on the above-mentioned display device 200, the embodiment of the present disclosure also provides a screen projection continuation method, including: in response to a playback instruction for playing a projection screen, obtaining the continuation information of the projection screen, the continuation information is the information recorded when exiting the playback of the projection screen, and the continuation information includes the projection protocol type, the media playback address, the media playback progress, and the playback effect information set by the user; creating a virtual session object based on the continuation information, the virtual session object is used to replace the terminal device to establish a virtual projection connection with the display device according to the projection protocol type; obtaining projection data from the media playback address through the virtual session object; playing the start time of the projection data according to the media playback progress, and playing the projection data from the start time based on the playback effect information.

[0157] From the above process, it can be seen that this method can record the resume information when the display device cuts out the projection interface, and when returning to the projection interface, it supports the display device to obtain the previously saved resume information, and jump to the corresponding media playback progress according to the resume information, so as to realize the resume of the projection screen and solve the problem that the projection playback process cannot be resumed.

Claims

1. A display device, comprising: A display configured to display a screen projection interface; A user input interface configured to receive instructions from a user; A communication device, configured to establish a screen projection connection with a terminal device; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, the user input interface, the communication device and the memory, is configured to execute computer instructions to cause the display device to perform: In response to a playback instruction for playing a projection screen, obtaining the resume information of the projection screen, the resume information being information recorded when exiting the playback of the projection screen, the resume information including the projection protocol type, the media resource playback address, the media resource playback progress, and the playback effect information set by the user; Creating a virtual session object based on the resume broadcast information, where the virtual session object is used to replace the terminal device to establish a virtual screen projection connection with the display device according to the screen projection protocol type; Acquire the projection data from the media resource playback address through the virtual session object; The start time of playing the projection data is played according to the media resource playing progress, and the projection data is played from the start time based on the playing effect information.

2. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Get the exit command for exiting the screen projection interface; In response to the exit instruction, recording the projection protocol type and media playback address of the projection interface, and recording the media playback progress and playback effect information at the time when the exit instruction is input; Generate the resume broadcast information according to the screen projection protocol type, the media resource playback address, the media resource playback progress, and the playback effect information; The resume broadcast information is stored, and the screen projection connection channel with the terminal device is closed.

3. The display device according to claim 2, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Before storing the resumed broadcast information, a resumed broadcast image is generated according to the media resource playback address and the media resource playback progress; the resumed broadcast image is a picture frame extracted from the data stream of the projection screen according to the media resource playback progress; or, the resumed broadcast image is an image obtained by calling a screenshot interface to perform a screenshot on the projection screen; Performing compression processing on the projection image to generate a thumbnail; The thumbnail is added to the resume information.

4. The display device according to claim 3, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: After the replay information is broadcasted, the storage duration and preset storage threshold of the replay information are obtained; If the storage duration is greater than or equal to the preset storage threshold, deleting the resume broadcast information; If the storage duration is less than the preset storage threshold, monitor the start event of the projection screen corresponding to the resume broadcast information, and delete the resume broadcast information after monitoring the start event.

5. The display device according to claim 2, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: After recording the media resource playback address, the media resource playback progress and the playback effect information, obtain the projection screen The total duration of the media assets in the screen; Calculate the remaining duration of the media resource according to the total duration of the media resource and the progress of the media resource playback; If the remaining duration of the media resource is greater than or equal to the preset duration threshold, the step of generating the resume information according to the screen projection protocol type, the media resource playback address, and the media resource playback progress is executed; If the remaining duration of the media resource is less than the preset duration threshold, the step of closing the screen projection connection channel with the terminal device is executed.

6. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device receives the playback instruction in the following manner: Calling a screen casting entry application, wherein the screen casting entry application is an application that provides a screen casting start interface; Obtaining the resume broadcast information of the projected screen through the projection entry application; Control the screen casting entry application to generate a screen casting start interface according to the resume broadcast information, wherein the screen casting start interface includes at least one screen casting control, and display content of the screen casting control is generated according to the resume broadcast information; Control the display to display the screen casting start interface to receive the play instruction through the screen casting start interface.

7. The display device according to claim 6, wherein the at least one processor is specifically configured to execute computer instructions so that the display device creates a virtual session object based on the resume information by performing the following: In response to a trigger indication of the at least one screen projection control, determining a target screen projection control corresponding to the trigger indication; Querying the continued broadcast application associated with the projection screen according to the target projection screen control, where the continued broadcast application is an application used to form the projection screen; Running the resume broadcast application, and sending the resume broadcast information to the resume broadcast application; The resume application is controlled to create a virtual session object based on the resume information.

8. The display device according to claim 7, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Detecting an application type of the resumed broadcast application, where the application type is a first type of application that supports screen casting and resumed broadcast or a second type of application that does not support screen casting and resumed broadcast; If the application type is a first type application, executing the step of sending the resume broadcast information to the resume broadcast application; If the application type is a second type of application, a screen projection connection request is sent to the terminal device, and screen projection data fed back by the terminal device according to the screen projection connection request is received.

9. The display device according to claim 8, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: If the application type is a second type of application, monitoring a switching event of the projection interface; In response to the switching event, generating a pause playback instruction, wherein the pause playback instruction is used to pause playback of the projection data corresponding to the projection screen; Switch the screen projection interface to background operation; Monitor the return event of the projection interface within a preset time; In response to the return event, a resume playback instruction is generated, and the display is controlled to display the projection interface, and the resume playback instruction is used to resume playing the projection data corresponding to the projection screen.

10. A screen projection and continuous broadcasting method, comprising: In response to a playback instruction for playing a projection screen, obtaining the resume information of the projection screen, the resume information being information recorded when exiting the playback of the projection screen, the resume information including the projection protocol type, the media resource playback address, the media resource playback progress, and the playback effect information set by the user; Creating a virtual session object based on the resume broadcast information, where the virtual session object is used to replace the terminal device to establish a virtual screen projection connection with the display device according to the screen projection protocol type; Acquire the projection data from the media resource playback address through the virtual session object; The start time of playing the projection data is played according to the media resource playing progress, and the projection data is played from the start time based on the playing effect information.

Images