Screen projection method and display equipment

Abstract

The invention discloses a screen projection method and a display device, and relates to the technical field of display device.The method comprises the steps that in a multi-path screen projection mode, first screen content, sent by the terminal device through mirror image screen projection connection, of the terminal device is received, and a first screen projection interface corresponding to the first screen content is displayed in a first screen projection area; in response to a first control instruction sent by the control equipment, performing screenshot on the content currently displayed by the display equipment to obtain a first screenshot image; acquiring first action information; generating a first anti-control instruction according to the first action information, wherein the first anti-control instruction is used for controlling the terminal equipment to initiate a first DLNA projection screen connection; sending a first anti-control instruction to the terminal equipment through the mirror image projection screen connection; and in response to a first screen projection instruction sent by the terminal equipment, obtaining a first media resource according to the first resource address, and playing the first media resource in the second screen projection area. According to the invention, the terminal equipment can be reversely controlled to carry out screen projection by controlling the display equipment.

Description

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on April 30, 2025, with application number 202510573616.6, the contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the technical field of display devices, and in particular to a screen projection method and a display device. Background Art

[0003] With the continuous development of terminal technology, multi-screen interaction technology is becoming more and more mature. For example, users can cast the content displayed on a small-screen device (such as a mobile phone) to a large-screen device (such as a smart TV) for better display effects and viewing experience.

[0004] When projecting content displayed on a small-screen device to a large-screen device, when the user wants to switch viewing content, he or she needs to shift his or her attention to the small-screen device and perform corresponding operations, and the viewing content will be switched synchronously on the large-screen device.

[0005] However, this method causes the user's visual focus to repeatedly switch between large-screen devices and small-screen devices, and the operation is cumbersome and the user experience is poor. Summary of the Invention

[0006] The present application provides a screen projection method and a display device. The user only needs to trigger a split-screen projection command, and the display device reversely controls the terminal device to project the media resources currently displayed on the terminal device onto the display device. In this way, during the screen projection process, the user does not need to operate on the terminal device, avoiding the user's visual focus repeatedly switching between the display device and the terminal device, thereby improving the user experience.

[0007] In a first aspect, some embodiments of the present application provide a screen projection method, which is applied to a display device, and the method includes: in a multi-way screen projection mode, receiving the first screen content currently displayed by the terminal device sent by the terminal device through the mirror projection connection, displaying a first interface, the first interface including a first projection area and a second projection area, displaying a first projection interface corresponding to the first screen content in the first projection area, the first projection interface including a first designated element, and the first designated element corresponding to the first control in the first screen content; in response to a first control instruction sent by the control device, taking a screenshot of the content currently displayed by the display device to obtain a first screenshot image; obtaining first action information corresponding to the first designated element, and the first action information is obtained by identifying the first designated element in the first screenshot image. a designated element and the first position information of the first designated element, and is determined based on the action type corresponding to the first position information and the first designated element; according to the first action information, a first reverse control instruction is generated, the first reverse control instruction is used to control the terminal device to initiate a first DLNA screen projection connection to the display device, and send a first screen projection instruction to the display device through the first DLNA screen projection connection, the first screen projection instruction carries a first resource address, and the first resource address is the resource address of the first media resource corresponding to the first screen content currently displayed by the terminal device; through the mirror screen projection connection, the first reverse control instruction is sent to the terminal device; in response to the first screen projection instruction sent by the terminal device, the first media resource is obtained according to the first resource address, and the first media resource is played in the second screen projection area.

[0008] In some examples, the first interface also includes: a first functional control, and the method further includes: obtaining a first control instruction in response to a selection instruction for the first functional control.

[0009] In some examples, before playing the first media resource in the second projection area, the method further includes: displaying a guide animation in the second projection area, where the guide animation is used to guide the execution of the first control instruction.

[0010] In some examples, before receiving the first screen content currently displayed by the terminal device sent by the terminal device via the mirroring screen projection connection, the method also includes: responding to a startup instruction of the multi-way screen projection mode, starting the multi-way screen projection mode, and displaying a guide interface, wherein the guide interface is used to guide the split-screen display of media resources corresponding to the screen content sent by the terminal device via the mirroring screen projection connection and the screen content currently displayed by the terminal device.

[0011] In a second aspect, some embodiments of the present application provide a screen projection method, which is applied to an intelligent body, and the method includes: receiving a first screenshot image sent by a display device, the first screenshot image is a screenshot image obtained by the display device in response to a first control instruction, the first screenshot image includes a first screen projection interface displayed by the display device in a first screen projection area, the first screen projection interface corresponds to the first screen content currently displayed by the terminal device, the first screen projection interface includes a first specified element, and the first specified element corresponds to the first control in the first screen content; identifying the first specified element in the first screenshot image and the first position information of the first specified element; determining the first action information based on the first position information of the first specified element and the action type corresponding to the first specified element; sending the first action information to the display device, so that the display device generates a first reverse control instruction based on the first action information, the first reverse control instruction is used to control the terminal device to initiate a first DLNA screen projection connection to the display device, and send the first screen projection instruction to the display device through the first DLNA screen projection connection, the first screen projection instruction carries a first resource address, and the first resource address is the resource address of the first media resource corresponding to the first screen content currently displayed by the terminal device.

[0012] On the third aspect, some embodiments of the present application also provide a display device, including a display; a controller connected to the display, the controller being configured to: in a multi-way screen projection mode, receive the first screen content currently displayed by the terminal device sent by the terminal device via the mirror projection connection, display a first interface, the first interface including a first projection area and a second projection area, and display a first projection interface corresponding to the first screen content in the first projection area, the first projection interface including a first designated element, and the first designated element corresponds to the first control in the first screen content; in response to a first control instruction sent by the control device, take a screenshot of the content currently displayed by the display device to obtain a first screenshot image; obtain first action information corresponding to the first designated element, the first action information is obtained by identifying the first The first designated element in the screenshot image and the first position information of the first designated element are determined based on the first position information and the action type corresponding to the first designated element; according to the first action information, a first reverse control instruction is generated, and the first reverse control instruction is used to control the terminal device to initiate a first DLNA screen projection connection to the display device, and send a first screen projection instruction to the display device through the first DLNA screen projection connection, the first screen projection instruction carries a first resource address, and the first resource address is the resource address of the first media resource corresponding to the first screen content currently displayed by the terminal device; through the mirror screen projection connection, the first reverse control instruction is sent to the terminal device; in response to the first screen projection instruction sent by the terminal device, the first media resource is obtained according to the first resource address, and the first media resource is played in the second screen projection area.

[0013] In a fourth aspect, some embodiments of the present application also provide an electronic device, including: a receiving module for receiving a first screenshot image sent by a display device, the first screenshot image is a screenshot image obtained by the display device in response to a first control instruction, the first screenshot image includes a first projection interface displayed by the display device in a first projection area, the first projection interface corresponds to the first screen content currently displayed by the terminal device, the first projection interface includes a first specified element, and the first specified element corresponds to the first control in the first screen content; an identification module for identifying the first specified element in the first screenshot image and the first position information of the first specified element; an action generation module for determining the first action information based on the first position information of the first specified element and the action type corresponding to the first specified element; a sending module for sending the first action information to the display device, so that the display device generates a first reverse control instruction based on the first action information, the first reverse control instruction is used to control the terminal device to initiate a first DLNA projection connection to the display device, and send the first projection instruction to the display device through the first DLNA projection connection, the first projection instruction carries a first resource address, and the first resource address is the resource address of the first media resource corresponding to the first screen content currently displayed by the terminal device.

[0014] In the fifth aspect, some embodiments of the present application provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the screen projection method described in the first aspect, or the processor executes the screen projection method described in the second aspect.

[0015] In a sixth aspect, some embodiments of the present application provide a computer program product, which includes: a computer program code, which, when the computer program code runs on a display device, enables the display device to execute the screen projection method described in the first aspect.

[0016] In an embodiment of the present application, in a multi-channel screen projection mode, a first screen content of a terminal device is sent by a receiving terminal device via a mirrored screen projection connection, and a first screen projection interface corresponding to the first screen content is displayed in a first screen projection area; in response to a first control instruction sent by a control device, the display device can take a screenshot, and by identifying and analyzing the screen content currently displayed by the terminal device in the screenshot image, a reverse control instruction is generated and sent to the terminal device to control the terminal device to switch the screen content. After the switched screen content of the terminal device is displayed in the screen projection area 1 of the display device, the display device can take another screenshot, and based on the screenshot image, a reverse control instruction is generated and sent to the terminal device, thereby reversely controlling the terminal device to establish a DLNA screen projection connection with the display device and projecting the media resources currently displayed by the terminal device onto the display device. In other words, the user only needs to trigger the split-screen projection instruction, and the display device reversely controls the terminal device to project the media resources currently displayed by the terminal device onto the display device. In this way, during the screen projection process, the user does not need to operate on the terminal device, avoiding the user's visual focus from repeatedly switching between the display device and the terminal device, thereby improving the user experience.

[0017] Furthermore, in an embodiment of the present application, the mirrored content and other media resources of the terminal device can be displayed in split screen on the display device, so that the user can view the screen content of the terminal device on the display device while watching other media resources, without affecting the user's use of the terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 This is a schematic diagram of an interactive process of a screen projection process in the prior art;

[0020] Figure 2 This is a schematic diagram of the interactive process of another screen projection process in the prior art;

[0021] Figure 3 A schematic diagram of an operation scenario between a display device and a control device provided in an embodiment of the present application;

[0022] Figure 4 A hardware configuration block diagram of a display device provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of a software system architecture of a display device provided in an embodiment of the present application;

[0024] Figure 6 A software schematic diagram of an intelligent agent provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of a screen projection application scenario provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of the interactive process of activating the split-screen projection function provided in an embodiment of the present application;

[0027] Figure 9 A schematic diagram of the interactive process of a mirror screen projection process provided in an embodiment of the present application;

[0028] Figure 10 This is one of the interactive flow diagrams of the drag split screen process provided in an embodiment of the present application;

[0029] Figure 11 The second interactive flow diagram of the drag split screen process provided in the embodiment of the present application;

[0030] Figure 12 The third interactive flow diagram of the drag split screen process provided in the embodiment of the present application;

[0031] Figure 13 The fourth interactive flow diagram of the drag split screen process provided in the embodiment of the present application;

[0032] Figure 14 The fifth interactive flow diagram of the drag split screen process provided in the embodiment of the present application;

[0033] Figure 15 Schematic diagram of the interactive process of dragging the split screen provided in the embodiment of the present application;

[0034] Figure 16 Schematic diagram of the interactive process of dragging the split screen according to the embodiment of the present application;

[0035] Figure 17 A schematic diagram of the interactive process of a one-click split-screen projection process provided in an embodiment of the present application;

[0036] Figure 18 A schematic diagram of another interactive process of a one-click split-screen projection process provided in an embodiment of the present application;

[0037] Figure 19 A schematic diagram of a knowledge graph provided in an embodiment of the present application;

[0038] Figure 20 A schematic diagram of another knowledge graph provided in an embodiment of the present application;

[0039] Figure 21 This is a flowchart of a screen projection method according to an embodiment of the present application;

[0040] Figure 22 The second flowchart of the screen projection method provided in the embodiment of the present application;

[0041] Figure 23 The third flowchart of the screen projection method provided in the embodiment of the present application;

[0042] Figure 24 This is a fourth flowchart of the screen projection method provided in an embodiment of the present application;

[0043] Figure 25 This is a fifth flowchart of the screen projection method provided in an embodiment of the present application;

[0044] Figure 26 This is the sixth flow chart of the screen projection method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0045] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all possible implementations consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the claims.

[0046] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0047] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0048] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0049] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functionality associated with that element.

[0050] In order to better understand the embodiments of the present application, the terms or concepts that may be involved in the embodiments of the present application are explained below.

[0051] 1. Mirror screen

[0052] Mirror screen projection (also known as system screen projection) refers to a technology that allows the sending device (also known as the screen projection source device) and the receiving device (also known as the screen projection target device) to synchronize screen images based on protocols such as Miracast when they are in the same local area network.

[0053] Taking the sending device as mobile phone 1 and the receiving device as smart TV 2 as an example, after mobile phone 1 and smart TV 2 establish a connection through the WiFi network, in response to the mirroring screen projection instruction, mobile phone 1 scans for available receiving devices nearby. After mobile phone 1 finds smart TV 2, mobile phone 1 can establish a connection with smart TV 2 based on the Miracast protocol. After the connection is successful, mobile phone 1 can project its screen display content to smart TV 2 for display.

[0054] It is understandable that during the mirroring process, the screen display content of the receiving device will change as the screen display content of the sending device changes, and the screen display content of the receiving device is always consistent with the screen display content of the sending device.

[0055] 2. Streaming media screen projection

[0056] Streaming media screen casting refers to the technology of pushing streaming media resources based on the protocol authorized by the Digital Living Network Alliance (DLNA) (abbreviated as DLNA protocol) when the sending device and the receiving device are in the same local area network.

[0057] Continuing with the example of the sending device being mobile phone 1 and the receiving device being smart TV 2, when mobile phone 1 and smart TV 2 are in the same local area network, when mobile phone 1 plays media resources (such as videos) through an application that supports the DLNA protocol, in response to the streaming media screen casting instruction, mobile phone 1 can negotiate with smart TV 2 based on the DLNA protocol to cast the screen. After the negotiation is completed, mobile phone 1 can send the link address of the media resource it is playing to smart TV 2. Smart TV 2 obtains the media resource from the server according to the link address and displays the media resource on the screen of smart TV 2, thereby realizing the push casting of streaming media resources.

[0058] It can be understood that during the process of streaming media screen projection, the receiving device can directly obtain media resources and play them independently. Therefore, operations on the sending device will not affect the playback of media resources by the receiving device. In this way, users can perform other application operations on the sending device while watching the projected video through the receiving device.

[0059] In addition to the Miracast and DLNA protocols mentioned above, Airplay is also a common screen projection protocol. In actual applications, Airplay supports both streaming media projection and mirroring.

[0060] 3. Reverse control

[0061] Reverse control refers to the technology in which the receiving device remotely controls the sending device through a reverse control protocol during screen projection.

[0062] Continuing with the example of the sending device being mobile phone 1 and the receiving device being smart TV 2, during the mirroring screen projection process, when mobile phone 1 and smart TV 2 negotiate the reverse control protocol, after the negotiation is completed, the user can reversely control the mobile phone by operating the smart TV, so that the screen display content of smart TV 2 and the screen display content of mobile phone 1 remain synchronized.

[0063] It should be noted that in the embodiment of the present application, the display device can simultaneously support mirroring screen projection (such as the screen projection method based on the Miracast protocol) and multi-channel streaming media screen projection (such as the screen projection method based on the DLNA protocol), among which mirroring screen projection supports the reverse control protocol.

[0064] The above section describes the terms or concepts involved in the embodiments of the present application. The following describes the technical solutions of the embodiments of the present application.

[0065] With the continuous development of terminal technology, multi-screen interaction technology is becoming more and more mature. For example, users can project the content displayed on a small-screen device to a large-screen device for display, obtaining a better display effect and viewing experience.

[0066] Currently, when projecting content displayed on a small-screen device to a large-screen device, when a user wants to switch viewing content, he or she needs to shift his or her attention to the small-screen device and perform corresponding operations, and only then will the viewing content be switched synchronously on the large-screen device.

[0067] The following takes the small-screen device as a mobile phone 1, the large-screen device as a smart TV 2, and the projection content as a live video as an example to illustrate the process of switching different projection content.

[0068] In some scenarios, mobile phone 1 and smart TV 2 are in the same WiFi network. The user opens the live broadcast application on mobile phone 1, selects the live broadcast content to watch, and casts the live broadcast content to smart TV 2 for display through the DLNA protocol.

[0069] For example, Figure 1 As shown in (a) of FIG, the mobile phone 1 can display an interface 11, which includes icons of multiple applications (APPs) installed on the mobile phone 1, such as an icon of a clock application, an icon of a calendar application, an icon of a gallery application, an icon 111 of a live broadcast application, etc. In response to a user clicking on the icon 111 of the live broadcast application, the mobile phone 1 displays Figure 1 The interface 12 shown in (b) is the application interface of the live broadcast application. For example, the live broadcast interface of Live Broadcast 1 is presented in the interface 12, and the "click to enter the live broadcast room" control 121 is displayed in the live broadcast interface. In response to the user's click operation on the "click to enter the live broadcast room" control 121, the mobile phone 1 displays Figure 1 The interface 13 shown in (c) is the live broadcast interface of the live broadcast 1. The interface 13 may include a "control panel" control 131. In response to the user's click operation on the "control panel" control 131, the mobile phone 1 displays Figure 1 The interface 14 shown in (d) in FIG. The interface 14 includes a control panel 141, which displays a "share" control, a "chat channel" control, a "gift special effect" control, a "forward" control, a "cast screen" control 142, etc. In response to the user's click operation on the "cast screen" control 142, the mobile phone 1 displays Figure 1 The interface 15 shown in (e) is a device selection interface. The interface 15 includes a device selection panel 151, on which multiple device options are displayed, such as a "Smart TV 2" device option 152 and a "Smart TV 3" device option. In response to the user clicking on the "Smart TV 2" device option 152, the mobile phone 1 can negotiate with the smart TV 2 based on the DLNA protocol to cast the screen. After the negotiation is successful, the mobile phone 1 can display Figure 1 The interface 16 shown in (f) is the screen projection completion interface. The text "Smart TV 2" is displayed in the interface 16 to prompt that the negotiation between the mobile phone 1 and the smart TV 2 is successful. And after the negotiation is successful, the mobile phone 1 sends the link address of the live broadcast 1 to the smart TV 2. The smart TV 2 obtains the media resources of the live broadcast 1 from the server according to the link address and plays the live broadcast 1. That is, the smart TV 2 now displays the following Figure 1 In the interface 17 shown in (g), the live broadcast interface 171 of the live broadcast 1 is displayed in the interface 17.

[0070] Next, if the user wants to cast the screen to watch other live content, the user needs to perform a series of operations on the phone. For example, Figure 2 As shown in (a) of FIG, the mobile phone 1 displays the interface 21, which is the live broadcast interface of the live broadcast 1. In response to the user's upward sliding operation on the interface 21, the mobile phone 1 displays the following Figure 2 The interface 22 shown in (b) is the live broadcast interface of Live Broadcast 2, that is, the content presented by mobile phone 1 switches from Live Broadcast 1 to Live Broadcast 2. Similar to interface 13, interface 22 includes a "control panel" control 221. In response to the user's click operation on the "control panel" control 221, mobile phone 1 displays Figure 2 Interface 23 shown in (c) of FIG. Similar to interface 14, interface 23 includes a control panel 231, and the control panel 231 displays a "cast screen" control 232. In response to the user clicking the "cast screen" control 232, the mobile phone 1 displays Figure 2 Interface 24 shown in (d) in FIG. Similar to interface 15, interface 24 is also a device selection interface. Interface 24 displays a device selection panel 241, on which multiple device options are displayed, such as a "Smart TV 2" device option 242 and a "Smart TV 3" device option. In response to the user clicking on the "Smart TV 2" device option 242, mobile phone 1 can negotiate with smart TV 2 based on the DLNA protocol to cast the screen. After the negotiation is successful, mobile phone 1 can display Figure 2 The interface 25 shown in (e) is the screen projection completion interface. The text "Smart TV 2" is displayed in the interface 25 to prompt that the negotiation between the mobile phone 1 and the smart TV 2 is successful. And after the negotiation is successful, the mobile phone 1 sends the link address of the live broadcast 3 to the smart TV 2. The smart TV 2 obtains the media resources of the live broadcast 3 from the server according to the link address and plays the live broadcast 3. That is, the smart TV 2 now displays the following Figure 2 In the interface 26 shown in (f), the live broadcast interface 261 of the live broadcast 3 is displayed in the interface 26, that is, the content presented on the smart TV 2 is switched from the live broadcast 1 to the live broadcast 2.

[0071] In other words, when switching screen projection content, the user's visual focus needs to switch from the smart TV to the mobile phone. The user must then perform the aforementioned operations on the mobile phone, such as opening the control panel, clicking the "Cast" control, and selecting a projection device. Only then will the content playing on the smart TV switch to the new media resource (such as Live 2). At this point, the user's visual focus must be shifted back to the smart TV to watch. This method causes the user's visual focus to repeatedly switch between the smart TV and the mobile phone, is cumbersome, and provides a poor user experience.

[0072] In other scenarios, mobile phone 1 and smart TV 2 are in the same WiFi network. In response to the user's operation, mobile phone 1 initiates mirroring screen projection to smart TV 2. Mobile phone 1 negotiates with smart TV 2 based on the Miracast protocol to cast the screen. After the negotiation is completed, mobile phone 1 can project its screen display content to smart TV 2 for display. Afterwards, in response to the user's operation, mobile phone 1 opens the live broadcast application and enters the live broadcast room of Live Broadcast 1. During the user's operation of mobile phone 1, the display content of smart TV 2 changes with the changes in the display content of mobile phone 1, that is, Live Broadcast 1 is playing on smart TV 2 at this time. In this scenario, if the user wants to cast the screen to watch other live broadcast content, the user needs to operate on mobile phone 1, such as performing an upward swipe operation on mobile phone 1, and the display content of mobile phone 1 switches from Live Broadcast 1 to Live Broadcast 2. At this time, the content played on smart TV 2 also switches from Live Broadcast 1 to Live Broadcast 2.

[0073] It can be seen that compared with the DLNA screen projection method, the operation of switching the projection content during the mirroring screen projection process is relatively simple. However, the user's visual focus still needs to be repeatedly switched between the smart TV and the mobile phone, resulting in a poor user experience.

[0074] In addition, for the mirroring screen projection method, the display content of the smart TV is always synchronized with the display content of the mobile phone. This will cause the user to be unable to perform other operations on the mobile phone when watching a certain media resource (such as live broadcast) on the screen, which will also affect the user experience.

[0075] In some embodiments, a smart TV can simultaneously support Miracast and DLNA casting. In response to user operations, mobile phone 1 establishes a Miracast casting connection with smart TV 2 and displays the mirrored image of mobile phone 1 on the smart TV. Furthermore, the user can select the media resource they want to watch on mobile phone 1 and push it to the smart TV for display via DLNA casting. However, in this method, DLNA casting and Miracast casting operate independently, and the media resources pushed by DLNA casting are unrelated to the mirrored image of mobile phone 1 displayed on the smart TV. Furthermore, similar to the other prior art methods mentioned above, this method still requires the user's visual focus to repeatedly switch between the smart TV and mobile phone, and the user needs to perform cumbersome operations on the mobile phone, resulting in a poor user experience. In summary, during the process of casting and playing media resources, if the user wants to switch between different media resources, they need to shift their attention to the mobile phone and perform a series of operations before the smart TV will synchronize the media resources. This method causes the user's visual focus to repeatedly switch between the smart TV and mobile phone, and the operation process is also relatively cumbersome, resulting in a poor user experience.

[0076] Furthermore, during the above-mentioned screen projection and playback of media resources, the smart TV cannot simultaneously play multiple projection contents from the same electronic device (such as mobile phone 1), and cannot meet the user's demand for simultaneously watching multiple projection contents from the same electronic device.

[0077] In view of this, an embodiment of the present application provides a screen projection method, which is applied to a display device, and the terminal device establishes a mirror projection connection with the display device, and the display device displays the screen content of the terminal device in projection area 1 (i.e., the first projection area hereinafter). In response to the split-screen projection instruction sent to the remote control, the display device takes a screenshot of the content currently displayed, and identifies and analyzes the screenshot image to generate a reverse control instruction. The display device sends a reverse control instruction to the terminal device through the mirror projection connection to control the terminal device to initiate a DLNA projection connection to the display device in response to the reverse control instruction, and sends a first projection instruction to the display device through the DLNA projection connection, wherein the first projection instruction carries a first resource address, and the first resource address is the resource address of the first media resource corresponding to the screen content currently displayed by the terminal device. When receiving the first projection instruction sent by the terminal device, the display device obtains the first media resource according to the first resource address, and plays the first media resource in projection area 2 (i.e., the second projection area hereinafter).

[0078] In an embodiment of the present application, in response to a split-screen projection instruction (such as a first control instruction or a second control instruction), the display device can take a screenshot, and by identifying and analyzing the screen content currently displayed by the terminal device in the screenshot image, generate and send a reverse control instruction to the terminal device to control the terminal device to switch the screen content. After the screen content after the terminal device switches is displayed in the projection area 1 of the display device, the display device can take a screenshot again, and based on the screenshot image, generate and send a reverse control instruction to the terminal device, and then reversely control the terminal device to establish a DLNA projection connection with the display device, and project the media resources currently displayed by the terminal device onto the display device. In other words, the user only needs to trigger the split-screen projection instruction, and the display device reversely controls the terminal device to project the media resources currently displayed by the terminal device onto the display device. In this way, during the projection process, the user does not need to operate on the terminal device, avoiding the user's visual focus from repeatedly switching between the display device and the terminal device, thereby improving the user experience.

[0079] Furthermore, in an embodiment of the present application, the mirrored content and other media resources of the terminal device can be displayed in split screen on the display device, so that the user can view the screen content of the terminal device on the display device while watching other media resources, without affecting the user's use of the terminal device.

[0080] The following describes the display device to which the screen projection method provided in the embodiment of the present application is applicable.

[0081] In the embodiments of the present application, a display device generally refers to a device capable of displaying images and processing data. For example, a display device includes but is not limited to smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0082] Figure 3 This is a schematic diagram of an operation scenario between a display device and a control device provided in an embodiment of the present application. Figure 3 As shown, the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device is used to perform human-computer interaction between the user and the display device 200.

[0083] In the implementation of this application, the control device can be a remote control with a specific pointing function. Here, the control device can be called a pointing remote control. The pointing remote control introduces a new way of interaction, breaking away from the operational limitations of traditional remote controls. The control method is like using a laser pen, which can accurately point to the target control without switching icon selection step by step. It is efficient and convenient.

[0084] In some embodiments, the pointing remote control has a pointing switch. When the pointing switch is turned on, the remote control enters pointing mode, allowing the user to point, slide, and perform other operations on a displayed object (such as a cursor). When the pointing switch is turned off, the remote control switches from pointing mode to traditional button mode, in which operations such as up / down / left / right / confirm buttons can be used.

[0085] In some embodiments, after turning on the pointing switch and entering the pointing mode, it can coexist with the traditional button mode. The user can point, slide, click, and perform other operations through the displayed object (such as the cursor), or use the confirmation key, as well as the up / down / left / right / volume keys and other button operations.

[0086] In some embodiments, the remote control is in low-power mode in traditional button mode. However, when the pointing switch is turned on and the remote control enters pointing mode, the remote control is no longer in low-power mode. Therefore, in some embodiments, the remote control can be equipped with a charging port, such as a Type-C port.

[0087] In some embodiments, the pointing remote control is a remote control device capable of detecting its own position and direction in three-dimensional space. The pointing remote control has a built-in gyroscope for detecting the rotation angle of the remote control and detecting the movement acceleration of the remote control. Through the sensor, the pointing remote control can determine its own pointing position in three-dimensional space in real time. The pointing position may include position coordinates and pointing direction, where the position coordinates are the physical position of the remote control relative to the display device (for example, the horizontal distance from the display device, the vertical height, etc.). The pointing direction is the pointing angle of the remote control (for example, the horizontal angle and vertical angle relative to the display device).

[0088] In some embodiments, after obtaining its own pointing position in three-dimensional space, the pointing remote control can send the pointing position data to the display device, so that the display device obtains the pointing position. The display device can perform screen coordinate conversion based on the pointing position, thereby converting the pointing position of the pointing remote control into the screen coordinate system, completing the mapping of the pointing position of the pointing remote control. At this time, the display device updates the position of the displayed object (such as the cursor) based on the calculated screen coordinates. When the user moves the remote control, the remote control's sensor will continue to detect changes in position and direction and send the data to the display device. The display device updates the position of the displayed object in real time, and the cursor moves in the pointing direction of the pointing remote control.

[0089] In some embodiments, a pointing remote control can control the display objects of the user interface. The cursor can be moved by pointing the remote control to different areas on the screen to achieve precise selection. The pointing remote control can switch between multiple modes. When the pointing switch of the pointing remote control is turned on, the pointing function is enabled, and operations such as pointing, sliding, and clicking can be performed on the displayed objects. When the pointing switch of the pointing remote control is turned off, the pointing function is turned off. When the pointing function of the pointing remote control is turned off, the pointing remote control can work in key mode, and the display device can be controlled through traditional key operations. The pointing remote control supports multiple operation modes such as jumping, tapping, circling, sliding, and dragging, making browsing content more convenient.

[0090] In some embodiments, when the pointing function of the pointing remote control is turned on, the confirmation button among the ordinary buttons in the pointing remote control can be shared in the pointing function.

[0091] The mobile terminal 300 can function as a control device for performing human-computer interaction between a user and the display device 200. The mobile terminal 300 can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, enabling connection and communication via a network communication protocol, enabling one-to-one control operations and data communication. Audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 for synchronized display.

[0092] like Figure 3 As shown, the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0093] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart TV, Internet Protocol television (IPTV), etc.

[0094] Figure 4 A hardware configuration block diagram of a display device provided in an embodiment of the present application.

[0095] In some embodiments, as Figure 4 As shown, the display device 200 may include at least one of a tuner and demodulator 210, a communicator (also referred to as a communication device) 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280.

[0096] In some embodiments, the controller 250 may include at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processor, and a power processor, and first to nth interfaces for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in a memory. The controller 250 controls the overall operation of the display device 200.

[0097] In some embodiments, the display 260 includes a display component for presenting images and a driver component for driving image display. The display 260 is configured to receive image signals output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces.

[0098] Exemplarily, the display 260 may be a liquid crystal display, an OLED display, a projection display, or a projection device and a projection screen.

[0099] In some embodiments, the communicator 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with multiple communicators 220 depending on the supported communication methods. For example, if the display device 200 supports wireless network communication, the display device 200 may be provided with a communicator 220 including WiFi functionality. If the display device 200 supports Bluetooth connection communication, the display device 200 may also be provided with a communicator 220 including Bluetooth functionality.

[0100] The communicator 220 can establish a communication connection between the display device 200 and an external device or server 400 via a wireless or wired connection. A wired connection can connect the display device 200 to an external device via a data cable, an interface, or other components. A wireless connection can connect the display device 200 to an external device via a wireless signal or wireless network. The display device 200 can establish a connection with an external device directly or indirectly through a gateway, router, or connection device.

[0101] Exemplarily, the display device 200 can obtain user instructions sent by the control device through the communicator 220, and implement the screen projection method provided in the embodiment of the present application according to the user instructions.

[0102] In some embodiments, detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 may include a light receiver, such as a sensor for collecting ambient light intensity; or an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or a sound collector, such as a microphone, for receiving external sounds.

[0103] In some embodiments, when implementing the screen projection method provided in the embodiments of the present application, the operations performed by the display device 200 can be implemented by the controller 250.

[0104] In some embodiments, the display device 200 may run an operating system to facilitate user interaction. An operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system may control the display device to provide a user interface. For example, the operating system may directly control the display device to provide a user interface, or it may provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0105] It should be noted that the above operating system may be a native operating system based on a specific operating platform, or a third-party operating system customized based on a specific operating platform, or an independent operating system specially developed for a display device.

[0106] Figure 5 A schematic diagram of the software system architecture of a display device provided in an embodiment of the present application.

[0107] For example, Figure 5 As shown, the software system at least includes: an application layer (also referred to as the application layer), an application framework layer (also referred to as the framework layer), a system runtime layer and a kernel layer (kernel).

[0108] The application layer is used to deploy TV-related functions. The application program layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with users based on these applications. In some embodiments, the application program layer runs at least one application. These applications can be native operating system applications, such as user interface applications, TV settings applications, and TV services, or applications developed by third-party developers. In specific implementations, the application packages in the application program layer are not limited to the examples above.

[0109] For example, Figure 5 As shown, the application layer may include a fusion screen projection application (which may be referred to as a fusion screen projection application). The fusion screen projection application has a split screen projection function. The fusion screen projection application may include a screen projection playback interface, a screen projection service, a network request module, and a view display module.

[0110] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0111] like Figure 5 As shown, the framework layer may include a view system, managers, content providers, etc. Among them, the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc.

[0112] The manager includes at least one of the following modules: an activity manager, which interacts with all running activities in the system; a location manager, which provides system services or applications with access to the system's location services; a package manager, which retrieves various information related to application packages currently installed on the device; a notification manager, which controls the display and clearing of notification messages; and a window manager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0113] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling application exit, opening, and back. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling changes in display windows, such as shrinking, shaking, or distorting the display window.

[0114] In some embodiments, the framework layer also includes player and system components.

[0115] like Figure 5 As shown, in some embodiments, the system runtime layer (i.e., the system library layer) can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0116] like Figure 5 As shown, in some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multi-tasking, and memory management. Figure 5 As shown, the kernel layer can be configured with hardware drivers, and the drivers included in the kernel layer can be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0117] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0118] Figure 6This is a schematic diagram of the software system architecture of an intelligent agent provided in an embodiment of the present application. For example, Figure 6 As shown, the intelligent agent includes a management service (Agent Processor) module, an identification service (Client) module, a page matching (Page Match Alg) module and a path planning (Route Plan) module.

[0119] The following examples of this application will be Figure 4 and Figure 5 Taking the display device of the shown structure as an example, the screen projection method provided in the embodiment of the present application is specifically explained in combination with the accompanying drawings and application scenarios.

[0120] Before introducing the screen projection method provided in the embodiment of the present application, the application scenario of the screen projection method provided in the present application is first described.

[0121] Take the display device as smart TV 2 and the mobile terminal as mobile phone 1 as an example. Figure 7 As shown, the fusion screen projection APP of smart TV 2 has a split screen projection function (also known as multi-screen projection mode). The split screen projection function refers to the function of smart TV 2 supporting multi-channel projection content push at the same time. Figure 7As shown, after the split-screen live broadcast function of Smart TV 2 is turned on, first, Mobile Phone 1 initiates mirroring to Smart TV 2. After Smart TV 2 and Mobile Phone 1 negotiate for mirroring, Mobile Phone 1 connects to Smart TV 2 and sends the screen content displayed by Mobile Phone 1 to Smart TV 2. Smart TV 2 then displays the screen content of Mobile Phone 1 in projection area 1. Then, in response to the split-screen projection command sent via the pointing remote control, Smart TV 2 takes a screenshot of the currently displayed content, obtains a screenshot image, and sends the screenshot image to the agent. The agent sends a recognition request to the recognition model, which carries the screenshot image. The recognition model recognizes the screenshot image and returns the recognition result to the agent. The agent analyzes the recognition result, generates and sends touch action information to Smart TV 2. Smart TV 2 parses and converts the touch action information, generates a reverse control command, and sends the reverse control command to Mobile Phone 1 to control Mobile Phone 1 to execute the reverse control command. Mobile phone 1 responds to the reverse control instruction, displays the new screen content, and sends the new screen content to smart TV 2 through the mirroring connection, so that smart TV 2 can synchronously display the updated screen content of mobile phone 1 in the projection area 1. After displaying the updated screen content of mobile phone 1, the aforementioned process of issuing reverse control instructions is repeated until mobile phone 1 is reversely controlled to initiate a DLNA projection instruction to smart TV 2. The DLNA projection instruction carries the resource address of the media resource corresponding to the screen content currently displayed by mobile phone 1. After receiving the DLNA projection instruction, smart TV 2 obtains the media resource corresponding to the screen content currently displayed by mobile phone 1 according to the resource address, and plays the media resource corresponding to the screen content currently displayed by mobile phone 1 in projection area 2, thereby realizing split-screen projection of the screen content (mirrored image) of mobile phone 1 and media resources.

[0122] It should be noted that the split-screen projection method provided in the embodiment of the present application can be used in live broadcast scenarios, that is, split-screen projection of live broadcast media resources, so the above-mentioned split-screen projection function can also be called a split-screen live broadcast function.

[0123] It should also be noted that the intelligent agent and recognition model can be deployed in the cloud or locally on the smart TV. In addition, the intelligent agent and recognition model can be deployed independently or run in an integrated manner.

[0124] The following is an explanation of the process of enabling the split-screen projection function with reference to the accompanying drawings.

[0125] In some examples, a fusion screen projection APP is installed in the smart TV, and the fusion screen projection APP can provide an entrance to the split-screen projection function to start the split-screen projection function.

[0126] Figure 8 A schematic diagram of the interactive process of activating the split-screen projection function provided in an embodiment of the present application.

[0127] For example, the display device is a smart TV 2, and the split-screen projection function is a split-screen live broadcast function. Figure 8 As shown in (a) of FIG, the smart TV 2 can display an interface 71, which is an application interface of the smart TV 2. The interface 71 may include icons of multiple APPs installed in the smart TV 2, such as the icon of the Juhaokan APP and the icon 711 of the Fusion Cast APP. In response to a click operation on the icon of the Fusion Cast APP, the smart TV 2 can display Figure 8 Interface 72 shown in (b) is the application interface of the integrated screen casting APP. Exemplarily, the network name and device name can be displayed in interface 72. Among them, the network name is used to indicate the name of the network in which the smart TV is currently located, such as the network name is wifi123_5G. The device name refers to the name of Smart TV 2, such as the device name is Smart TV 2. The interface 72 also displays the animation effect of "Waiting for screen casting" to prompt that the screen casting connection is not currently established. The interface 72 also includes a "Taste Lab" control 721, and the "Taste Lab" control 721 is the entrance to the split-screen live broadcast function.

[0128] After the smart TV 2 displays the interface 72, in response to the click operation on the "Taste Lab" control 721, the smart TV 2 turns on the split-screen live broadcast function and displays the following: Figure 8 Interface 73 shown in (c) of the figure. Interface 73 is a guide page for the split-screen live broadcast function, which is used to guide the user to perform split-screen live broadcast operations. For example, interface 73 may include instructions for the split-screen live broadcast function, such as "After mirroring your phone to the large screen, you can split the screen to play the live broadcast when watching the live broadcast on your phone." Interface 73 also includes the steps for starting split-screen live broadcast and the corresponding operation guidance screens for each step.

[0129] For example, step 1: the mobile phone connects to the current network, such as "wifi123_5G", and selects the current device, such as "Smart TV 2"; accordingly, the screen casting guide screen 731 is presented in the interface 73, and the device option of "Smart TV 2" in the screen casting guide screen 731 is highlighted to prompt the initiation of mirror casting to "Smart TV 2". Step 2, open the live broadcast channel of the mobile phone (such as opening the live broadcast APP and entering the live broadcast room), and trigger the split-screen casting instruction by pointing the remote control, such as dragging the mirror screen of the mobile phone to the right by pointing the remote control; accordingly, the drag split screen guide screen 732 is presented in the interface 73 to prompt the selection of the specified button and drag the live broadcast to the right. Step 3, the mirror casting content and the live broadcast content of the mobile phone are displayed in split screen, and accordingly, a schematic diagram of the split-screen live broadcast interface 733 is presented in the interface 73 to prompt the display effect of the split-screen live broadcast. Follow the steps in interface 73 to achieve split-screen live broadcast.

[0130] In some examples, interface 72 also includes options for different screen casting tutorials, such as "Cast from Juhaokan," "Cast from Phone or Tablet," "Cast from Computer," and "Cast via Wired." Through the Cast Tutorial option, you can view the corresponding casting tutorial to guide the user in initiating a screen cast.

[0131] The above is a schematic description of the process of enabling the split-screen projection function. The following is a schematic description of the split-screen projection process.

[0132] The embodiments of this application provide a variety of split-screen projection triggering methods, such as drag split-screen method and one-click split-screen projection method. The following uses these two triggering methods as examples to explain the split-screen projection process in detail.

[0133] 1. Drag split screen mode

[0134] 1.1. The process of mirroring the screen content of mobile phone 1 to smart TV 2

[0135] Taking live broadcast media resources as an example, mobile phone 1 opens the live broadcast app and enters the live broadcast room of live broadcast 1. Then mobile phone 1 initiates mirror screen projection to smart TV 2. After mobile phone 1 and smart TV 2 negotiate for screen projection, smart TV 2 displays the mirror screen projection content in split-screen area 1.

[0136] For example, Figure 9 As shown in (a) of FIG, the mobile phone 1 can display an interface 81, which is the desktop of the mobile phone 1. The desktop 1 may include icons of multiple APPs installed in the mobile phone 1, such as an icon of a clock APP, an icon of a calendar APP, an icon of a weather APP, an icon 811 of a live broadcast APP, etc. In response to a user clicking on the icon 811 of the live broadcast application, the mobile phone 1 displays Figure 9 The interface 82 shown in (b) is the main interface of the live broadcast APP. For example, the interface 82 includes multiple page options for switching different pages, such as the "Home" option, the "Friends" option, the "Message" option and the "Me" option, wherein the "Home" option is highlighted, indicating that the current interface 82 is the main interface of the live broadcast APP. The interface 82 also includes multiple theme tags for switching different themes, such as the "Experience" tag, the "Qingdao" tag, the "Follow" tag, the "Mall" tag and the "Recommendation" tag. In addition, the live broadcast screen of Live Broadcast 1 is also presented in the interface 82, and the "Click to enter the live broadcast room" control 821 is displayed in the live broadcast screen of Live Broadcast 1. In response to the user's click operation on the "Click to enter the live broadcast room" control 821, the mobile phone 1 displays Figure 9 Interface 83 shown in (c) is the live broadcast interface of live broadcast 1.

[0137] Afterwards, the mobile phone 1 initiates mirroring of the screen to the smart TV 2. For example, in response to the user's pull-down operation on the interface 83, the mobile phone 1 may display Figure 9 The interface 84 shown in (d) is the control center interface. The interface 84 includes multiple function control switches for turning on or off the corresponding functions of the mobile phone 1. For example, the interface 84 may include a "mute" control switch, a "do not disturb" control switch, a "flight mode" control switch, a "wireless screen casting" control switch 841, etc. Among them, the "wireless screen casting" control switch 841 is used to turn on the mirror screen casting function of the mobile phone. In response to the user's click operation on the "wireless screen casting" control switch 841, the mobile phone 1 turns on the mirror screen casting function and searches for the screen casting target device in the same local area network as the mobile phone 1. When the mobile phone 1 searches for the smart TV 2 in the same local area network as the mobile phone 1, the mobile phone 1 can display Figure 9 The interface 85 shown in (e) is a device selection interface. The interface 85 includes a device selection panel 851, on which multiple device options are displayed, such as a "Smart TV 2" device option 852 and a "Smart TV 3" device option. In response to the user clicking on the "Smart TV 2" device option 852, the mobile phone 1 establishes a wireless connection with the smart TV 2, and exchanges information to negotiate whether the mobile phone 1 and the smart TV 2 support screen projection. Figure 9 As shown in (f), during the negotiation process between the mobile phone 1 and the smart TV 2, the smart TV 2 can display an interface 86, and a prompt message such as "screen casting" can be displayed in the interface 86.

[0138] After mobile phone 1 establishes a connection with smart TV 2 and completes the negotiation, mobile phone 1 can display Figure 9 In the interface 87 shown in (g), the mobile phone 1 cancels the display device selection panel, and the content displayed on the mobile phone 1 is projected onto the smart TV 2. For example, Figure 9 As shown in (h), the smart TV 2 can display an interface 88, which is a split-screen live broadcast interface. The interface 88 can include two projection areas, such as projection area 1 and projection area 2, which are respectively used to display the multi-channel projection content of the mobile phone 1. Among them, the mirror projection content 881 of the mobile phone 1 is displayed in the projection area 1, that is, the live broadcast interface of the live broadcast 1. The projection area 2 can be used to display other projection content of the mobile phone 1, that is, the projection content pushed by the mobile phone 1 through the DLNA protocol in response to the reverse control command sent by the smart TV 2, so that the split-screen display of the multi-channel projection content of the mobile phone 1 can be achieved.

[0139] Of course, in some examples, mobile phone 1 can also initiate mirroring screen projection to smart TV 2 first. After mobile phone 1 and smart TV 2 successfully establish a mirroring screen projection connection, in response to the user's operation, mobile phone 1 opens the live broadcast APP and enters the live broadcast room of live broadcast 1. At this time, the content in the projection area 1 of smart TV 2 is synchronously updated to the live broadcast interface of live broadcast 1.

[0140] 1.2. Drag split screen process

[0141] During the mirroring process between mobile phone 1 and smart TV 2, the user can point the remote control and drag the mirrored screen of mobile phone 1 displayed in projection area 1 to project the media resources currently displayed on mobile phone 1 to projection area 2 via DLNA projection, thereby realizing split-screen projection of the mirrored screen of mobile phone 1 and other media resources.

[0142] For example, Figure 10 As shown in (a), after the mobile phone 1 establishes a mirroring connection with the smart TV 2, the smart TV 2 can display interface 91, which is a split-screen live broadcast interface (i.e., the first interface below). Similar to interface 88, interface 91 includes a projection area 1 and a projection area 2. The projection area 1 displays the mirrored content of the mobile phone 1, i.e., the live broadcast interface A1 of the live broadcast 1. The projection area 2 can display prompt information, such as "Please drag the live broadcast here to start the screen projection", and display a split-screen live broadcast guide animation 911 to guide the user to drag the split screen operation.

[0143] After the smart TV 2 enters the interface 91, the user points the remote control toward the smart TV 2 to display a cursor 912 in the interface 91. The user adjusts the position of the cursor 912 on the screen (i.e., the display area) of the smart TV 2 by moving the remote control. When the cursor 912 is located in the projection area 1, the smart TV 2 can display Figure 10 Interface 92 shown in (b) in the figure. Similar to interface 91, interface 92 also includes projection area 1 and projection area 2. The projection area 1 displays the mirror content of mobile phone 1, that is, the live broadcast interface A2 of live broadcast 1. Unlike interface 91, in interface 92, a drag control 921 is also displayed in the projection area 1, and the drag control 921 is used to trigger one-key projection (that is, split-screen live broadcast). Exemplarily, the drag control 921 is located at the right edge of the projection area 1, and the drag control 921 is suspended on the projection content (that is, the live broadcast interface A2). In addition, the drag control 921 can also be marked with the text "Drag right here to split the screen" to prompt the dragging control 921 to drag the live broadcast interface of live broadcast 1 (that is, the currently displayed mirror content) to the right to project the currently displayed mirror content (that is, live broadcast 1) to the projection area 2 for split-screen display.

[0144] It is understandable that the live broadcast interface A2 may be different from the live broadcast interface A1. For example, the play time corresponding to the live broadcast interface A2 may be later than the play time corresponding to the live broadcast interface A1.

[0145] Next, the user continues to adjust the position of the cursor 912 on the screen (i.e., display area) of the smart TV 2 by moving the pointing remote control. When the cursor 912 is located on the drag control 921, the smart TV 2 can display Figure 10 In the interface 93 shown in (c), the live broadcast interface A3 of the live broadcast 1 is presented in the projection area 1. Moreover, unlike the interface 92, in the interface 93, the drag control 921 switches from the first display effect (such as the gray effect) to the second display effect (such as the highlight effect).

[0146] When the cursor 912 is located on the drag control 921, the user presses the designated button of the remote control (such as the OK button) to drag the current mirrored content (i.e., the live broadcast interface A3 of Live Broadcast 1) to the right. After the current mirrored content is dragged to the projection area 2, the designated button of the remote control is released, that is, the instruction to drag the current mirrored content from the projection area 1 to the projection area 2 (also referred to as the first control instruction) is sent to the smart TV 2 through the remote control. In response to the instruction to drag the current mirrored content from the projection area 1 to the projection area 2, the smart TV 2 takes a screenshot of the content currently displayed, identifies and analyzes the screenshot image, generates a reverse control instruction, and sends the reverse control instruction to the mobile phone 1 to reversely control the mobile phone 1 to execute the reverse control instruction. In response to the reverse control instruction, the mobile phone 1 displays the new screen content and sends the new screen content displayed by the mobile phone 1 to the smart TV 2 through the mirror projection connection, so that the smart TV 2 synchronously displays the updated screen content of the mobile phone 1 in the projection area 1. After displaying the updated screen content of mobile phone 1, the aforementioned process of issuing reverse control instructions is repeated until mobile phone 1 is reversely controlled to initiate a DLNA screen casting connection to smart TV 2, and through this DLNA screen casting connection, DLNA screen casting instruction 1 (i.e., the first screen casting instruction below) is sent to smart TV 2. DLNA screen casting instruction 1 carries the resource address 1 of live broadcast 1 corresponding to the screen content currently displayed by mobile phone 1. After receiving DLNA screen casting instruction 1, smart TV 2 obtains live broadcast 1 according to resource address 1. After obtaining live broadcast 1, smart TV 2 can display Figure 10In the interface 94 shown in (d), compared with the interface 93, the live broadcast 1 is played in the projection area 2 of the interface 94, that is, the live broadcast interface A5 of the live broadcast 1 is displayed. At this time, the mirror content corresponding to the mobile phone 1 (that is, the live broadcast interface A4 of the live broadcast 1) is still presented in the projection area 1, thereby realizing the split-screen projection of the screen content (mirror picture) of the mobile phone 1 and other media resources, and completing the media resources corresponding to the screen content currently displayed by the mobile phone 1, through DLNA projection, and projecting them to the split-screen display in the projection area 2 on the right.

[0147] It can be understood that after the split-screen live broadcast is completed, the mirrored content presented in the projection area 1 and the DLNA projection content presented in the projection area 2 are both Live Broadcast 1. Since the DLNA projection content (i.e., the live broadcast interface A5 of Live Broadcast 1 currently displayed in the projection area 2) is related to the mirrored image displayed in the projection area 1 when the DLNA projection is triggered, the live broadcast interface A5 in the projection area 2 and the live broadcast interface A4 in the projection area 1 can be the same or different.

[0148] The above is a schematic description of the process of split-screen projection of live broadcast 1 in projection area 2. The following is a detailed description of the jump process from interface 93 to interface 94.

[0149] like Figure 11 As shown in (a) in the figure, the smart TV 2 displays the live broadcast interface A3 of the live broadcast 1 in the projection area 1. At this time, the smart TV 2 takes a screenshot of the content currently displayed, obtains the screenshot image 1, and identifies and analyzes the screenshot image 1, determines that the interface jump position of the currently displayed live broadcast interface A3 is the live broadcast room interface, and determines that the control to be triggered is the "control panel" control A31, and then generates the reverse control instruction 1 corresponding to the "control panel" control A31, that is, click the reverse control instruction of the "control panel" control A31. The smart TV 2 sends the reverse control instruction 1 to the mobile phone 1, and the mobile phone 1 responds to the reverse control instruction 1 and displays Figure 11 The interface 101 shown in (b) is the control panel interface. Moreover, the mobile phone 1 sends the screen content corresponding to the interface 101 to the smart TV 2 through the mirroring screen connection. Figure 11 As shown in (c), the smart TV 2 receives the screen content corresponding to the interface 101 and displays the projection interface A6 in the projection area 1. The projection interface A6 corresponds to the interface 101 currently displayed on the mobile phone 1.

[0150] After the smart TV 2 displays the screen projection interface A6, the smart TV 2 takes a screenshot of the content currently displayed, obtains the screenshot image 2, and identifies and analyzes the screenshot image 2, determines that the interface jump position of the currently displayed screen projection interface A6 is the control panel interface, and determines that the control to be triggered is the "screen projection" control A61, and then generates the reverse control instruction 2 corresponding to the "screen projection" control A61, that is, click the reverse control instruction of the "screen projection" control A61. The smart TV 2 sends the reverse control instruction 2 to the mobile phone 1, and the mobile phone 1 responds to the reverse control instruction 2 and displays Figure 11 The interface 102 shown in (d) is a device selection interface. Moreover, the mobile phone 1 sends the screen content corresponding to the interface 102 to the smart TV 2 through the mirroring screen connection. Figure 11 As shown in (e), the smart TV 2 receives the screen content corresponding to the interface 102 and displays the projection interface A7 in the projection area 1. The projection interface A7 corresponds to the interface 102 currently displayed on the mobile phone 1.

[0151] After the smart TV 2 displays the projection interface A7, the smart TV 2 takes a screenshot of the content currently displayed, obtains the screenshot image 3, and identifies and analyzes the screenshot image 3, determines that the interface jump position of the currently displayed projection interface A7 is the device selection interface, and determines that the control to be triggered is the "smart TV 2" option A71, and then generates the reverse control instruction 3 corresponding to the "smart TV 2" option A71, that is, click the reverse control instruction of the "smart TV 2" option A71. Smart TV 2 sends the reverse control instruction 3 to mobile phone 1, and mobile phone 1 responds to the reverse control instruction 3 and displays Figure 11 The interface 103 shown in (f) is the screen projection completion interface. Moreover, the mobile phone 1 sends the screen content corresponding to the interface 103 to the smart TV 2 through the mirror projection connection. Figure 11 As shown in (g), the smart TV 2 receives the screen content corresponding to the interface 103 and displays the projection interface A8 in the projection area 1. The projection interface A8 corresponds to the interface 103 currently displayed on the mobile phone 1.

[0152] After the smart TV 2 displays the screen projection interface A8, the smart TV 2 takes a screenshot of the content currently displayed, obtains the screenshot image 4, and identifies and analyzes the screenshot image 4 to determine that the interface jump position of the currently displayed screen projection interface A8 is the screen projection completion interface, that is, the smart TV 2 determines that the live broadcast 1 currently displayed by the mobile phone 1 has been projected to the smart TV 2 via DLNA. Afterwards, the smart TV 2 can send a preset counter-control instruction 4 to the mobile phone 1, and the counter-control instruction 4 is used to control the mobile phone 1 to cancel the display of the screen projection pop-up window 1031. The mobile phone 1 responds to the counter-control instruction 4, and the mobile phone 1 cancels the display of the screen projection pop-up window 1031 and returns Figure 11The interface 104 shown in (h) is the live broadcast room interface of the live broadcast 1. Moreover, the mobile phone 1 sends the screen content corresponding to the interface 104 to the smart TV 2 through the mirroring screen connection. Figure 10 As shown in (d), the smart TV 2 receives the screen content corresponding to the interface 104 and displays the projection interface A4 in the projection area 1. The projection interface A4 corresponds to the interface 104 currently displayed on the mobile phone 1.

[0153] Among them, the above-mentioned process of identifying and analyzing the screenshot image, determining the interface jump position of the currently displayed projection interface, and generating the corresponding reverse control instructions can be found in the relevant description of the knowledge graph below, which will not be repeated here.

[0154] After mobile phone 1 associates its currently displayed screen content with live broadcast 1 and projects it to projection area 2 of smart TV 2 via a DLNA projection connection for split-screen display, smart TV 2 can send a pre-set reverse control command to mobile phone 1 to control mobile phone 1 to switch the currently displayed screen content.

[0155] For example, the smart TV 2 displays Figure 12 After the live broadcast interface A4 of the live broadcast 1 shown in (a) of the figure is displayed, the smart TV 2 can send a reverse control instruction 5 to the mobile phone 1, and the reverse control instruction 5 is used to control the mobile phone 1 to execute the upward sliding instruction. In response to the reverse control instruction 5 (such as the upward sliding instruction) sent by the smart TV 2, the mobile phone 1 switches the currently displayed screen content, that is, the mobile phone 1 displays Figure 12 The interface 1101 shown in (b) includes the live broadcast room screen of the live broadcast 2. In addition, the mobile phone 1 sends the screen content corresponding to the interface 1101 to the smart TV 2 through the mirroring screen connection. Figure 12 As shown in (c) in the figure, the smart TV 2 receives the screen content corresponding to the interface 1101 and displays the projection interface A9 in the projection area 1. The projection interface A9 corresponds to the interface 1101 currently displayed on the mobile phone 1, that is, the smart TV 2 displays the live broadcast 2 in the projection area 1. In this way, the user can watch the live broadcast 1 on the smart TV 2 while also watching other screen contents on the mobile phone 1.

[0156] In some examples, during the process of mirroring the screen between mobile phone 1 and smart TV 2, the user can point the remote control and drag the mirrored screen of mobile phone 1 displayed in projection area 1 to project the media resources currently displayed on mobile phone 1 (i.e. live broadcast 2) to projection area 3 through DLNA projection, thereby realizing split-screen projection of the mirrored screen of mobile phone 1 and multiple other media resources.

[0157] For example, Figure 13As shown in (a) in FIG, the smart TV 2 can display the interface 1201, and display the screen projection interface A9 in the screen projection area 1 of the interface 1201, that is, the live screen of the live broadcast 2 corresponding to the screen content currently displayed on the mobile phone 1. After the smart TV 2 enters the interface 1201, the user points the remote control toward the smart TV 2 to display the cursor 12011 in the interface 1201. The user adjusts the position of the cursor 12011 on the screen of the smart TV 2 by moving the remote control. When the cursor 12011 is in the screen projection area 1, the smart TV 2 can display Figure 13 Interface 1202 shown in (b) in the figure. Similar to interface 1201, interface 1202 also includes projection area 1 and projection area 2. Projection area 1 displays the mirror content of mobile phone 1, that is, the projection interface A10 corresponding to the live screen of live broadcast 2. Unlike interface 1201, in interface 1202, a drag control 12021 is also displayed in projection area 1. Drag control 12021 is used to trigger one-click projection (i.e. split-screen live broadcast). Exemplarily, drag control 12021 is located at the right edge of projection area 1, and drag control 12021 is suspended on the projection content (i.e., projection interface A10). In addition, the text "Drag right here to split screen" can also be marked on drag control 12021 to prompt dragging the projection interface A10 corresponding to live broadcast 2 to the right (i.e., the currently displayed mirror content) to project the currently displayed mirror content (i.e., live broadcast 2) to projection area 3 (i.e., the third projection area below) for split-screen display.

[0158] It is understood that the projection screen A10 may be different from the projection screen A9. For example, the playback time corresponding to the projection screen A10 may be later than the playback time corresponding to the projection screen A9.

[0159] Next, the user continues to adjust the position of the cursor 12011 on the screen of the smart TV 2 by moving the remote control. When the cursor 12011 is located on the drag control 12021, the smart TV 2 can display Figure 13 In interface 1203 shown in (c), projection interface A11 is displayed in projection area 1. Projection interface A11 is the live broadcast interface of live broadcast 2. In addition, unlike interface 1202, in interface 1203, dragging control 12021 switches from the first display effect (e.g., gray effect) to the second display effect (e.g., highlight effect).

[0160] When the cursor 12011 is located on the drag control 12021, the user presses the designated button (such as the OK button) on the remote control and drags the current mirror content (i.e., the screen projection interface A11) to the right. At this time, the smart TV 2 can display Figure 13The interface 1204 shown in (d) is different from the interface 1203 in that the interface 1204 includes the projection area 1, the projection area 2 and the projection area 3, wherein the projection area 1 displays the projection interface A12 (i.e., the mirror image of the mobile phone 1), the projection area 2 displays the live screen of the live broadcast 1, and the projection area 3 can display prompt information, such as "Please drag the live broadcast here to start the screen projection", and display the split-screen live broadcast guide animation 12041 to guide the user to drag the split screen operation.

[0161] When the current mirrored content is dragged to the projection area 3 by pointing to the remote control, and then the designated button pointing to the remote control is released, the instruction to drag the current mirrored content from the projection area 1 to the projection area 3 (also referred to as the second control instruction) is sent to the smart TV 2 by pointing to the remote control. In response to the instruction to drag the current mirrored content from the projection area 1 to the projection area 3, the smart TV 2 takes a screenshot of the content currently displayed, identifies and analyzes the screenshot image, generates a reverse control instruction, and sends the reverse control instruction to the mobile phone 1 to reversely control the mobile phone 1 to execute the reverse control instruction. In response to the reverse control instruction, the mobile phone 1 displays the new screen content, and sends the new screen content displayed by the mobile phone 1 to the smart TV 2 through the mirror projection connection, so that the smart TV 2 synchronously displays the updated screen content of the mobile phone 1 in the projection area 1. After displaying the updated screen content of mobile phone 1, the aforementioned process of issuing reverse control instructions is repeated until mobile phone 1 is reversely controlled to initiate another DLNA screen casting connection to smart TV 2, and through this DLNA screen casting connection, DLNA screen casting instruction 2 is sent to smart TV 2. DLNA screen casting instruction 2 carries the resource address 1 of live broadcast 2 corresponding to the screen content currently displayed by mobile phone 1. After receiving DLNA screen casting instruction 2, smart TV 2 obtains live broadcast 2 according to resource address 2. After obtaining live broadcast 2, smart TV 2 can display Figure 13 In the interface 1205 shown in (e), compared with the interface 1204, the live broadcast 2 is played in the projection area 3 of the interface 1205. At this time, the mirror content corresponding to the mobile phone 1 is still presented in the projection area 1, thereby realizing the split-screen projection of the screen content of the mobile phone 1 (mirror picture) and multiple other media resources (such as live broadcast 1 and live broadcast 2), and completing the projection of the media resources corresponding to the screen content currently displayed by the mobile phone 1 to the split-screen display in the projection area 3 on the right through another DLNA projection.

[0162] It can be understood that after the split-screen live broadcast is completed, the mirrored content presented in the projection area 1 and the DLNA projection content presented in the projection area 3 are both Live Broadcast 2. Since the DLNA projection content (i.e., the live broadcast interface of Live Broadcast 2 currently displayed in the projection area 3) is related to the mirrored screen displayed in the projection area 1 when the other DLNA projection is triggered, the projection interface in the projection area 3 can be the same as or different from the projection interface in the projection area 1.

[0163] Of course, after mobile phone 1 associates the live broadcast 2 with the currently displayed screen content and projects it to the projection area 3 of smart TV 2 through another DLNA projection connection for split-screen display, smart TV 2 can send a pre-set reverse control command to mobile phone 1 to control mobile phone 1 to switch the currently displayed screen content.

[0164] For example, the smart TV 2 displays Figure 13 After the interface 1205 shown in (e) is displayed, the smart TV 2 can send a reverse control instruction 6 to the mobile phone 1. The reverse control instruction 6 is used to control the mobile phone 1 to execute the upward sliding instruction. In response to the reverse control instruction 6 (such as the upward sliding instruction) sent by the smart TV 2, the mobile phone 1 switches the currently displayed screen content, that is, the mobile phone 1 displays Figure 13 The interface 1206 shown in (f) in FIG. 1 includes the live broadcast room screen of Live Broadcast 3. Moreover, the mobile phone 1 sends the screen content corresponding to the interface 1206 to the smart TV 2 through the mirroring screen connection. After the smart TV 2 receives the screen content corresponding to the interface 1206, it displays Figure 13 Interface 1207 shown in (g) is different from interface 1205 in that a projection interface corresponding to interface 1206 currently displayed on mobile phone 1 is displayed in projection area 1 of interface 1207, i.e., smart TV 2 displays the live image of Live Broadcast 3 in projection area 1. In this way, the user can watch Live Broadcast 1 and Live Broadcast 2 on smart TV 2 while also watching other screen contents of mobile phone 1, such as Live Broadcast 3.

[0165] The above is a schematic description of the process of split-screen projection of live broadcast 2 in projection area 3. The following is a detailed description of the process of jumping from interface 1206 to interface 1207.

[0166] like Figure 14 As shown in (a) in the figure, the smart TV 2 displays the interface 1204, and displays the projection interface A13 in the projection area 1 of the interface 1204. The projection interface A13 corresponds to the live broadcast interface of the live broadcast 2 currently displayed by the mobile phone 1. At this time, the smart TV 2 takes a screenshot of the content currently displayed, obtains the screenshot image 5, and identifies and analyzes the screenshot image 5, determines that the interface jump position of the currently displayed projection interface A13 is the live broadcast room interface, and determines that the control to be triggered currently is the "control panel" control A131, and then generates the reverse control instruction 7 corresponding to the "control panel" control A131, that is, click the reverse control instruction of the "control panel" control A131. The smart TV 2 sends the reverse control instruction 7 to the mobile phone 1, and the mobile phone 1 responds to the reverse control instruction 7 and displays Figure 14 The interface 1301 shown in (b) is the control panel interface. Moreover, the mobile phone 1 sends the screen content corresponding to the interface 1301 to the smart TV 2 through the mirroring screen connection. Figure 14As shown in (c), the smart TV 2 receives the screen content corresponding to the interface 1301 and displays the projection interface A14 in the projection area 1. The projection interface A14 corresponds to the interface 1301 currently displayed on the mobile phone 1.

[0167] After the smart TV 2 displays the screen projection interface A14, the smart TV 2 takes a screenshot of the content currently displayed, obtains the screenshot image 6, and identifies and analyzes the screenshot image 6, determines that the interface jump position of the currently displayed screen projection interface A14 is the control panel interface, and determines that the control to be triggered is the "screen projection" control A141, and then generates the reverse control instruction 8 corresponding to the "screen projection" control A141, that is, click the reverse control instruction of the "screen projection" control A141. The smart TV 2 sends the reverse control instruction 8 to the mobile phone 1, and the mobile phone 1 responds to the reverse control instruction 8 and displays Figure 14 The interface 1302 shown in (d) is a device selection interface. Moreover, the mobile phone 1 sends the screen content corresponding to the interface 1302 to the smart TV 2 through the mirroring screen connection. Figure 14 As shown in (e), the smart TV 2 receives the screen content corresponding to the interface 1302 and displays the projection interface A15 in the projection area 1. The projection interface A15 corresponds to the interface 1302 currently displayed on the mobile phone 1.

[0168] After the smart TV 2 displays the screen projection interface A15, the smart TV 2 takes a screenshot of the content currently displayed, obtains the screenshot image 7, and identifies and analyzes the screenshot image 7, determines that the interface jump position of the currently displayed screen projection interface A15 is the device selection interface, and determines that the control to be triggered is the "Smart TV 2" option A151, and then generates the reverse control instruction 9 corresponding to the "Smart TV 2" option A151, that is, click the reverse control instruction of the "Smart TV 2" option A151. Smart TV 2 sends the reverse control instruction 9 to mobile phone 1, and mobile phone 1 responds to the reverse control instruction 9 and displays Figure 14 The interface 1303 shown in (f) is the screen projection completion interface. Moreover, the mobile phone 1 sends the screen content corresponding to the interface 1303 to the smart TV 2 through the mirror projection connection. Figure 14 As shown in (g), the smart TV 2 receives the screen content corresponding to the interface 1303 and displays the projection interface A16 in the projection area 1. The projection interface A16 corresponds to the interface 1303 currently displayed on the mobile phone 1.

[0169] After the smart TV 2 displays the screen projection interface A16, the smart TV 2 takes a screenshot of the content currently displayed, obtains the screenshot image 8, and identifies and analyzes the screenshot image 8 to determine that the interface jump position of the currently displayed screen projection interface A16 is the screen projection completion interface, that is, the smart TV 2 determines that the live broadcast 2 currently displayed by the mobile phone 1 has been projected to the smart TV 2 via DLNA. Afterwards, the smart TV 2 can send a preset counter-control instruction 10 to the mobile phone 1, and the counter-control instruction 10 is used to control the mobile phone 1 to cancel the display of the screen projection pop-up window 13031. The mobile phone 1 responds to the counter-control instruction 10, and the mobile phone 1 cancels the display of the screen projection pop-up window 13031, that is, returns to Figure 14 The interface 1304 shown in (h) is the live broadcast room interface of Live Broadcast 2. Moreover, mobile phone 1 sends the screen content corresponding to interface 1304 to smart TV 2 through mirroring screen connection. For example, smart TV 2 receives the screen content corresponding to interface 1304, and smart TV 2 cancels the display of screen projection pop-up window 13031 in screen projection area 1, that is, smart TV 2 displays Figure 13 The display interface 1205 is shown in (e).

[0170] Among them, the above-mentioned process of identifying and analyzing the screenshot image, determining the interface jump position of the currently displayed projection interface, and generating the corresponding reverse control instructions can be found in the relevant description of the knowledge graph below, which will not be repeated here.

[0171] In some examples, for multi-channel mirrored content, Smart TV 2 displays the mirrored content while the player simultaneously outputs the corresponding audio and video data. For other DLNA mirrored content, silent mode can be used. Of course, users can also choose to output the audio and video data corresponding to any of the mirrored content according to their actual needs.

[0172] For example, Figure 15As shown in (a), the smart TV 2 can display a split-screen live broadcast interface 1, which includes a projection area 1, a projection area 2, and a projection area 3. The screen content 1401 currently displayed on the mobile phone 1 is displayed in the projection area 1, that is, the smart TV 2 displays the live screen of the live broadcast 3 in the projection area 1, and the mute control 1 is also displayed in the projection area 1. At this time, the display state of the mute control 1 is non-mute, indicating that the player of the smart TV 2 is currently playing the audio and video data corresponding to the screen content 1. In addition, the mute control 1 can be used to adjust the audio and video output mode of the screen content 1. The live broadcast room interface 1402 of the live broadcast 1 is displayed in the projection area 2, and the mute control 2 is also displayed in the projection area 2. At this time, the display state of the mute control 2 is mute, indicating that the live broadcast 1 is currently played in a silent mode. In addition, the mute control 2 can be used to adjust the audio and video output mode of the live broadcast 1. The live broadcast room interface 1403 of Live Broadcast 2 is displayed in the projection area 3, and the mute control 3 is also displayed in the projection area 3. At this time, the display state of the mute control 3 is mute, indicating that Live Broadcast 2 is currently being played in mute mode. In addition, the mute control 3 can be used to adjust the audio and video output mode of Live Broadcast 2.

[0173] In response to the click operation on the mute control 1 and the click operation on the mute control 2, the smart TV 2 controls the player to pause outputting the audio and video data corresponding to the screen content 1, and controls the player to play the audio and video data corresponding to the live broadcast 1. And, the smart TV 2 displays Figure 15 (b) shows split-screen live broadcast interface 2. Unlike split-screen live broadcast interface 1, in projection area 1, the display state of mute control 1 switches from non-mute to mute. At the same time, in projection area 2, the display state of mute control 2 switches from mute to non-mute.

[0174] In response to the click operation on the mute control 1 and the click operation on the mute control 3, the smart TV 2 controls the player to pause outputting the audio and video data corresponding to the screen content 1, and controls the player to play the audio and video data corresponding to the live broadcast 2. And, the smart TV 2 displays Figure 15 The split-screen live broadcast interface 3 shown in (c) of FIG. Unlike the split-screen live broadcast interface 1, in the projection area 1, the display state of the mute control 1 switches from the non-mute state to the mute state. At the same time, in the projection area 3, the display state of the mute control 3 switches from the mute state to the non-mute state.

[0175] In some examples, users can also turn off any screen projection content according to actual needs.

[0176] For example, Figure 16As shown in (a) in the figure, the smart TV 2 can display a split-screen live broadcast interface 1, which includes a projection area 1, a projection area 2, and a projection area 3. The screen content 1501 currently displayed on the mobile phone 1 is displayed in the projection area 1, that is, the smart TV 2 displays the live picture of the live broadcast 3 in the projection area 1. The live broadcast room interface 1502 of the live broadcast 1 is displayed in the projection area 2, and the close control 1 is also displayed in the projection area 2. The close control 1 is used to trigger the closing of the projection content currently being played in the projection area 2. The live broadcast room interface 1503 of the live broadcast 2 is displayed in the projection area 3, and the close control 2 is also displayed in the projection area 2. The close control 2 is used to trigger the closing of the projection content currently being played in the projection area 3.

[0177] In response to the click operation on the close control 1, the smart TV 2 closes the screen projection content corresponding to the live broadcast 1. At this time, the smart TV 2 displays Figure 16 The split-screen live broadcast interface 4 shown in (b) of FIG. Different from the split-screen live broadcast interface 1, the split-screen live broadcast interface 4 includes a projection area 1 and a projection area 2, wherein the projection area 1 displays the screen content 1501 currently displayed on the mobile phone 1, and the projection area 2 displays the live broadcast room interface 1503 of the live broadcast 2.

[0178] In response to the click operation on the close control 2, the smart TV 2 closes the screen content corresponding to the live broadcast 2. At this time, the smart TV 2 displays Figure 16 The split-screen live broadcast interface 4 shown in (c) is different from the split-screen live broadcast interface 3 in that a guide screen is displayed in the projection area 2.

[0179] 2. One-click split screen projection

[0180] 2.1. The process of mirroring the screen content of mobile phone 1 to smart TV 2

[0181] like Figure 17 As shown in (a), mobile phone 1 can display interface 1601, which is the main interface of the live broadcast APP. After that, mobile phone 1 initiates mirroring screen projection to smart TV 2. After mobile phone 1 establishes a connection with smart TV 2 and the negotiation is completed, smart TV 2 can display Figure 17The interface 1602 shown in (b) is a split-screen live broadcast interface. Similar to the drag split-screen method, the interface 1602 may include two projection areas, such as projection area 1 and projection area 2, which are respectively used to display the multi-channel projection content of mobile phone 1. Among them, the mirror projection content 16021 of mobile phone 1 is displayed in projection area 1, that is, the main interface of the live broadcast APP. Projection area 2 can be used to display other projection content of mobile phone 1. Different from the drag split-screen method, the interface 1602 can also display a "one-click projection" control 16022. The "one-click projection" control 16022 is used to trigger the media resources currently displayed in projection area 1 (that is, the media resources associated with the screen content currently displayed by mobile phone 1) to be split-screen projected in projection area 2 through DLNA projection.

[0182] Among them, the process of mobile phone 1 entering interface 1601 and the process of mobile phone 1 establishing a mirror screen connection with smart TV 2 can be referred to the above 1.1 process of mobile phone 1 mirroring its screen content to smart TV 2, which will not be repeated here.

[0183] 2.2 One-click split screen process

[0184] During the mirroring process between mobile phone 1 and smart TV 2, the user can point to the remote control and click the "one-click screen casting" control to cast the media resources currently displayed in the projection area 1 of smart TV 2 (that is, the media resources associated with the screen content currently displayed on mobile phone 1) in the projection area 2 through DLNA screen casting, thereby realizing split-screen casting of the mirrored screen of mobile phone 1 and other media resources.

[0185] For example, Figure 18 As shown in (a) of FIG, after the smart TV 2 enters the interface 1602, the user points the pointing remote control toward the smart TV 2 to display a cursor 16023 in the interface 1602. The user adjusts the position of the cursor 16023 on the screen (i.e., the display area) of the smart TV 2 by moving the pointing remote control. When the cursor 16023 is located on the "one-key screen casting" control 16022, the user presses a designated button on the pointing remote control (e.g., the OK button), thereby triggering the one-key screen casting command by pointing the remote control.

[0186] In response to the first screen projection instruction, the smart TV 2 takes a screenshot of the content currently displayed, identifies and analyzes the obtained screenshot image, generates a reverse control instruction, and sends the reverse control instruction to the mobile phone 1 to reversely control the mobile phone 1 to execute the reverse control instruction. In response to the reverse control instruction, the mobile phone 1 displays the new screen content, and sends the new screen content displayed by the mobile phone 1 to the smart TV 2 through the mirror projection connection, so that the smart TV 2 synchronously displays the updated screen content of the mobile phone 1 in the projection area 1. After displaying the updated screen content of the mobile phone 1, the aforementioned process of issuing reverse control instructions is repeated until the mobile phone 1 is reversely controlled to initiate a DLNA screen projection connection to the smart TV 2, and send a DLNA screen projection instruction 1 to the smart TV 2 through the DLNA screen projection connection. The DLNA screen projection instruction 1 carries the resource address 1 of the live broadcast 1 corresponding to the screen content currently displayed by the mobile phone 1. After receiving the DLNA screen projection instruction 1, the smart TV 2 obtains the live broadcast 1 according to the resource address 1. After obtaining the live broadcast 1, the smart TV 2 can display Figure 18 In the interface 1701 shown in (b), compared with the interface 1602, the live broadcast 1 is played in the projection area 2 of the interface 1701, that is, the live broadcast interface of the live broadcast 1 is displayed. At this time, the mirror content corresponding to the mobile phone 1 is still presented in the projection area 1, thereby realizing the split-screen projection of the screen content (mirror picture) of the mobile phone 1 and other media resources, and completing the media resources corresponding to the screen content currently displayed by the mobile phone 1 through DLNA projection, and projecting them to the projection area 2 on the right for split-screen display.

[0187] It should be noted that, unlike the drag split screen method, in this scenario, the current screen content of mobile phone 1 displayed in projection area 1 is the main interface of the live broadcast APP. Therefore, in the subsequent split screen projection process, the screen content displayed in projection area 1 (i.e., the screen content of mobile phone 1) can be moved from the main interface of the live broadcast APP to the live broadcast room of live broadcast 1 corresponding to the main interface, and then the DLNA projection process of live broadcast 1 is performed, and finally, the main interface of the live broadcast APP is returned. It can be understood that in this scenario, compared with the drag split screen method, the interface jump path is different. However, the process of issuing each counter-control instruction is the same. Therefore, the process of interface 1602 jumping to interface 1701 can be referred to above. Figure 11 The description is not repeated here.

[0188] In the embodiment of the present application, when generating a counter-control instruction based on a screenshot image, the screenshot image can be matched with a pre-built knowledge graph to generate the counter-control instruction. The following is a schematic illustration of the knowledge graph with reference to the accompanying drawings.

[0189] The knowledge graph can also be referred to as preset path data. The knowledge graph can include interface information for multiple preset interfaces, as well as information indicating interface jump relationships between the multiple preset interfaces. The multiple preset interfaces can include multiple preset interfaces that jump from a preset homepage interface to a preset completion interface.

[0190] The interface information of a preset interface may include an interface jump location, element information, and a preset action type. The interface jump location is used to uniquely identify the preset interface. For example, when constructing a knowledge graph, an interface jump location may be set for each preset interface.

[0191] The element information may include multiple key elements in the preset interface and the preset coordinates of each key element. Key elements may be text elements, such as the text "Cast Screen" and the text "Click to enter the live broadcast room". Key elements may also be icon elements (i.e., icons), such as the "Cast Screen" icon and the icon of the "Control Panel" control. It should be noted that the key elements in a preset interface may include all interface elements in the preset interface, or may include some interface elements in the preset interface. There is no specific limitation on the number and type of key elements here.

[0192] For a preset interface, a key element among multiple key elements in the preset interface is a designated key element. The designated key element is the key element corresponding to the interactive control to be triggered when the preset interface is displayed on the terminal device. For example, when the preset interface is the control panel interface, the designated key element is the "Cast Screen" icon corresponding to the "Cast Screen" control.

[0193] The preset coordinates of the key element refer to the coordinates of the key element on the screen of the smart TV in the display coordinate system of the smart TV. For example, the preset coordinates of the key element can be the coordinates of the center point of the key element, or can include the coordinates of the four corner points of the key element.

[0194] The element information may also include weight parameters for each key element. In an embodiment of the present application, different preset interfaces may include the same interface elements. Therefore, weight parameters may be set for the key elements in the preset interface based on the degree of association between the key elements and the preset interface. The larger the weight parameter corresponding to a key element (such as close to 1), the greater the degree of association between the key element and the corresponding preset interface. For example, a key element only belongs to a certain preset interface, indicating that the key element has a greater degree of association with the preset interface, and a larger weight parameter (such as 0.8) may be set. The smaller the weight parameter corresponding to a key element (such as close to 0), the smaller the degree of association between the key element and the corresponding preset interface. For example, multiple preset interfaces all include a certain key element, indicating that the key element has a smaller degree of association with the preset interface, and a smaller weight parameter (such as 0.8) may be set.

[0195] The preset action type is used to indicate the touch action type for the interactive control to be triggered when the interface displayed on the terminal device is the preset interface. For example, the preset action type can be a click or a slide. It is understood that there is a corresponding relationship between the preset action type and the interface jump position corresponding to the preset interface.

[0196] For example, taking the live broadcast application as an example, the knowledge graph 1 is the knowledge graph corresponding to the drag split screen instruction. Figure 19 As shown in Figure 1, Knowledge Graph 1 includes four preset interfaces for jumping from the live broadcast room interface to the screen projection completion interface. The interface information of each preset interface in Knowledge Graph 1 can be found in Table 1 below.

[0197] Table 1 Interface information corresponding to knowledge graph 1

[0198]

[0199] It should be noted that the coordinates and weight parameters corresponding to each key element are not shown in Table 1.

[0200] From the above Table 1 and Figure 19 It can be seen that when the projection interface in the screenshot image is the live broadcast room interface, the interface jump position corresponding to the projection interface is Page 1, the designated key element to be triggered is the "control panel" icon, the touch position corresponding to Page 1 is the coordinates corresponding to the "control panel" icon, and the action type corresponding to Page 1 is click.

[0201] When the projection interface in the screenshot image is the control panel interface, the interface jump position corresponding to the projection interface is Page2, the designated key element to be triggered is the "Cast" icon, the touch position corresponding to Page2 is the coordinates corresponding to the "Cast" icon, and the action type corresponding to Page2 is click.

[0202] When the projection interface in the screenshot image is the projection completion interface, the interface jump position corresponding to the projection interface is Page 3, the designated key element to be triggered is the "Smart TV 2" option icon, the touch position corresponding to Page 3 is the coordinates corresponding to the "Smart TV 2" option icon, and the action type corresponding to Page 3 is click.

[0203] For example, taking the live broadcast application as an example, the knowledge graph 2 is the knowledge graph corresponding to the drag split screen instruction. Figure 20 As shown in Figure 1, Knowledge Graph 1 includes five preset interfaces for jumping from the main interface of the live broadcast app to the screen projection completion interface. The interface information of each preset interface in Knowledge Graph 1 can be found in Table 1 below.

[0204] Table 2 Interface information corresponding to knowledge graph 2

[0205]

[0206]

[0207] It should be noted that the coordinates and weight parameters corresponding to each key element are not shown in Table 1.

[0208] From the above Table 1 and Figure 20 It can be seen that when the projection interface in the screenshot image is the main interface, the interface jump position corresponding to the projection interface is Page 1, the designated key element to be triggered is the "Click to enter the live broadcast room" control icon, the touch position corresponding to Page 1 is the coordinates corresponding to the "Click to enter the live broadcast room" control icon, and the action type corresponding to Page 1 is click.

[0209] When the projection interface in the screenshot image is the live broadcast room interface, the interface jump position corresponding to the projection interface is Page2, the designated key element to be triggered is the "control panel" icon, the touch position corresponding to Page2 is the coordinates corresponding to the "control panel" icon, and the action type corresponding to Page2 is click.

[0210] When the projection interface in the screenshot image is the control panel interface, the interface jump position corresponding to the projection interface is Page 3, the designated key element to be triggered is the "Cast" icon, the touch position corresponding to Page 3 is the coordinates corresponding to the "Cast" icon, and the action type corresponding to Page 3 is click.

[0211] When the projection interface in the screenshot image is the projection completion interface, the interface jump position corresponding to the projection interface is Page 4, the designated key element to be triggered is the "Smart TV 2" option icon, the touch position corresponding to Page 4 is the coordinates corresponding to the "Smart TV 2" option icon, and the action type corresponding to Page 4 is click.

[0212] The above is a schematic description of the application scenario and knowledge graph of the screen projection method provided in the embodiment of the present application. The screen projection method provided in the embodiment of the present application is described below.

[0213] Figure 21 This is a flowchart of a screen projection method provided in an embodiment of the present application. Figure 21 As shown, the screen projection method may include the following steps:

[0214] S101, in the multi-channel screen projection mode, the first screen content currently displayed by the terminal device is received by the terminal device and sent via the mirror projection connection, and the first interface is displayed. The first interface includes a first projection area and a second projection area. The first projection interface corresponding to the first screen content is displayed in the first projection area. The first projection interface includes a first designated element, and the first designated element corresponds to the first control in the first screen content.

[0215] In an embodiment of the present application, in multi-channel screen projection mode, the display device can simultaneously support one mirroring screen projection and at least one DLNA screen projection push. The mirroring screen projection connection is established based on a mirroring screen projection protocol that supports a reverse control protocol, and the mirroring screen projection protocol includes the Miracast protocol. For example, the display device simultaneously supports one mirroring screen projection and one DLNA screen projection push. For another example, the display device simultaneously supports one mirroring screen projection and two DLNA screen projection pushes.

[0216] After the terminal device establishes a connection with the display device based on the mirroring screen projection protocol, the terminal device collects screen frame data (such as audio and video streams, etc.) in real time, and sends the screen frame data to the display device. Among them, the first screen content refers to the screen frame data currently received by the display device. After receiving the first screen content, the display device renders and displays the first interface, which may include a first projection area and a second projection area, and presents the first screen content in the first projection area. It can be understood that the first projection area (i.e., the projection area 1 mentioned above, which can also be referred to as the mirroring screen projection area) is used to display the screen content currently displayed by the terminal device received through the mirroring screen projection connection. (i.e., the projection area 2 mentioned above, which can also be referred to as the first area to be projected) is used to display other projection content besides mirroring screen projection, such as media resources pushed by the terminal device through DLNA projection.

[0217] For example, Figure 10 As shown in (a), the interface 91 (i.e., the first interface) includes a projection area 1 and a projection area 2. The first projection interface corresponding to the first screen content of the terminal device is displayed in the projection area 1, such as the live screen of live broadcast 1.

[0218] The first screen projection interface is a screen in which the display device decodes and renders the first screen content (such as an audio stream) transmitted by the terminal device. The first screen projection interface includes a first specified element, which is an interface element corresponding to the first control. The first control is a specified interactive control in the first screen content displayed by the terminal device. Through the first control, the terminal device can be triggered to perform a target action, which is to initiate a DLNA screen projection command to the display device.

[0219] In some examples, before receiving the first screen content currently displayed by the terminal device sent by the terminal device via the mirroring screen projection connection, the screen projection method also includes: responding to the start instruction of the multi-way screen projection mode, starting the multi-way screen projection mode, and displaying a guide interface, the guide interface is used to guide the split-screen display of media resources corresponding to the screen content sent by the terminal device via the mirroring screen projection connection and the screen content currently displayed by the terminal device.

[0220] S102 : In response to a first control instruction sent by the control device, taking a screenshot of content currently displayed by the display device to obtain a first screenshot image.

[0221] The first control instruction is used to trigger the display device to send multiple counter-control instructions to the terminal device, and the multiple counter-control instructions are used to control the terminal device to push its currently displayed media resources to the display device for split-screen display via the DLNA protocol, that is, to control the terminal device to perform the target action.

[0222] In some examples, the first control instruction is an instruction to drag the first projection screen interface displayed in the first projection screen area to the second projection screen area. Exemplarily, after the first screen content currently displayed by the terminal device is received and sent by the terminal device through the mirror projection connection, and the first interface is displayed, the method further includes: in response to the cursor movement instruction sent by the control device, displaying a pointing cursor in the first interface, and adjusting the position of the pointing cursor in the first interface; when the pointing cursor is located in the first projection screen area, displaying a drag control in the first projection screen area, the drag control is located in the upper layer of the first projection screen interface, and the drag control is used to drag the first projection screen interface. In this regard, the first control instruction is: selecting the drag control by the pointing cursor, and dragging the first projection screen interface to the second projection screen area by the drag control, and an instruction to deselect the drag control (i.e., release the drag control or the first projection screen interface) after dragging the first projection screen interface to the second projection screen area.

[0223] In this embodiment, after the first projection interface in the first interface is displayed, by dragging the first projection interface displayed in the first projection area to the second projection area, the display device can be triggered to send multiple reverse control instructions to the terminal device to control the terminal device to project its currently displayed media resources to the second projection area for split-screen display, which is convenient to operate and provides a better user experience.

[0224] For example, the display device is a smart TV 2, and the first interface is Figure 10 In the interface 91 shown in (a), the first projection area is projection area 1, the second projection area is projection area 2, and the control device is a pointing remote control, as an example. Figure 10As shown, after the smart TV 2 displays the interface 91, the cursor (i.e., the pointing cursor) 912 is adjusted in the interface 91 by moving the pointing remote control. When the cursor 912 moves into the projection area 1, a drag control 921 is displayed in the projection area 1. Move the cursor 912 to the drag control 921 and press the designated button of the pointing remote control (i.e., select the drag control). Then, press and hold the designated button of the pointing remote control to move the first projection interface (i.e., Figure 10 Drag the projection interface A3) shown in (c) to the projection area 2, and then release the designated button on the remote control (i.e., release the drag control or release the first projection interface).

[0225] Exemplarily, the display state of the drag control can change with user operations. Specifically, when the pointing cursor is in the first projection area and not on the drag control, the display state of the drag control is unselected. In this way, the position of the drag control can be prompted while reducing the interference of the drag control on the displayed interface content. When the pointing cursor is on the drag control, the display state of the drag control switches from unselected to selected. In this way, the interaction status can be fed back to the user, guiding the user to continue the drag split screen operation.

[0226] For example, Figure 10 As shown in (b) of FIG, when the cursor 912 is in the projection area 1 (the first projection area) and not on the drag control 921, the drag control is gray. When the cursor 912 is on the drag control 921 (i.e., the drag control is selected), the drag control switches from gray to a highlighted effect.

[0227] For example, when the drag control is displayed in the first projection area, a first prompt message is presented on the drag control, and the first prompt message is used to prompt the user to drag the first projection interface to the second projection area by dragging the control. Figure 10 As shown in (b) in the figure, the first prompt message is "Drag right here to split the screen". In this way, the user can be guided to drag the first projection interface to project the split screen of the first screen content corresponding to the first projection interface.

[0228] In other examples, the screen projection method further includes: displaying a first function control in the first interface, wherein the first control instruction is a selection instruction for the first function control. The first function control is used to trigger the display device to send multiple reverse control instructions to the terminal device, and the multiple reverse control instructions are used to control the terminal device to push its currently displayed media resources to the display device via the DLNA protocol for split-screen display, that is, to control the terminal device to perform the target action.

[0229] For example, Figure 17As shown in (a) in FIG, the first function control is the “one-key screen casting” control 16022. Accordingly, the first control instruction may be an operation of moving the cursor to the “one-key screen casting” control 16022 and pressing a designated button on the remote control.

[0230] In this way, when the display device displays the first interface, the first function control is displayed in the first interface. In this way, through the first function control, the display device can be triggered with one click to send multiple reverse control instructions to the terminal device to control the terminal device to project its currently displayed media resources to the second projection area for split-screen display. The operation is convenient and the user experience is better.

[0231] In an embodiment of the present application, in response to a first control instruction sent by a control device, a screenshot of the content currently displayed on the display device is taken, and a reverse control instruction is sent to the terminal device by identifying and analyzing the screenshot image. After sending the reverse control instruction, a screenshot of the content currently displayed on the display device is continued, and the next reverse control instruction is sent to the terminal device by identifying and analyzing the screenshot image. The aforementioned process is repeated, and multiple reverse control instructions can be sent to the display device to control the terminal device to project its currently displayed media resources to a second projection area for split-screen display.

[0232] That is, in response to the first control instruction sent by the control device, a screenshot is taken of the content currently displayed by the display device to obtain a first screenshot image, and a reverse control instruction is sent to the terminal device based on the first screenshot image.

[0233] S103, obtaining first action information corresponding to the first designated element (i.e., action information 1 above), the first action information is determined by identifying the first designated element and the first position information of the first designated element in the first screenshot image, and based on the first position information and the action type corresponding to the first designated element.

[0234] Specifically, after obtaining the first screenshot image, the display device sends the first screenshot image to the intelligent agent, so that the intelligent agent recognizes and analyzes the first screenshot image to obtain the first action information. The display device receives the first action information sent by the intelligent agent. The generation process of the first action information is described later. Figure 22 The description is not repeated here.

[0235] S104. Generate a first reverse control instruction based on the first action information. The first reverse control instruction is used to control the terminal device to initiate a first DLNA screen projection connection to the display device, and send a first screen projection instruction to the display device through the first DLNA screen projection connection. The first screen projection instruction carries a first resource address (i.e., resource address 1 above). The first resource address is the resource address of the first media resource (i.e., media resource 1 above) corresponding to the first screen content currently displayed by the terminal device.

[0236] In some examples, a first reverse control instruction is generated based on the first action information, including: performing coordinate conversion on the first position information based on a preset coordinate conversion relationship to obtain fourth position information, wherein the preset coordinate conversion relationship is used to indicate the coordinate conversion relationship between the display device and the terminal device; based on the event rules corresponding to the terminal device, a first reverse control instruction is generated according to the fourth position information and the action type corresponding to the first specified element.

[0237] The event rules corresponding to the terminal device refer to the interaction event rules of the terminal device. The interaction events of the terminal device may include MotionEvent and KeyEvent.

[0238] For example, taking MotionEvent as an example,

[0239] Create and dispatch a touch press event:

[0240] MotionEvent downEvent=createMotionEvent(MotionEvent.ACTION_DOWN,x,y);

[0241] dispatchTouchEvent(downEvent).

[0242] Create and dispatch touchup events

[0243] MotionEvent upEvent=createMotionEvent(MotionEvent.ACTION_UP,x,y);

[0244] dispatchTouchEvent(upEvent).

[0245] In an embodiment of the present application, in response to a control instruction, the display device sends multiple reverse control instructions to the terminal device. That is, if the first control is not a preset screen casting control, before sending the first reverse control instruction to the terminal device, one or more reverse control instructions are sent to the terminal device until the display device displays a screen casting interface including the preset screen casting control. The preset screen casting control is used to trigger the terminal device to initiate a first DLNA screen casting connection to the display device.

[0246] The following takes the first projection interface as the control panel interface as an example. Generating the first reverse control instruction according to the first action information may include the following steps:

[0247] S201: Generate a second reverse control instruction according to the first action information.

[0248] Among them, the second reverse control instruction is a click instruction for the "screen projection" control in the control panel interface.

[0249] S202, sending a second reverse control instruction to the terminal device through the mirroring screen connection.

[0250] S203, receiving the second screen content sent by the terminal device in response to the second counter-control instruction, and displaying the second projection interface corresponding to the second screen content in the first projection area, the second projection interface including a second specified element, and the second specified element corresponding to the preset projection control in the second screen content.

[0251] S204: Take a screenshot of the content currently displayed on the display device to obtain a second screenshot image.

[0252] S205 , obtaining second action information corresponding to the second designated element, where the second action information is determined by identifying the second designated element and the second position information of the second designated element in the second screenshot image, and based on the second position information and the action type corresponding to the second designated element.

[0253] S206: Generate a first reverse control instruction according to the second action information.

[0254] The specific implementation process of generating the first reverse control instruction according to the second action information can be found in the process of generating the first reverse control instruction according to the first action information, which will not be repeated here.

[0255] S105, in response to the first screen projection instruction sent by the terminal device, obtain the first media resource according to the first resource address, and play the first media resource in the second screen projection area.

[0256] In some examples, before playing the first media resource in the second projection area, the projection method further includes: displaying a guide animation in the second projection area, the guide animation being used to guide the execution of the first control instruction. The guide animation can guide the user to perform split-screen projection.

[0257] In some embodiments, after the first media resource is played in the second projection area, the projection method further includes: sending a third reverse control instruction to the terminal device through a mirror projection connection, so that the terminal device responds to the third reverse control instruction and displays a third screen content, the third screen content includes a third control, and the third screen content corresponds to the second media resource; receiving the third screen content sent by the terminal device through the mirror projection connection, and displaying a third projection interface corresponding to the third screen content in the first projection area, the third projection interface includes a third designated element, and the third designated element corresponds to the third control.

[0258] The third reverse control instruction is used to control the terminal device to switch the currently displayed screen content. For example, the third reverse control instruction is Figure 12The reverse control command 5 shown is an upward slide command. The third screen content is Figure 12 The live broadcast room screen of live broadcast 2 shown in (b) in the figure. The third screen projection interface can be Figure 12 The projection interface A9 shown in (c) in the figure.

[0259] It should be noted that the specific implementation method of controlling the terminal device to switch the currently displayed screen content through the third reverse control instruction can be found in the Figure 12 The description is not repeated here.

[0260] In this embodiment, the display device responds to the first control instruction and reversely controls the terminal device to push its currently displayed media resources to the display device for split-screen display through the DLNA protocol. The display device can continue to send a third reverse control instruction to the terminal device to control the terminal device to switch its displayed media resources. In this way, the user can watch other screen contents of the terminal device while viewing the first media resources of the display device.

[0261] In some embodiments, after the first projection area displays the third projection interface corresponding to the third screen content, the projection method further includes:

[0262] S301 : In response to a second control instruction sent by a control device, taking a screenshot of content currently displayed by a display device to obtain a third screenshot image.

[0263] Before S301, in response to the third control instruction sent by the control device, a second interface is displayed, the second interface includes a first projection area, a second projection area and a third projection area, the third projection interface is displayed in the first projection area, the first media resource is played in the second projection area, and a guide animation is displayed in the third projection area.

[0264] Among them, the third control instruction may include: moving the cursor into the first projection area and selecting the instruction to drag the control.

[0265] For example, Figure 13 As shown in (d), interface 1204 (i.e., the second interface) includes projection area 1, projection area 2 and projection area 3 (i.e., the third projection area, or the second area to be projected), wherein projection area 1 displays projection interface A12 (i.e., the third projection interface), projection area 2 displays the live screen of live broadcast 1, and projection area 3 can display prompt information, such as "Please drag the live broadcast here to start projection", and display the guide animation of split-screen live broadcast to guide the user to drag the split screen operation.

[0266] After the second interface is displayed, in response to the second control instruction sent by the control device, a screenshot is taken of the content currently displayed by the display device to obtain a third screenshot image.

[0267] The second control instruction is used to trigger the display device to send multiple counter-control instructions to the terminal device, and the multiple counter-control instructions are used to control the terminal device to push its currently displayed media resources to the display device for split-screen display via the DLNA protocol, that is, to control the terminal device to perform the target action.

[0268] Exemplarily, the second control instruction is an instruction to drag the third projection screen interface displayed in the first projection screen area to the third projection screen area. For example, the second control instruction is an instruction to drag the third projection screen interface to the third projection screen area by dragging the control, and to deselect the drag control (i.e., release the drag control or the third projection screen interface) after dragging the third projection screen interface to the third projection screen area. The implementation process of the second control instruction can be referred to above regarding Figure 13 The description is not repeated here.

[0269] S302, obtaining third action information corresponding to a third designated element, where the third action information is determined by identifying a third designated element and third position information of the third designated element in a third screenshot image, and based on the third position information and the action type corresponding to the third designated element.

[0270] Specifically, after obtaining the third screenshot image, the display device sends the third screenshot image to the intelligent agent, so that the intelligent agent recognizes and analyzes the third screenshot image to obtain the third action information. The display device receives the third action information sent by the intelligent agent. The generation process of the third action information is described later. Figure 22 The description is not repeated here.

[0271] S303: Generate a fourth reverse control instruction based on the third action information. The fourth reverse control instruction is used to control the terminal device to initiate a second DLNA screen projection connection to the display device, and send a second screen projection instruction to the display device through the second DLNA screen projection connection. The second screen projection instruction carries a second resource address, which is the resource address of the second media resource corresponding to the third screen content currently displayed by the terminal device.

[0272] S304: Send a fourth reverse control instruction to the terminal device through the mirroring screen projection connection; in response to the second screen projection instruction sent by the terminal device, obtain the second media resource according to the second resource address.

[0273] The specific implementation of S303 to S304 can be found in S104 to S105 above, which will not be repeated here.

[0274] S305: Play the second media resource in the third projection area.

[0275] In some examples, after playing the second media resource in the third projection area, the projection method also includes: sending a fifth reverse control instruction to the terminal device through a mirror projection connection, so that the terminal device responds to the fifth reverse control instruction and displays the fourth screen content, and the fourth screen content corresponds to the third media resource; receiving the fourth screen content sent by the terminal device through the mirror projection connection, and displaying a fourth projection interface corresponding to the fourth screen content in the first projection area, and the fourth projection interface includes a resource screen of the third media resource.

[0276] Optionally, the first media resource, the second media resource, and the third media resource are all live broadcast media resources.

[0277] Among them, the fifth reverse control instruction is used to control the terminal device to switch the currently displayed screen content. For example, the fifth reverse control instruction is Figure 13 The reverse control command 6 shown is a slide up command. The fourth screen content is Figure 13 The live broadcast room screen of live broadcast 3 shown in (f).

[0278] It should be noted that the specific implementation of reversely controlling the terminal device to play the second media resource in the third projection area and controlling the terminal device to switch the currently displayed screen content through the fifth reverse control instruction can be found in the description of the embodiment of the present invention. Figure 13 The description is not repeated here.

[0279] In this embodiment, the display device responds to the second control instruction and reversely controls the terminal device to push its currently displayed media resources to the display device for split-screen display through the DLNA protocol. The display device can continue to send a fifth reverse control instruction to the terminal device to control the terminal device to switch its displayed media resources. In this way, while the first media resource is played in the second projection area and the second media resource is played in the third projection area, other screen contents of the terminal device can also be viewed.

[0280] In an embodiment of the present application, in response to a split-screen projection instruction (such as a first control instruction or a second control instruction), the display device can take a screenshot, and by identifying and analyzing the screen content currently displayed by the terminal device in the screenshot image, generate and send a reverse control instruction to the terminal device to control the terminal device to switch the screen content. After the screen content after the terminal device switches is displayed in the projection area 1 of the display device, the display device can take a screenshot again, and based on the screenshot image, generate and send a reverse control instruction to the terminal device, and then reversely control the terminal device to establish a DLNA projection connection with the display device, and project the media resources currently displayed by the terminal device onto the display device. In other words, the user only needs to trigger the split-screen projection instruction, and the display device reversely controls the terminal device to project the media resources currently displayed by the terminal device onto the display device. In this way, during the projection process, the user does not need to operate on the terminal device, avoiding the user's visual focus from repeatedly switching between the display device and the terminal device, thereby improving the user experience.

[0281] Furthermore, in an embodiment of the present application, the mirrored content and other media resources of the terminal device can be displayed in split screen on the display device, so that the user can view the screen content of the terminal device on the display device while watching other media resources, without affecting the user's use of the terminal device.

[0282] Figure 22 This is a flow chart of another screen projection method provided in an embodiment of the present application. Figure 22 As shown, the screen projection method is applied to an intelligent agent, and the screen projection method may include the following steps:

[0283] S401, receiving a first screenshot image sent by a display device, where the first screenshot image is a screenshot image obtained by the display device in response to a first control instruction, and the first screenshot image includes a first projection interface displayed by the display device in a first projection area, and the first projection interface corresponds to the first screen content currently displayed by the terminal device.

[0284] S402: Identify a first designated element and first position information of the first designated element in a first screenshot image.

[0285] The first designated element is an interface element corresponding to the first control, which is a designated interactive control in the first screen content displayed by the terminal device. Through the first control, the terminal device can be triggered to perform a target action, which is to initiate a DLNA screen projection command to the display device.

[0286] like Figure 23 As shown, the screenshot image recognition process may include the following steps:

[0287] S501, obtaining preset path data; wherein the preset path data includes interface information of each of multiple preset interfaces, and data indicating the jump relationship between the multiple preset interfaces; the interface information includes multiple preset elements in the preset interface, preset position information of each preset element, and a preset action type corresponding to a specified preset element among the multiple preset elements.

[0288] The preset path data (also known as the knowledge graph) includes interface information of each of the multiple preset interfaces, as well as data indicating the jump relationship between the multiple preset interfaces. It is understood that the multiple preset interfaces may include multiple preset interfaces that jump from the preset home interface to the preset completion interface.

[0289] The interface information includes multiple preset elements in the preset interface and the preset position information of each preset element. Among them, the preset elements in a preset interface (i.e., the key elements mentioned above) may include all interface elements in the preset interface, or may include some interface elements in the preset interface. Exemplarily, the preset elements may be text elements, such as the text "Cast Screen" and the text "Click to Enter the Live Room". The key elements may also be icon elements (i.e., icons), such as the "Cast Screen" icon and the icon of the "Control Panel" control.

[0290] The preset position information of a preset element (i.e., the preset coordinates hereinabove) refers to the coordinate position of the preset element on the screen of the display device in the display coordinate system of the display device. For example, the preset position information of the preset element may include the coordinates of the center point of the preset element, or may include the coordinates of the four corner points of the preset element.

[0291] For a preset interface, one of the multiple preset elements in the preset interface is a designated preset element (i.e., the designated key element mentioned above). The designated preset element is the preset element corresponding to the interactive control to be triggered when the preset interface is displayed on the terminal device. For example, when the preset interface is a control panel interface, the designated preset element is the "Cast Screen" icon corresponding to the "Cast Screen" control.

[0292] The preset action type corresponding to a specified preset element indicates the touch action type to be triggered for the interactive control when the terminal device displays the preset interface. For example, the preset action type can be a click or a slide. It is understood that there is a corresponding relationship between the preset action type and the interface jump position corresponding to the preset interface.

[0293] The interface information also includes weight parameters corresponding to preset elements. In an embodiment of the present application, different preset interfaces may include the same interface elements. Therefore, weight parameters can be set for the preset elements in the preset interface according to the degree of association between the preset elements and the preset interface. The larger the weight parameter corresponding to a preset element (such as close to 1), the greater the degree of association between the preset element and the corresponding preset interface. For example, a preset element only belongs to a certain preset interface, indicating that the preset element has a greater degree of association with the preset interface, and a larger weight parameter (such as 0.8) can be set. The smaller the weight parameter corresponding to a preset element (such as close to 0), the smaller the degree of association between the preset element and the corresponding preset interface. For example, multiple preset interfaces all include a preset element, indicating that the preset element has a smaller degree of association with the preset interface, and a smaller weight parameter (such as 0.8) can be set.

[0294] In the embodiment of the present application, different preset path data can be pre-set for different applications and different split-screen projection scenarios. Different split-screen projection scenarios can be triggered by different control instructions. For example, taking the application as a live broadcast application, for the drag split-screen scenario, the first control instruction is the instruction to drag the first projection screen interface to the second projection screen area, and the preset path data can be Figure 19 For the one-key screen casting scenario, the first control instruction is the selection instruction for the "one-key screen casting" control, and the preset path data can be Figure 20 Based on this, after establishing multiple preset path data, an association can be established between the preset path data, the instruction type of the control instruction corresponding to the preset path data, and the application identifier of the application corresponding to the preset path data. In this way, the preset path data can be obtained according to the instruction type of the control instruction and the application identifier of the application to which the projection interface in the first screenshot belongs.

[0295] In some examples, S501 may include: obtaining the instruction type of the first control instruction; determining the application identifier of the application to which the first projection interface belongs based on the first screenshot image; and obtaining preset path data based on the instruction type and application identifier of the first control instruction.

[0296] The command type is used to distinguish different control commands. For example, if the first control command is a command to drag the first projection screen interface to the second projection screen area, the command type of the first control command is the drag split screen type. If the first control command is a command to select the "one-click projection screen" control, the command type of the first control command is the one-click projection screen type.

[0297] For different applications, the application interface is different, and the screen projection interface in the screenshot image is also different. Therefore, by performing image recognition on the first screenshot image, the application identifier of the application to which the first screen projection interface in the first screenshot image belongs can be determined. Furthermore, based on the instruction type and application identifier of the first control instruction, the preset path data can be obtained. In this way, based on the instruction type of the first control instruction and the application identifier of the application to which the first screen projection interface belongs, the preset path data can be obtained more accurately, and then automatic split-screen projection can be achieved based on the preset path data.

[0298] S502: Identify the first projection interface in the first screenshot image to obtain at least one first interface element and position information of each first interface element.

[0299] The first interface element is an interface element identified from the first projection interface of the first screenshot image, and the position information of the first interface element refers to the coordinate position of the first interface element in the first interface.

[0300] S503, for each preset interface, based on the first interface element and the position information of the first interface element, match at least one first interface element with multiple preset elements in the preset interface, determine the first preset interface corresponding to the first projection interface, and the designated preset element in the first preset interface is the first designated element.

[0301] Specifically, S503 may include:

[0302] S5031, for each preset interface, compare the first interface element with the first preset element in the preset interface, and compare the position information of the first interface element with the preset position information of the preset element in the preset interface to determine the second preset element.

[0303] The first preset element is any preset element in the preset interface. The second preset element is a preset element that matches the first interface element among multiple preset elements in the preset interface. Exemplarily, the second preset element can be one or more.

[0304] Exemplarily, the first interface element is compared with the first preset element, and the position information of the first interface element is compared with the preset position information of the first preset element to determine the second preset element, including: if the first interface element matches the first preset element, and the coordinates of the first interface element match the coordinates of the first preset element, then the first preset element is determined as the second preset element.

[0305] That is to say, after the first interface element is identified, it is determined whether there is a preset element matching the first interface element in each preset interface in the preset path data. Specifically, taking any preset interface as an example, the first interface element is compared with each preset element (i.e., the first preset element) in the preset interface. When comparing the first interface element with the first preset element, the element itself (i.e., text or icon) of the first interface element is compared with the element itself of the first preset element, and the position information of the first interface element is compared with the preset position information of the first preset element. If the element itself of the first interface element matches the element itself of the first preset element, and the position information of the first interface element is the same as the preset position information of the first preset element, it is determined that the first preset element matches the first interface element, and the first preset element is determined to be the second preset element.

[0306] Of course, when multiple first interface elements are identified from the first screenshot image, for each first interface element, the first interface element is compared with each preset element.

[0307] For example, taking the preset interface as the main interface of a live broadcast application, the preset elements include a "click to enter the live broadcast room" text element, a "home" text element, a "friends" text element, a "message" text element, a "me" text element, an "experience" text element, a "follow" text element, a "mall" text element, a "recommendation" text element, a "click to enter the live broadcast room" control icon element, a "search" icon element, and a "publish" icon element. The first interface element is compared with each of the above elements, and the position information of the first interface element is compared with the preset position information of each of the above elements to determine a second preset element that matches the main interface.

[0308] S5032: Determine a matching degree corresponding to the preset interface according to a weight parameter corresponding to the second preset element and the number of preset elements in the preset interface.

[0309] Specifically, for each preset interface, after determining the second preset element from the first interface element, the matching degree corresponding to the preset interface is determined based on the weight parameter corresponding to the second preset element and the number of preset elements in the preset interface. The matching degree can represent the degree of association between the second preset element and different preset interfaces.

[0310] Exemplarily, when determining the matching degree corresponding to the preset interface, the weight parameters of each second preset element are summed to obtain the total weight; the ratio of the total weight to the number of preset elements in the preset interface is determined as the matching degree corresponding to the preset interface.

[0311] S5033. Determine a first preset interface based on the matching degree corresponding to each preset interface, wherein the matching degree corresponding to the first preset interface is the maximum value among the matching degrees corresponding to the multiple preset interfaces, and the matching degree corresponding to the first preset interface is greater than the matching degree threshold corresponding to the first preset interface.

[0312] Among them, a matching threshold can be set for each preset interface. When the calculated matching degree is greater than the matching threshold, it means that the preset interface is a preset interface that matches the first interface element, that is, the preset interface matches the first projection interface in the first screenshot image.

[0313] In an embodiment of the present application, different preset interfaces may include the same preset elements. Therefore, when determining the first preset interface corresponding to the first projection interface, for each preset interface, the second preset element is determined by comparing the first interface element with the first preset element in the preset interface. According to the weight parameter corresponding to the second preset element and the number of preset elements in the preset interface, the matching degree corresponding to the preset interface is determined, and then according to the matching degree corresponding to each preset interface, the first preset interface is determined. In this way, the accuracy of determining the first preset interface can be improved.

[0314] S504: Determine the preset position information of the designated preset element in the first preset interface as the first position information of the first designated element, and determine the preset action type corresponding to the designated preset element in the first preset interface as the action type corresponding to the first designated element.

[0315] After the embodiment of the present application, after determining the first preset interface that matches the first projection interface, it indicates that the specified preset element in the first preset interface is the interface element corresponding to the first control to be triggered by the terminal device to be controlled, and the specified preset element corresponding to the first preset interface is determined as the first specified element, and the preset position information of the specified preset element is used to determine the first position of the first specified element, and the preset action type corresponding to the specified preset element is determined as the action type corresponding to the first specified element.

[0316] For example, when determining the preset action type corresponding to a specified preset element in a first preset interface, the preset action type corresponding to the specified preset element can be determined based on the interface jump position corresponding to the first preset interface. The interface jump position is used to uniquely identify the preset interface. For example, when constructing a knowledge graph, an interface jump position can be set for each preset interface.

[0317] For example, when the first preset interface matched by the first projection interface is the main interface of the live broadcast application, the interface jump position corresponding to the projection interface is Page1, the designated preset element to be triggered is the "click to enter the live broadcast room" control icon, and the action type corresponding to Page1 is click. Then the first position information of the first designated element is the coordinates corresponding to the "click to enter the live broadcast room" control icon, and the action type corresponding to the first designated element is click.

[0318] For example, when the first preset interface matched by the first projection interface is the control panel interface of the live broadcast application, the interface jump position corresponding to the projection interface is Page2, the designated preset element to be triggered is the "projection" control icon, and the action type corresponding to Page2 is click. Then the first position information of the first designated element is the coordinates corresponding to the "projection" control icon, and the action type corresponding to the first designated element is click.

[0319] S403: Determine first action information according to the first position information of the first designated element and the action type corresponding to the first designated element.

[0320] Exemplarily, the first action information may be represented by JSON data.

[0321] For example, the JSON data is:

[0322]

[0323] The "action" field represents the first action information, "CLICK" represents the action type is click, and "x" and "y" represent the first position information, that is, the touch coordinates.

[0324] S404: Send first action information to the display device so that the display device generates a first reverse control instruction based on the first action information. The first reverse control instruction is used to control the terminal device to initiate a first DLNA screen projection connection to the display device, and send a first screen projection instruction to the display device through the first DLNA screen projection connection. The first screen projection instruction carries a first resource address, which is the resource address of the first media resource corresponding to the first screen content currently displayed by the terminal device.

[0325] In an embodiment of the present application, the intelligent agent can identify the screenshot image sent by the display device, obtain the first designated element and the first position information of the first designated element, determine the first action information based on the first position information of the first designated element and the action type corresponding to the first designated element, and the first action information is used to generate a first reverse control instruction. In this way, the intelligent agent can identify the control to be triggered based on the screenshot image and generate corresponding action information. Then, the display device can send a reverse control instruction to the terminal device based on the action information, thereby controlling the terminal device to perform automatic split-screen projection. In addition, using the intelligent agent to identify and analyze the screenshot image can improve the accuracy of recognition and improve processing efficiency.

[0326] Next, we will continue to use the display device as a smart TV 2, the terminal device as a mobile phone 1, the control device as a pointing remote control, and the mirror projection protocol as the Miracast protocol as an example to illustrate the screen projection method provided in the embodiment of this application.

[0327] Figure 24 This is a flowchart of a screen projection method provided in an embodiment of the present application. Figure 24 As shown, the screen projection method may include the following steps:

[0328] S601. In response to the opening operation 1, the pointing remote control sends an opening instruction 1 to the fusion screen projection application of the smart TV 2.

[0329] Among them, the opening operation 1 is used to trigger the remote control to send the opening instruction 1. For example, the split screen projection function can be opened through the entrance provided by the integrated projection application in the smart TV 2. For example, the opening operation 1 can be for Figure 8 Of course, the above-mentioned opening operation 1 is only an example, and the embodiment of the present application does not specifically limit the type of the opening operation 1.

[0330] The opening instruction 1 is used to trigger the smart TV 2 to start the split-screen projection function, that is, to start the multi-channel projection mode.

[0331] S602. In response to the open instruction 1, the fusion screen projection application starts the multi-channel screen projection mode and displays the guide page.

[0332] In multi-screen projection mode, Smart TV 2 can simultaneously support one mirroring screen projection and at least one DLNA screen projection push. Specifically, in multi-screen projection mode, when Smart TV 2 receives the screen content sent by mobile phone 1, Smart TV 2 displays the corresponding screen content of mobile phone 1 in the designated projection area (such as projection area 1) in the display area.

[0333] For example, the guide page may be as follows: Figure 8 Interface 73 shown in (c) in FIG.

[0334] Specifically, the default boot page CastPlayerActivity is started through Android's native Activity startup method. After the application is started, the View is initialized. When entering the application, the default boot page will be displayed, which will display some device information, such as the device name of the current large-screen end and the corresponding network information. The boot page is displayed in equal proportion on the left and right through the Android standard controls TextView and ImageView and the corresponding Layout.

[0335] S603 , in response to the opening operation 2 , the mobile phone 1 opens the live broadcast APP and displays the screen content 01 , which includes the live broadcast picture of the media resource 1 .

[0336] The opening operation 2 can be used to trigger the mobile phone 1 to open the live screen of the media resource 1 of the live broadcast APP. Figure 9 , the click operation of the "click to enter the live broadcast room" control 821 shown in (b) of FIG. For example, the opening operation 2 may also be an operation of instructing the mobile phone 1 to open the live broadcast screen of the media resource 1 of the live broadcast APP through voice. In the embodiment of the present application, the type of the opening operation 2 is not specifically limited.

[0337] S604. In response to the screen casting operation 1, the mobile phone 1 sends a Miracast screen casting request to the integrated screen casting application.

[0338] The screen casting operation 1 can be used to trigger the mobile phone 1 to initiate a mirror screen casting connection to the smart TV 2. Figure 9 (d) to Figure 9 As shown in (f) in the figure, screen casting operation 1 may include a pull-down operation on interface 83, a click operation on the "wireless screen casting" control switch 841, and a click operation on the "Smart TV 2" device option 852. For example, screen casting operation 1 may also be an operation of initiating a mirror screen casting connection from mobile phone 1 to smart TV 2 through voice instruction. The type of screen casting operation 1 is not specifically limited in the embodiments of the present application.

[0339] S605 . In response to the Miracast screen projection request, the smart TV 2 negotiates with the mobile phone 1 to establish a mirror screen projection connection (i.e., a Miracast connection).

[0340] S606. Mobile phone 1 sends the screen content 01 of mobile phone 1 to the integrated screen projection application through the mirroring screen projection connection.

[0341] S607. In the multi-channel screen projection mode, the smart TV 2 displays the first interface, which includes the screen projection area 1 and the screen projection area 2, and displays the first screen projection interface corresponding to the screen content 01 in the screen projection area 1.

[0342] For example, the first interface may be a split-screen live broadcast interface of a display device, such as Figure 9 The interface 88 shown in (h) in FIG. Figure 17 Interface 1602 shown in (b).

[0343] S608. In response to the screen projection operation 2, point the remote control to send the split-screen projection instruction 1 to the fusion screen projection application.

[0344] Among them, the split-screen projection instruction 1 (that is, the first control instruction above) can be used to control the smart TV 2 to perform split-screen projection.

[0345] For example, the screen projection operation 2 may be an operation of dragging the screen content 01 (i.e., the first screen projection interface) currently displayed in the screen projection area 1 to the screen projection area 2. Figure 10 As shown in (c) in the figure, move the pointing remote control and move the cursor to the drag control of projection area 1. Press the OK button on the pointing remote control and drag to the right. When dragging to projection area 2, release the OK button.

[0346] Exemplarily, the first interface includes a first functional control, such as Figure 17 When the "One-key Cast" control 16022 shown in (b) of the first interface is pressed, the split-screen casting instruction 1 can also be a selection instruction for the "One-key Cast" control 16022 in the first interface. For example, move the pointing remote control, move the cursor to the "One-key Cast" control 16022 in the casting area 1, and press the OK button on the pointing remote control.

[0347] S609. In response to the split-screen projection instruction 1, the fusion projection application obtains the device information 1 and the instruction type 1 of the split-screen projection instruction 1.

[0348] The device information 1 is used to uniquely identify the smart TV 2. For example, the device information 1 is the name of the smart TV, such as smart TV 2.

[0349] The command type is used to identify the split-screen casting command that is triggered. Different types of split-screen casting commands correspond to different preset path data (i.e., knowledge graphs).

[0350] S610. The fusion screen projection application takes a screenshot of the currently displayed content to obtain a first screenshot image, which includes a first screen projection interface corresponding to the screen content 01.

[0351] It can be understood that here the screenshot operation is performed on the content currently displayed on the smart TV 2.

[0352] S611. The fusion screen projection application sends an action generation request 1 to the intelligent agent, and the action generation request 1 carries the first screenshot image, instruction type 1 and device information 1.

[0353] S612 , in response to action generation request 1 , the intelligent agent uses the recognition model to recognize the first screenshot image and obtains recognition result 1 .

[0354] Among them, recognition result 1 includes multiple interface elements 1 in the first projection interface and the first coordinates (i.e., first position information) of each interface element 1. The first coordinates are used to indicate the position of the interface element 1 in the first screenshot image, that is, the coordinates corresponding to the interface element 1 in the display coordinate system of the smart TV 2. For example, the first coordinates may include the coordinates of the center point of the interface element 1, or the coordinates of the four corner points of the interface element 1.

[0355] S613. The intelligent agent obtains the knowledge graph 1 according to the instruction type 1, and generates the action information 1 based on the knowledge graph 1 and the recognition result 1.

[0356] In an embodiment of the present application, multiple knowledge graphs are established in advance. The knowledge graph may include interface information of multiple preset interfaces, and information indicating the interface jump relationship between multiple preset interfaces. Multiple preset interfaces may include multiple preset interfaces that jump from a preset home page interface to a preset completion interface. The interface information of a preset interface includes an interface jump position (such as Page1), multiple key elements in the preset interface, and preset coordinates of each key element. The multiple key elements include a designated key element, and the designated key element is the key element corresponding to the interactive control to be triggered when the display interface of the terminal device is the preset interface, such as the "screen projection" icon.

[0357] Among them, key elements can include text elements and icon elements (icons), such as "Cast Screen" text and "Cast Screen" icon. The preset coordinates can include the coordinates of the center point of the key element, and can also include the coordinates of the four corner points of the key element. It can be understood that the coordinates of the key elements in the interface information are also the coordinates in the display coordinate system of the smart TV.

[0358] In some examples, different knowledge graphs can be established for different applications. Therefore, the intelligent agent can determine the application identifier 1 of the application to which the first projection interface belongs based on the recognition result 1, and obtain the knowledge graph 1 based on the instruction type 1 and the application identifier 1.

[0359] S614. The intelligent agent sends action information 1 to the fusion projection application, and the action information 1 includes touch position 1 and action type 1.

[0360] S615: The fusion screen projection application generates a first reverse control instruction based on the action information 1. The first reverse control instruction is used to control the mobile phone 1 to initiate a DLNA screen projection connection 1 to the smart TV 2, and send a screen projection instruction 1 to the smart TV 2 through the DLNA screen projection connection 1.

[0361] Specifically, the smart TV 2 analyzes the action information 1 and performs coordinate conversion on the touch position 1 , and generates a first reverse control instruction according to the converted touch position 1 , the action type 1 and the event rule corresponding to the mobile phone 1 .

[0362] S616. The fusion screen projection application sends a first reverse control instruction to mobile phone 1.

[0363] S617. In response to the first reverse control instruction, mobile phone 1 sends a DLNA screen projection request 1 to the integrated screen projection application.

[0364] S618 . In response to the DLNA screen projection request 1 , the smart TV 2 establishes a DLNA screen projection connection 1 with the mobile phone 1 .

[0365] S619. Mobile phone 1 sends screen projection instruction 1 to the integrated screen projection application through DLNA screen projection connection 1. Screen projection instruction 1 carries resource address 1 of media resource 1. Media resource 1 is the media resource corresponding to screen content 01 currently displayed by mobile phone 1.

[0366] S620. The integrated screen projection application obtains media resource 1 according to resource address 1 and plays media resource 1 in the screen projection area 2.

[0367] S621 . In response to the first reverse control instruction 1 , the mobile phone 1 displays the screen content 02 .

[0368] S622. Mobile phone 1 sends the screen content 02 of mobile phone 1 to smart TV 2 through mirroring.

[0369] S623. The integrated screen projection application displays screen content 02 in the screen projection area 1.

[0370] S624. The fusion screen projection application sends a third reverse control instruction to mobile phone 1. The third reverse control instruction is used to control mobile phone 1 to switch screen content.

[0371] For example, the third reverse control instruction may be Figure 12 The counter-control instruction 5 in .

[0372] S625 . In response to the third counter-control instruction, mobile phone 1 displays screen content 03 , where screen content 03 is the live broadcast content corresponding to media resource 2 .

[0373] S626. The integrated screen projection application displays screen content 03 in the screen projection area 1.

[0374] S627. In response to the screen projection operation 3, the pointing remote control sends a split-screen projection instruction 2 to the integrated screen projection application. The split-screen projection instruction 2 is used to instruct the smart TV 2 to perform split-screen projection.

[0375] Among them, the split-screen projection instruction 2 (that is, the second control instruction above) can be used to control the smart TV 2 to perform split-screen projection.

[0376] For example, the screen projection operation 2 may be an operation of dragging the screen content 03 currently displayed in the screen projection area 1 to the screen projection area 3. Figure 13 As shown in (d) in the figure, move the pointing remote control and move the cursor to the drag control of projection area 1. Press the OK button on the pointing remote control and drag to the right. When dragging to projection area 3, release the OK button.

[0377] S628. In response to the split-screen projection instruction 2, the fusion projection application displays the second interface, which includes projection area 1, projection area 2 and projection area 3, and displays the screen content of mobile phone 1 in projection area 1, plays media resource 1 in projection area 2, and plays media resource 2 in projection area 3.

[0378] For example, the first interface may be a split-screen live broadcast interface of a display device, such as Figure 13 Interface 1205 shown in (e).

[0379] Among them, the process of the smart TV 2 playing the media resource 2 in the projection area 3 (ie S628) can refer to the above-mentioned process of playing the media resource 1 in the projection area 2 (ie S601 to S620).

[0380] The above is a schematic description of the overall process of the screen projection method provided in the embodiment of the present application. Below, taking the split screen projection of the media resource 1 currently displayed on the mobile phone 1 to the projection area 2 as an example, the split screen projection process (i.e., the above S611 to S616) is described in detail.

[0381] Figure 25 This is a flowchart of a screen projection method provided in an embodiment of the present application. Figure 25 As shown, the screen projection method may include the following steps:

[0382] S701. In response to action generation request 1, the management service module in the intelligent agent sends recognition request 1 to the recognition service module, and the recognition request 1 carries a first screenshot image.

[0383] S702: The recognition service module sends a recognition request 2 to the recognition model, where the recognition request 1 carries the first screenshot image.

[0384] S703 : The recognition model recognizes the first screenshot image to obtain a recognition result 1 , which includes a plurality of interface elements 1 and coordinates 1 of each interface element 1 .

[0385] S704: The recognition model sends recognition result 1 to the management service module.

[0386] S705. The management service module sends a page matching request to the page matching module in the intelligent agent. The page matching request carries recognition result 1 and instruction type 1.

[0387] S706. The page matching module obtains knowledge graph 1 according to instruction type 1.

[0388] S707. The page matching module determines whether there is a first preset interface that matches the first projection interface based on the knowledge graph 1.

[0389] S708. When there is a first preset interface that matches the first projection interface, the page matching module determines the interface jump position 1 corresponding to the first preset interface and the designated key element 1 in the first preset interface.

[0390] S709. The page matching module sends a path planning request to the path planning module in the intelligent agent. The path planning request carries the interface jump position 1 and the specified key element 1.

[0391] S710. The path planning module obtains the action type 1 corresponding to the interface jump position 1 according to the knowledge graph 1.

[0392] S711 , the path planning module generates action information 1 according to the action type 1 and the preset coordinates corresponding to the designated key element 1 .

[0393] S712. The path planning module sends action information 1 to the management service module.

[0394] S713. The management service module sends action information 1 to the integrated screen projection application of smart TV 2.

[0395] S714. The fusion projection application parses and converts the action information 1 to generate a reverse control instruction corresponding to the action information 1.

[0396] S715. The fusion screen projection application sends a reverse control instruction corresponding to action information 1 to mobile phone 1.

[0397] When the interface jump position 1 indicates that the current screen projection interface is the device selection interface, the reverse control instruction corresponding to the action information 1 is used to control the mobile phone 1 to initiate a DLNA screen projection connection to the smart TV 2, that is, the reverse control instruction corresponding to the action information 1 is the first reverse control instruction.

[0398] In some examples, when the interface jump position 1 indicates that the current screen projection interface is the screen projection completion interface, the path planning module can obtain a pre-stored preset reverse control instruction corresponding to the preset jump position. Among them, the preset reverse control instruction can be used to control mobile phone 1 to cancel the display of the screen projection pop-up window and control mobile phone 1 to switch screen content.

[0399] In some examples, if there is no first preset interface that matches the first projection interface, the page matching module sends an identification error message to the management service module. The management service module sends the identification error message to the fusion projection application. In response to the error identification message, the fusion projection application stops the screenshot operation.

[0400] S716. The management service module obtains the interface jump position 1 from the page matching module, and determines whether the interface jump position 1 is a preset jump position.

[0401] Among them, the preset jump position indicates that the screen content of the terminal device jumps to the last preset interface in the knowledge graph 1, such as the screen projection completion interface.

[0402] S717. When the interface jump position 1 is not the preset jump position, the management service module sends notification information 1 to the fusion screen projection application. Notification information 1 is used to notify the fusion screen projection application 2 that the split-screen projection instruction has not been executed.

[0403] S718. In response to notification information 1, the integrated screen projection application takes a screenshot of the content currently displayed on the smart TV 2.

[0404] After S718, the fusion projection application sends a new screenshot image to the intelligent agent to obtain new action information from the intelligent agent, that is, repeating S702 to S716.

[0405] S719 . When the interface jump position 1 is the preset jump position, the management service module determines whether there is an interface element matching the device information of the smart TV 1 in the recognition result 1 .

[0406] S720. The management service module finds that there is an interface element matching the device information of smart TV 1 in recognition result 1, and sends notification information 2 to the fusion screen projection application. Notification information 2 is used to notify fusion screen projection application 2 that the split-screen projection instruction has been executed.

[0407] S721. In response to notification information 2, the fusion screen projection application stops the screenshot operation.

[0408] Next, we will continue to use the example of splitting the screen of the media resource 1 currently displayed on the mobile phone 1 to the screen projection area 2 to explain the screen projection method in detail.

[0409] Figure 26This is a flowchart of a screen projection method provided in an embodiment of the present application. Figure 26 As shown, the screen projection method may include the following steps:

[0410] S801. In response to the opening operation 1, the remote controller sends an opening instruction 1 to the screen projection playback interface of the fusion application.

[0411] S802. In response to the open instruction 1, the screen projection playback interface starts the guide page, that is, the multi-channel screen projection mode is started.

[0412] The default boot page CastPlayerActivity is started through Android's native Activity startup method. After the application is started, the View is initialized. When entering the application, the default boot page will be displayed, which will display some device information, such as the device name of the current large-screen terminal and the corresponding network information. The boot page is displayed in left and right proportional proportions through the Android standard controls TextView and ImageView and the corresponding Layout.

[0413] At the same time, CastPlayerActivity initializes the internal NetAgentProgress object to prepare for automated screen casting. The NetAgentProgress object primarily communicates with the Agent over the network, parses the Agent's commands, and sends them to the terminal device via the screen casting reverse control protocol channel.

[0414] S803. The screen projection playback interface sends notification information 3 to the view display module. The notification information 1 carries the guide page data, which is used to notify the display of the guide page.

[0415] S804: The view display module draws and displays the guide page according to the guide page data.

[0416] S805. The screen projection playback interface sends notification information 4 to the screen projection service to notify the start of the multi-channel screen projection mode.

[0417] S806 , in response to the opening operation 2 , the mobile phone 1 opens the live broadcast application and displays the screen content 01 , in which the live broadcast picture of the media resource 1 is presented.

[0418] S807. In response to screen casting operation 1, mobile phone 1 sends a Miracast screen casting request to the screen casting service.

[0419] S808. In response to the Miracast screen projection request, mobile phone 1 establishes a mirror screen projection connection with the screen projection service, that is, mobile phone 1 establishes a mirror screen projection connection with smart TV 2.

[0420] Specifically, after the application page (i.e., the screen projection playback interface) is started, the corresponding screen projection service CastServer will be started, and some initialization preparations will be done, such as starting the monitoring mirror or pushing the corresponding protocol service; after CastServer is started, the listeners of UsesionConnectionListener, UsesionPrepareListener, USessionDisplayListener and UsesionEventListener will be registered.

[0421] The screen casting service is always ready to connect to new sessions. When Phone 1 is ready to connect, the screen casting service listens to the new session's onConnect connection through the UsessionConnectionListener() method. It then monitors the current casting content type, such as MiraCast mirroring, and notifies the casting application CastPlayerActivity through the protocol layer's onConnect callback message to create the playback control SurfaceView / TextureView. After successful creation, the ViewModule class is started to set and manage the display position of the current session. To facilitate View and Session management, this application sets a management object MultiCastPlayView, which associates Session, SurfaceView / TextureView, mViewPosition, and other information. When CastPlayerActivity prepares the View, it notifies the mobile end through UsesionPrepareListener that it can record or send the video stream. When the mobile end is ready and needs to send video information, the server end learns from onFirstFrame() of UsesionDisplayListener that the large screen end is going to play, obtains the current Client1 video width and height from the Session, determines the direction of the current video stream based on the width and height, and then presents the current projection content through the ViewModule class.

[0422] S809. The screen projection service sends notification information 5 to the screen projection playback interface to notify the use of multi-channel screen projection mode for split-screen projection.

[0423] S810. Mobile phone 1 is connected via mirroring and screen projection, and sends audio and video data 1 corresponding to the screen content 01 of mobile phone 1 to the screen projection service.

[0424] S811. The screen projection service sends audio and video data 1 to the screen projection playback interface.

[0425] S812. The screen projection playback interface determines the view data 1 of the screen projection interface 1 corresponding to the screen content 01 according to the audio and video data 1, and determines the target display position 1 of the view data 1 (that is, determines the screen projection area 1).

[0426] S813. The screen projection playback interface sends notification information 6 to the view display module. The notification information 6 carries view data 1 and target display position 1, which is used to notify the display of the screen projection interface 1.

[0427] S814. The view display module displays interface 1, which includes projection area 1 and projection area 2. According to view data 1, projection interface 1 is drawn and displayed in projection area 1.

[0428] S815. The screen projection playback interface sends notification information 7 to the player, which is used to notify the player to play the audio and video data 1. The notification information 7 carries the audio and video data 1.

[0429] S816. The player plays the audio and video data 1.

[0430] S817. In response to the screen projection operation 2, the remote control sends a split-screen projection instruction 1 to the screen projection playback interface. The split-screen projection instruction 1 is used to instruct the smart TV 2 to perform split-screen projection.

[0431] S818. In response to the split-screen projection instruction 1, the screen projection playback interface obtains the device information of the smart TV 2 and the instruction type 1 of the split-screen projection instruction 1.

[0432] S819: The screen projection playback interface takes a screenshot of the content currently displayed on the smart TV 2 to obtain a first screenshot image.

[0433] S820. The screen casting and playback interface sends an action generation request 1 to the network request module. The action generation request 1 carries the first screenshot image, the instruction type 1, and the device information 1 of the smart TV 2.

[0434] S821. The network request module sends an action generation request 1 to the intelligent agent. The action generation request 1 carries the first screenshot image, instruction type 1 and device information 1.

[0435] The display device application CastPlayerActivity will send a request for screen casting command identification through the NetAgentProgress object. The request method is an HTTP request, which carries the byte data stream of the screen casting content screenshot, network request configuration items, the device name of the current target TV to cast the screen, signature verification information, etc.

[0436] S822. In response to action generation request 1, the intelligent agent uses the recognition model to recognize the first screenshot image and obtains recognition result 1.

[0437] S823. The intelligent agent obtains the knowledge graph 1 according to the instruction type 1, and generates the action information 1 based on the knowledge graph 1 and the recognition result 1.

[0438] S824. The agent sends action information 1 to the network request module.

[0439] S825. The network request module sends action information 1 to the screen projection playback interface.

[0440] S826. The screen projection playback interface generates a first reverse control instruction according to action information 1.

[0441] Taking the Click event as an example, the NetAgentProgress message is called back to the CastPlayerActivity layer. After CastPlayerActivity parses the corresponding controlFlow information, it performs standard conversion processing of corresponding MotionEvent, KeyEvent and other instructions. The mobile phone can receive and process the standard MotionEvent, KeyEvent and other events.

[0442] S827. The screen projection playback interface sends a first counter-control instruction to mobile phone 1.

[0443] S828. In response to the first counter-control instruction, mobile phone 1 sends a DLNA screen projection request 1 to the screen projection service.

[0444] S829. In response to DLNA screen casting request 1, DLNA screen casting connection 1 is established between mobile phone 1 and screen casting service, that is, mobile phone 1 establishes DLNA screen casting connection 1 with smart TV 2.

[0445] S830. The screen projection service sends notification information 8 to the screen projection playback interface to notify the use of multi-channel screen projection mode for split-screen projection.

[0446] S831. In response to notification information 8, the screen casting playback interface determines the target display position 2 corresponding to the DLNA screen casting connection 1.

[0447] S832. Mobile phone 1 sends screen projection instruction 1 to the screen projection service through DLNA screen projection connection 1. The screen projection instruction 1 carries the resource address 1 of the media resource 1.

[0448] S833. The screen projection service sends resource address 1 to the screen projection playback interface.

[0449] S834. The screen projection playback interface obtains media resource 1 according to resource address 1.

[0450] S835. The screen projection playback interface sends view data 2 and target display position 2 of media resource 1 to the view display module.

[0451] S836. The view display module draws and displays the image of the media resource 1 in the projection area 2 according to the view data 2 of the media resource 1 and the target display position 2.

[0452] S837. The screen projection playback interface sends a notification message 9 to the player, and the notification message 9 carries the audio and video data 2 of the media resource 1.

[0453] S838. The player caches the audio and video data 2 of the media resource 1.

[0454] The present disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the various processes of the Bluetooth connection method described above and achieves the same technical effect. To avoid repetition, the details are not described here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0455] The present disclosure provides a computer program product, which includes a computer program. When the computer program is run on a computer, the computer is enabled to implement the above-mentioned Bluetooth connection method.

[0456] For ease of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion of some embodiments is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0457] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0458] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0459] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0460] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0461] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of units is merely a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0462] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0463] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0464] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A screen projection method, characterized in that: Applied to a display device, the method includes: In the multi-channel screen projection mode, the terminal device receives the first screen content currently displayed on the terminal device sent by the mirroring screen projection connection, displays a first interface, wherein the first interface includes a first projection area and a second projection area, and displays a first projection interface corresponding to the first screen content in the first projection area, wherein the first projection interface includes a first specified element, and the first specified element corresponds to a first control in the first screen content; In response to a first control instruction sent by the control device, taking a screenshot of content currently displayed by the display device to obtain a first screenshot image; Obtaining first action information corresponding to the first designated element, where the first action information is determined by identifying the first designated element and first position information of the first designated element in the first screenshot image, and based on the first position information and the action type corresponding to the first designated element; Generate a first reverse control instruction based on the first action information, where the first reverse control instruction is used to control the terminal device to initiate a first DLNA screen casting connection to the display device, and send a first screen casting instruction to the display device through the first DLNA screen casting connection, where the first screen casting instruction carries a first resource address, where the first resource address is a resource address of a first media resource corresponding to the first screen content currently displayed by the terminal device; Sending the first reverse control instruction to the terminal device through the mirror screen projection connection; In response to the first screen projection instruction sent by the terminal device, the first media resource is obtained according to the first resource address, and the first media resource is played in the second screen projection area.

2. The method according to claim 1, characterized in that The first control is not a preset screen projection control, and the preset screen projection control is used to trigger the terminal device to initiate the first DLNA screen projection connection to the display device. The generating a first counter-control instruction according to the first action information includes: generating a second reverse control instruction according to the first action information; Sending the second reverse control instruction to the terminal device through the mirror screen projection connection; Receiving the second screen content sent by the terminal device in response to the second counter-control instruction, and displaying a second screen projection interface corresponding to the second screen content in the first screen projection area, where the second screen projection interface includes a second designated element, and the second designated element corresponds to the preset screen projection control in the second screen content; Taking a screenshot of the content currently displayed on the display device to obtain a second screenshot image; Obtaining second action information corresponding to the second designated element, where the second action information is determined by identifying the second designated element and second position information of the second designated element in the second screenshot image, and based on the second position information and the action type corresponding to the second designated element; Generate the first reverse control instruction according to the second action information.

3. The method according to claim 1, characterized in that The first control instruction is an instruction to drag the first projection interface displayed in the first projection area to the second projection area.

4. The method according to claim 3, characterized in that After receiving the first screen content currently displayed by the terminal device sent by the terminal device via the mirroring screen projection connection and displaying the first interface, the method further includes: In response to a cursor movement instruction sent by the control device, displaying a pointing cursor in the first interface and adjusting a position of the pointing cursor in the first interface; When the pointing cursor is located in the first projection area, a drag control is displayed in the first projection area, the drag control is located on an upper layer of the first projection interface, and the drag control is used to drag the first projection interface; Among them, the first control instruction is: selecting the drag control through the pointing cursor, and dragging the first projection interface to the second projection area through the drag control, and deselecting the drag control after dragging the first projection interface to the second projection area.

5. The method according to claim 4, characterized in that When the pointing cursor is located in the first projection area and is not on the drag control, the display state of the drag control is unselected; When the pointing cursor is located on the drag control, the display state of the drag control is switched from an unselected state to a selected state.

6. The method according to claim 4, characterized in that When a drag control is displayed in the first projection area, first prompt information is presented on the drag control, and the first prompt information is used to prompt the first projection interface to be dragged to the second projection area through the drag control.

7. The method according to any one of claims 1 to 6, characterized in that After playing the first media resource in the second projection area, the method further includes: Sending a third reverse control instruction to the terminal device through the mirroring screen projection connection, so that the terminal device displays a third screen content in response to the third reverse control instruction, where the third screen content includes a third control and corresponds to a second media resource; Receive the third screen content sent by the terminal device through the mirror projection connection, and display the third projection interface corresponding to the third screen content in the first projection area, wherein the third projection interface includes a third designated element, and the third designated element corresponds to the third control.

8. The method according to claim 7, characterized in that After displaying the third projection interface corresponding to the third screen content in the first projection area, the method further includes: In response to a second control instruction sent by the control device, taking a screenshot of content currently displayed by the display device to obtain a third screenshot image; Obtaining third action information corresponding to the third designated element, where the third action information is determined by identifying the third designated element and third position information of the third designated element in the third screenshot image, and based on the third position information and the action type corresponding to the third designated element; Generate a fourth reverse control instruction based on the third action information, wherein the fourth reverse control instruction is used to control the terminal device to initiate a second DLNA screen casting connection to the display device, and send a second screen casting instruction to the display device through the second DLNA screen casting connection, wherein the second screen casting instruction carries a second resource address, and the second resource address is a resource address of a second media resource corresponding to the third screen content currently displayed by the terminal device; Sending the fourth reverse control instruction to the terminal device through the mirror screen projection connection; In response to the second screen projection instruction sent by the terminal device, obtaining the second media resource according to the second resource address; Display a second interface, which includes the first projection area, the second projection area and the third projection area, and play the second media resource in the third projection area.

9. The method according to claim 8, characterized in that After playing the second media resource in the third projection area, the method further includes: Sending a fifth reverse control instruction to the terminal device through the mirroring screen projection connection, so that the terminal device displays fourth screen content in response to the fifth reverse control instruction, where the fourth screen content corresponds to a third media resource; Receive the fourth screen content sent by the terminal device through the mirror projection connection, and display the fourth projection interface corresponding to the fourth screen content in the first projection area, wherein the fourth projection interface includes the resource screen of the third media resource.

10. The method according to claim 9, characterized in that The first media resource, the second media resource, and the third media resource are all live broadcast media resources.

11. A screen projection method, characterized in that: Applied to an intelligent agent, the method comprises: Receiving a first screenshot image sent by a display device, where the first screenshot image is a screenshot image acquired by the display device in response to a first control instruction, the first screenshot image including a first projection interface displayed by the display device in a first projection area, the first projection interface corresponding to first screen content currently displayed on the terminal device, the first projection interface including a first designated element, and the first designated element corresponding to a first control in the first screen content; identifying the first designated element and first position information of the first designated element in the first screenshot image; determining first action information according to the first position information of the first designated element and the action type corresponding to the first designated element; The first action information is sent to the display device so that the display device generates a first reverse control instruction based on the first action information, where the first reverse control instruction is used to control the terminal device to initiate a first DLNA screen projection connection to the display device, and send a first screen projection instruction to the display device through the first DLNA screen projection connection. The first screen projection instruction carries a first resource address, which is the resource address of the first media resource corresponding to the first screen content currently displayed by the terminal device.

12. The method according to claim 11, characterized in that The identifying the first designated element and the first position information of the first designated element in the first screenshot image includes: Obtaining preset path data; wherein the preset path data includes interface information of each of a plurality of preset interfaces and data indicating jump relationships between the plurality of preset interfaces; the interface information includes a plurality of preset elements in the preset interfaces, preset position information of each of the preset elements, and a preset action type corresponding to a specified preset element among the plurality of preset elements; Identify the first projection interface in the first screenshot image to obtain at least one first interface element and position information of each first interface element; For each of the preset interfaces, based on the first interface element and the position information of the first interface element, the at least one first interface element is matched with the multiple preset elements in the preset interface to determine the first preset interface corresponding to the first projection interface, and the designated preset element in the first preset interface is the first designated element; The preset position information of the designated preset element in the first preset interface is determined as the first position information, and the preset action type corresponding to the designated preset element in the first preset interface is determined as the action type corresponding to the first designated element.

13. The method according to claim 12, characterized in that The interface information further includes weight parameters corresponding to the preset elements; and for each of the preset interfaces, based on the first interface element and the position information of the first interface element, matching the at least one first interface element with multiple preset elements in the preset interface to determine the first preset interface corresponding to the first projection interface includes: For each of the preset interfaces, comparing the first interface element with a first preset element, and comparing the position information of the first interface element with the preset position information of the preset element, to determine a second preset element, where the first preset element is any one of the multiple preset elements, and the second preset element is a preset element among the multiple preset elements of the preset interface that matches the first interface element; determining a matching degree corresponding to the preset interface according to a weight parameter corresponding to the second preset element and the number of preset elements in the preset interface; The first preset interface is determined according to the matching degree corresponding to each of the preset interfaces, wherein the matching degree corresponding to the first preset interface is the maximum value among the matching degrees corresponding to the multiple preset interfaces, and the matching degree corresponding to the first preset interface is greater than the matching degree threshold corresponding to the first preset interface.

14. A display device, characterized in that: include: monitor; A controller connected to the display, the controller being configured to: In the multi-channel screen projection mode, the terminal device receives the first screen content currently displayed on the terminal device sent by the mirroring screen projection connection, displays a first interface, wherein the first interface includes a first projection area and a second projection area, and displays a first projection interface corresponding to the first screen content in the first projection area, wherein the first projection interface includes a first specified element, and the first specified element corresponds to a first control in the first screen content; In response to a first control instruction sent by the control device, taking a screenshot of content currently displayed by the display device to obtain a first screenshot image; Obtaining first action information corresponding to the first designated element, where the first action information is determined by identifying the first designated element and first position information of the first designated element in the first screenshot image, and based on the first position information and the action type corresponding to the first designated element; Generate a first reverse control instruction based on the first action information, where the first reverse control instruction is used to control the terminal device to initiate a first DLNA screen casting connection to the display device, and send a first screen casting instruction to the display device through the first DLNA screen casting connection, where the first screen casting instruction carries a first resource address, where the first resource address is a resource address of a first media resource corresponding to the first screen content currently displayed by the terminal device; Sending the first reverse control instruction to the terminal device through the mirror screen projection connection; In response to the first screen projection instruction sent by the terminal device, the first media resource is obtained according to the first resource address, and the first media resource is played in the second screen projection area.

Images