Screen projection method and display equipment

Abstract

The invention discloses a screen projection method and a display device, and relates to the technical field of display device.The method comprises the steps that in a multi-path screen projection mode, first screen content, sent by the terminal device through mirror image screen projection connection, of the terminal device is received, and a first screen projection interface corresponding to the first screen content is displayed in a first screen projection area; in response to the target control instruction, performing screenshot on content currently displayed by the display equipment to obtain a first screenshot image; acquiring first action information; generating a first anti-control instruction according to the first action information, and sending the first anti-control instruction to the terminal device, the first anti-control instruction being used for controlling the terminal device to search and display a first media resource corresponding to the target search information; second screen content sent by the terminal equipment is received, a second screen projection interface corresponding to the second screen content is displayed in the first screen projection area, and the second screen content is resource content of the first media resource. According to the invention, the terminal equipment can be reversely controlled to carry out screen projection by controlling the display equipment.

Description

[0001] This application claims priority to Chinese Patent Application No. 202510571730.5, filed with the Chinese Patent Office on April 30, 2025, the contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of display device technology, and in particular to a projection method and display device. Background Technology

[0003] With the continuous development of terminal technology, multi-screen interaction technology is becoming increasingly mature. For example, users can project content displayed on a small-screen device (such as a mobile phone) onto a large-screen device (such as a smart TV) for better display effects and viewing experience.

[0004] When casting content displayed on a small screen device to a large screen device, users need to manually search for the content they want to watch and then perform a series of operations before the content can be cast to the large screen device.

[0005] However, this approach causes users to repeatedly switch their visual focus between large-screen and small-screen devices, and the operation is cumbersome, resulting in a poor user experience. Summary of the Invention

[0006] This application provides a screen casting method and display device that eliminates the need for manual searching by the user. Instead, the user only needs to trigger a search and screen casting command. The display device can then control the terminal device to search for and display media resources corresponding to the target search information, thereby simplifying the operation process and improving the user experience.

[0007] In a first aspect, some embodiments of this application provide a screen mirroring method applied to a display device. The method includes: in a multi-channel screen mirroring mode, receiving the first screen content currently displayed on the terminal device sent by the terminal device via a mirrored screen mirroring connection; displaying a first interface, the first interface including a first mirroring area; displaying a first mirroring interface corresponding to the first screen content in the first mirroring area; the first mirroring interface including a first specified element, the first specified element corresponding to a first control in the first screen content; responding to a target control command sent by a control device, taking a screenshot of the content currently displayed on the display device to obtain a first screenshot image; and acquiring a first action corresponding to the first specified element. The information, first action information, is determined by identifying a first specified element and its first position information in a first screenshot image, and based on the first position information, the action type corresponding to the first specified element, and the target search information; according to the first action information, a first reverse control command is generated and sent to the terminal device via a mirroring connection. The first reverse control command is used to control the terminal device to search for and display the first media resource corresponding to the target search information; the second screen content sent by the terminal device in response to the first reverse control command is received, and the second screen interface corresponding to the second screen content is displayed in the first projection area. The second screen content is the resource content of the first media resource.

[0008] Secondly, some embodiments of this application provide a screen projection method applied to an intelligent agent. The method includes: receiving a first screenshot image sent by a display device, the first screenshot image being a screenshot image acquired by the display device in response to a target control command, the first screenshot image including a first projection interface displayed by the display device in a first projection area, the first projection interface corresponding to the first screen content currently displayed by the terminal device, the first projection interface including a first specified element, the first specified element corresponding to a first control in the first screen content; identifying the first specified element and its first position information in the first screenshot image; determining first action information based on target search information, the first position information of the first specified element, and the action type corresponding to the first specified element; and sending the first action information to the display device to cause the display device to generate a first counter-control command based on the first action information, the first counter-control command being used to control the terminal device to search for and display a first media resource corresponding to the target search information.

[0009] Thirdly, some embodiments of this application also provide a display device, including a display; and a controller connected to the display, the controller being configured to: in a multi-channel projection mode, receive the first screen content currently displayed by the terminal device sent by the terminal device via a mirror projection connection, display a first interface, the first interface including a first projection area, display a first projection interface corresponding to the first screen content in the first projection area, the first projection interface including a first designated element, the first designated element corresponding to a first control in the first screen content; in response to a target control command sent by the control device, take a screenshot of the content currently displayed by the display device to obtain a first screenshot image; and acquire the first designated element. The corresponding first action information is determined by identifying the first specified element and the first position information of the first specified element in the first screenshot image, and based on the first position information, the action type corresponding to the first specified element, and the target search information; according to the first action information, a first reverse control command is generated and sent to the terminal device through a mirroring connection. The first reverse control command is used to control the terminal device to search for and display the first media resource corresponding to the target search information; the second screen content sent by the terminal device in response to the first reverse control command is received, and the second screen interface corresponding to the second screen content is displayed in the first screen projection area. The second screen content is the resource content of the first media resource.

[0010] Fourthly, some embodiments of this application also provide an electronic device, including: a receiving module, configured to receive a first screenshot image sent by a display device, the first screenshot image being a screenshot image acquired by the display device in response to a target control command, the first screenshot image including a first projection interface displayed by the display device in a first projection area, the first projection interface corresponding to the first screen content currently displayed by the terminal device, the first projection interface including a first designated element, the first designated element corresponding to a first control in the first screen content; an identification module, configured to identify the first designated element in the first screenshot image and the first position information of the first designated element; an action generation module, configured to determine first action information based on target search information, the first position information of the first designated element, and the action type corresponding to the first designated element; and a sending module, configured to send the first action information to the display device, so that the display device generates a first counter-control command based on the first action information, the first counter-control command being used to control the terminal device to search for and display a first media resource corresponding to the target search information.

[0011] Fifthly, some embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the screen projection method described in the first aspect, or causes the processor to perform the screen projection method described in the second aspect.

[0012] In a sixth aspect, some embodiments of this application provide a computer program product, which includes computer program code that, when executed on a display device, causes the display device to perform the screen projection method described in the first aspect.

[0013] In this embodiment, the terminal device and the display device establish a mirrored screen projection connection, with the display device showing the screen content of the terminal device in projection area 1. In response to a target control command sent to the remote control, the display device takes a screenshot of its currently displayed content and identifies and analyzes the screenshot to generate a reverse control command. The display device sends the reverse control command to the terminal device via the mirrored screen projection connection, controlling the terminal device to respond to the command, search for and display media resources corresponding to the target search information, and send its currently displayed screen content to the display device via the mirrored screen projection connection to display the media resources corresponding to the target search information in projection area 1. This eliminates the need for manual searching by the user; simply triggering the search and projection command allows the display device to reverse-control the terminal device to search for and display media resources corresponding to the target search information, simplifying the operation process and improving the user experience. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device, provided as an embodiment of this application.

[0016] Figure 2 A hardware configuration block diagram of a display device provided in an embodiment of this application;

[0017] Figure 3 This application provides a schematic diagram of the software system architecture of a display device.

[0018] Figure 4 This is a software schematic diagram of an intelligent agent provided in an embodiment of this application;

[0019] Figure 5 A schematic diagram illustrating a screen mirroring application scenario provided in an embodiment of this application;

[0020] Figure 6 A schematic diagram of the interaction process for enabling a split-screen projection function provided in an embodiment of this application;

[0021] Figure 7 A schematic diagram of the interactive flow of a screen mirroring process provided in an embodiment of this application;

[0022] Figure 8 One of the interactive flow diagrams of the search and projection process provided in the embodiments of this application;

[0023] Figure 9 The second interactive flowchart of the search and projection process provided in the embodiments of this application;

[0024] Figure 10 The third interactive flowchart of the search and projection process provided in the embodiments of this application;

[0025] Figure 11 The fourth interactive flowchart of the search and projection process provided in the embodiments of this application;

[0026] Figure 12 A schematic diagram of a knowledge graph provided in an embodiment of this application;

[0027] Figure 13 One of the flowcharts of the screen projection method provided in the embodiments of this application;

[0028] Figure 14 The second schematic flowchart of the screen projection method provided in the embodiments of this application;

[0029] Figure 15 The third flowchart illustrating the screen projection method provided in this application embodiment;

[0030] Figure 16 The fourth flowchart illustrating the screen projection method provided in this application embodiment;

[0031] Figure 17 Fifth flowchart illustrating the screen projection method provided in this application embodiment;

[0032] Figure 18 This is the sixth flowchart illustrating the screen projection method provided in the embodiments of this application. Detailed Implementation

[0033] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0034] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0035] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0036] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0037] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0038] To better understand the embodiments of this application, the terms or concepts that may be involved in the embodiments of this application are explained below.

[0039] 1. Screen mirroring

[0040] Screen mirroring (also known as system screen mirroring) refers to a technology that allows the sending device (also known as the source device) and the receiving device (also known as the target device) to synchronize their screens based on protocols such as Miracast when they are on the same local area network.

[0041] Taking mobile phone 1 as the sending device and smart TV 2 as the receiving device as an example, after mobile phone 1 and smart TV 2 establish a connection through WiFi network, in response to the screen mirroring command, mobile phone 1 scans for available receiving devices nearby. After mobile phone 1 discovers smart TV 2, it can establish a connection with smart TV 2 based on the Miracast protocol. After the connection is successful, mobile phone 1 can project its screen display content onto smart TV 2 for display.

[0042] Understandably, during screen mirroring, the content displayed on the receiving device's screen will change as the content displayed on the sending device's screen changes, and the content displayed on the receiving device's screen will always remain consistent with the content displayed on the sending device's screen.

[0043] 2. Streaming media casting

[0044] Streaming media casting refers to the technology where the sending and receiving devices are on the same local area network, and the sending and receiving devices use a protocol authorized by the Digital Living Network Alliance (DLNA) (referred to as the DLNA protocol) to push streaming media resources.

[0045] Continuing with the example of mobile phone 1 as the sending device and smart TV 2 as the receiving device, when mobile phone 1 and smart TV 2 are on the same local area network, when mobile phone 1 plays media resources (such as videos) through an application that supports the DLNA protocol, in response to the streaming media casting command, mobile phone 1 can negotiate with smart TV 2 to cast the media resources based on the DLNA protocol. After the negotiation is completed, mobile phone 1 can send the link address of the media resource it is playing to smart TV 2. Smart TV 2 obtains the media resource from the server according to the link address and displays the media resource on the screen of smart TV 2, thereby realizing the push casting of streaming media resources.

[0046] Understandably, during the streaming media casting process, the receiving device can directly obtain the media resources and play them independently. Therefore, operations on the sending device will not affect the playback of media resources on the receiving device. This allows users to watch the cast video on the receiving device while performing other application operations on the sending device.

[0047] In addition to Miracast and DLNA protocols mentioned above, AirPlay is also a common screen mirroring protocol. In practical applications, AirPlay can support both streaming media mirroring and screen mirroring.

[0048] 3. Reverse control

[0049] Reverse control refers to the technology where the receiving device remotely controls the sending device through a reverse control protocol during screen projection.

[0050] Continuing with the example of mobile phone 1 as the sending device and smart TV 2 as the receiving device, during the screen mirroring process, when mobile phone 1 and smart TV 2 negotiate the reverse control protocol, after the negotiation is completed, the user can control the mobile phone in reverse by operating the smart TV, so that the screen display content of smart TV 2 and the screen display content of mobile phone 1 are kept synchronized.

[0051] It should be noted that in the embodiments of this application, the display device can simultaneously support screen mirroring (such as screen mirroring based on the Miracast protocol) and multi-channel streaming media screen mirroring (such as screen mirroring based on the DLNA protocol), wherein screen mirroring supports the reverse control protocol.

[0052] The terms or concepts involved in the embodiments of this application have been described above. The technical solutions of the embodiments of this application will be described below.

[0053] With the continuous development of terminal technology, multi-screen interaction technology is becoming increasingly mature. For example, users can project content displayed on a small-screen device (such as a mobile phone) onto a large-screen device (such as a smart TV) for better display effects and viewing experience.

[0054] When casting content displayed on a small screen device to a large screen device, users need to manually search for the content they want to watch and then perform a series of operations before the content can be cast to the large screen device.

[0055] However, this approach causes users to repeatedly switch their visual focus between large-screen and small-screen devices, and the operation is cumbersome, resulting in a poor user experience.

[0056] In view of this, this application provides a screen mirroring method applied to a display device. A mirrored screen mirroring connection is established between a terminal device and the display device, with the display device showing the screen content of the terminal device in the mirroring area 1. In response to a search and screen mirroring command sent to a remote control, the display device takes a screenshot of its currently displayed content and identifies and analyzes the screenshot to generate a reverse control command. The display device sends the reverse control command to the terminal device via the mirrored screen mirroring connection, controlling the terminal device to search for and display media resources corresponding to the target search information. The terminal device then sends its currently displayed screen content to the display device via the mirrored screen mirroring connection to display the media resources corresponding to the target search information in the mirroring area 1. This eliminates the need for manual searching by the user; simply triggering the search and screen mirroring command allows the display device to reverse-control the terminal device to search for and display media resources corresponding to the target search information, simplifying the operation process and improving the user experience.

[0057] Furthermore, after displaying the screen content of the terminal device in projection area 1, the display device takes a screenshot of the currently displayed content and identifies and analyzes the screenshot to generate a reverse control command. The display device sends the reverse control command to the terminal device via a mirrored projection connection. This command instructs the terminal device to initiate a DLNA projection connection to the display device and send a first projection command, which carries a first resource address—the resource address of the first media resource corresponding to the target search information. Upon receiving the first projection command from the terminal device, the display device retrieves the first media resource based on the first resource address and plays it in projection area 2. This eliminates the need for cumbersome user operations; the display device can reverse-control the terminal device to project its currently displayed media resource onto the display device. This simplifies the operation process, avoids the user's visual focus repeatedly switching between the display and terminal devices, and improves the user experience.

[0058] The following describes the display devices to which the screen projection method provided in the embodiments of this application is adapted.

[0059] In this application embodiment, "display device" refers to a device with screen display and data processing capabilities. For example, display devices include, but are not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, and augmented reality devices.

[0060] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device, provided as an embodiment of this application. Figure 1 As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. The control device is used to perform human-computer interaction between the user and the display device 200.

[0061] In this application, the control device can be a remote control with a specific pointing function. Here, the control device can be called a pointing remote control. The pointing remote control introduces a brand-new interaction method, gets rid of the operation limitations of traditional remote controls, and operates like a laser pointer. It can accurately point to the target control without having to switch icons step by step, which is efficient and convenient.

[0062] In some embodiments, the remote control has a pointing switch on its body. When the pointing switch is turned on, it enters pointing mode, and the user can point, slide, and perform other operations using a displayed object (such as a cursor). When the pointing switch is turned off, the remote control switches from pointing mode to traditional button mode. In traditional button mode, the user can use the up / down / left / right / confirm buttons.

[0063] In some embodiments, after turning on the pointing switch and entering the pointing mode, it can coexist with the traditional button mode. Users can perform pointing, sliding, and clicking operations through the displayed object (such as the cursor), or use the confirmation key and the up / down / left / right / volume keys.

[0064] In some embodiments, the remote control is in a low-power mode in traditional button mode; however, when the pointing switch is turned on and pointing mode is entered, the remote control is no longer in low-power mode. Therefore, in some embodiments, a charging interface, such as a Type-C interface, can be configured for the remote control.

[0065] In some embodiments, the pointing remote control is a remote control device capable of detecting its own position and orientation in three-dimensional space. The pointing remote control has a built-in gyroscope for detecting the remote control's rotation angle and its movement acceleration. Through sensors, the pointing remote control can determine its pointing position in three-dimensional space in real time. The pointing position can include position coordinates and pointing direction, where the position coordinates are the physical position of the remote control relative to the display device (e.g., horizontal distance, vertical height, etc.). The pointing direction is the pointing angle of the remote control (e.g., horizontal and vertical angles relative to the display device).

[0066] In some embodiments, after the remote control obtains its pointing position in three-dimensional space, it can send the pointing position data to the display device. This allows the display device to obtain the pointing position and perform screen coordinate transformation, thus mapping the pointing position of the remote control to the screen coordinate system. At this point, the display device updates the position of the displayed object (e.g., the cursor) based on the calculated screen coordinates. When the user moves the remote control, the remote control's sensors continuously detect changes in position and direction and send the data to the display device. The display device updates the position of the displayed object in real time, and the cursor moves in the direction pointed by the remote control.

[0067] In some embodiments, the pointing remote control can control the displayed objects on the user interface. The cursor can be moved by pointing the remote control at different areas of the screen for precise selection. The pointing remote control can switch between multiple modes: when the pointing switch is on, the pointing function is enabled, allowing operations such as pointing, swiping, and clicking on the displayed objects; when the pointing switch is off, the pointing function is disabled. When the pointing function is disabled, the pointing remote control can operate in button mode, allowing control of the display device via traditional button operations. The pointing remote control supports various operation methods such as skipping, tapping, circling, swiping, and dragging, making content browsing more convenient.

[0068] In some embodiments, when the pointing function of the pointing remote control is enabled, the OK button among the ordinary buttons on the pointing remote control can be shared in the pointing function.

[0069] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0070] like Figure 1 As shown, the display device 200 also communicates with the server 400 via various communication methods. The display device 200 can communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0071] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0072] Figure 2 This is a hardware configuration block diagram of a display device provided in an embodiment of this application.

[0073] In some embodiments, such as Figure 2 As shown, the display device 200 may include at least one of the following: a tuner 210, a communicator (also known as a communication device) 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280.

[0074] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0075] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0076] For example, the display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0077] In some embodiments, the communicator 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communicators 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communicator 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communicator 220 with Bluetooth functionality.

[0078] The communicator 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0079] For example, the display device 200 can obtain user instructions sent by the control device through the communicator 220, and implement the screen projection method provided in the embodiments of this application according to the user instructions.

[0080] In some embodiments, when implementing the screen projection method provided in this application, the operations performed by the display device 200 can be implemented by the controller 250.

[0081] In some embodiments, the display device 200 may run an operating system to perform user interaction. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface. For example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running an application. The operating system also allows users to interact with the display device 200.

[0082] It should be noted that the aforementioned operating systems can be native operating systems based on a specific operating platform, third-party operating systems customized for a specific operating platform, or independent operating systems specifically developed for display devices.

[0083] Figure 3 This is a schematic diagram of the software system architecture of a display device provided in an embodiment of this application.

[0084] For example, such as Figure 3 As shown, the software system includes at least the following layers: application layer (also referred to as application layer), application framework layer (also referred to as framework layer), system runtime library layer, and kernel layer.

[0085] The application layer is used to deploy television-related functions. The application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with users based on the applications. In some embodiments, at least one application runs in the application layer. These applications can be applications that come with the operating system, such as user interface applications, television settings applications, and television services; or they can be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0086] For example, such as Figure 3 As shown, the application layer can include a converged screen mirroring application (which can be simply referred to as a converged screen mirroring application). The converged screen mirroring application has split-screen mirroring functionality. The converged screen mirroring application can include a screen mirroring playback interface, a screen mirroring service, a network request module, and a view display module.

[0087] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0088] like Figure 3 As shown, the framework layer can include a view system, managers, and content providers. The view system designs and implements the application's interface and interactions; it includes lists, grids, text boxes, and buttons.

[0089] The manager includes at least one of the following modules: Activity Manager, which interacts with all activities running on the system; Location Manager, which provides system services or applications with access to system location services; Package Manager, which retrieves various information related to application packages currently installed on the device; Notification Manager, which controls the display and clearing of notification messages; and Window Manager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0090] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining whether a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0091] In some embodiments, the framework layer also includes a player and system components.

[0092] like Figure 3 As shown, in some embodiments, the system runtime library layer (i.e., the system library layer) can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0093] like Figure 3 As shown, in some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, as... Figure 5 As shown, hardware drivers can be configured in the kernel layer. The drivers included in the kernel layer can be at least one of the following: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0094] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0095] Figure 4This is a schematic diagram of a software system architecture for an intelligent agent provided in an embodiment of this application. For example, as shown... Figure 4 As shown, the intelligent agent includes a management service (Agent Processor) module, an identification service (Client) module, a page matching (Page Match Alg) module, and a route planning (Route Plan) module.

[0096] The following embodiments of this application will be used to illustrate having Figure 2 and Figure 3 Taking the display device with the structure shown as an example, and in conjunction with the accompanying drawings and application scenarios, the screen projection method provided in this application embodiment will be specifically described.

[0097] Before introducing the screen casting method provided in the embodiments of this application, the application scenarios of the screen casting method provided in this application will be explained first.

[0098] Taking a smart TV 2 as the display device and a mobile phone 1 as the mobile terminal as an example, such as Figure 5 As shown, the Smart TV 2's integrated screen mirroring app has a split-screen mirroring function (also known as multi-screen mirroring mode). The split-screen mirroring function means that the Smart TV 2 supports simultaneous push of multiple screen mirroring content. For example... Figure 7 As shown, after the split-screen live streaming function of Smart TV 2 is enabled, firstly, Mobile Phone 1 initiates screen mirroring to Smart TV 2. After Smart TV 2 and Mobile Phone 1 negotiate screen mirroring, Mobile Phone 1 sends its screen content to Smart TV 2 via the screen mirroring connection. Smart TV 2 then displays Mobile Phone 1's screen content in the mirroring area 1. Next, in response to the split-screen casting command sent via the remote control, Smart TV 2 takes a screenshot of the currently displayed content, obtaining a screenshot image that includes Mobile Phone 1's screen content, and sends the screenshot image to the agent. The agent sends a recognition request to the recognition model, carrying the screenshot image. The recognition model recognizes the screenshot image and returns the recognition result to the agent. The agent analyzes the recognition result, generates and sends touch action information to Smart TV 2. Smart TV 2 parses and converts the touch action information, generates a counter-control command, and sends the counter-control command to Mobile Phone 1 to control Mobile Phone 1 to execute the counter-control command. In response to the control command, mobile phone 1 searches for and displays the target media resources corresponding to the target search information, and sends the screen content currently displayed by mobile phone 1 (i.e. the target media resources corresponding to the target search information) to smart TV 2 through screen mirroring connection, so that smart TV 2 can synchronously display the updated screen content of mobile phone 1 in screen mirroring area 1.

[0099] After displaying the updated screen content of phone 1, the process of sending reverse control commands is repeated until phone 1 initiates a DLNA casting command to smart TV 2. This DLNA casting command carries the resource address of the media resource corresponding to the currently displayed screen content of phone 1. Upon receiving the DLNA casting command, smart TV 2 retrieves the target media resource corresponding to the currently displayed screen content of phone 1 based on the resource address, and plays the media resource corresponding to the currently displayed screen content of phone 1 in the casting area 2, thereby achieving split-screen casting of the screen content (mirror image) and media resource of phone 1.

[0100] It should be noted that the split-screen casting method provided in this application embodiment can be used in live streaming scenarios, that is, split-screen casting of live streaming media resources. Therefore, the above-mentioned split-screen casting function can also be called split-screen live streaming function.

[0101] It should also be noted that the intelligent agent and recognition model can be deployed either in the cloud or locally on the smart TV. Furthermore, the intelligent agent and recognition model can be deployed independently or integrated.

[0102] The following explanation, with reference to the accompanying diagram, illustrates the process of enabling the split-screen casting function.

[0103] In some examples, the smart TV has a unified screen mirroring app installed, which can provide an entry point for the split-screen mirroring function to be launched.

[0104] Figure 6 This is a schematic diagram of the interactive process for enabling a split-screen projection function according to an embodiment of this application.

[0105] For example, taking a smart TV 2 as the display device and a split-screen live streaming function as the screen mirroring function, such as... Figure 6 As shown in (a), the smart TV 2 can display interface 61, which is the application interface of the smart TV 2. Interface 61 may include icons of multiple apps installed on the smart TV 2, such as the icon of the Juhaokan app and the icon 611 of the Fusion Casting app. In response to a click on the icon of the Fusion Casting app, the smart TV 2 can display... Figure 6 Interface 62, shown in (b) of the diagram, is the application interface of the integrated screen mirroring app. For example, interface 62 can display the network name and device name. The network name indicates the name of the network the smart TV is currently on, such as wifi123_5G. The device name refers to the name of the smart TV 2, such as Smart TV 2. Interface 62 also displays an animation effect of "Waiting for screen mirroring" to indicate that a screen mirroring connection has not yet been established. Interface 82 also includes a "Trial Lab" control 621, which is the entry point for the split-screen live streaming function.

[0106] After the Smart TV 2 displays interface 62, in response to the click operation of the "Early Bird Lab" control 621, the Smart TV 2 activates the split-screen live streaming function and displays as shown below. Figure 6 Interface 63 is shown in (c) above. Interface 63 is a tutorial page for the split-screen live streaming function, used to guide users through the split-screen live streaming operation. For example, interface 63 may include an explanation of the split-screen live streaming function, such as "After mirroring the phone to the large screen, you can play the live stream in split screen when watching the live stream on your phone." Interface 63 also includes the steps to start split-screen live streaming, as well as the corresponding operation tutorial screen for each step.

[0107] For example, Step 1: Connect your phone to the current network, such as "wifi123_5G", and select the current device, such as "Smart TV 2". Correspondingly, screen mirroring guide screen 631 appears on interface 63, highlighting the "Smart TV 2" device option to prompt you to initiate screen mirroring to "Smart TV 2". Step 2: Open a live streaming channel on your phone (e.g., open a live streaming app and enter a live stream room). Trigger the split-screen mirroring command by pointing at the remote control, such as dragging the mirrored screen of your phone to the right. Correspondingly, dragging the split-screen guide screen 632 appears on interface 63, prompting you to select a specific button and drag the live stream to the right. Step 3: The mirrored screen content and the live stream content are displayed in a split-screen format. Correspondingly, a schematic diagram of the split-screen live stream interface 633 appears on interface 63, demonstrating the display effect of the split-screen live stream. By following the steps in interface 63, you can achieve split-screen live streaming.

[0108] In some examples, interface 62 also includes options for different screen mirroring tutorials, such as "Juhaokan Screen Mirroring," "Mobile / Tablet Screen Mirroring," "Computer Screen Mirroring," and "Wired Screen Mirroring." These tutorial options allow users to view corresponding screen mirroring instructions to guide them in initiating the mirroring process.

[0109] The above provides an illustrative explanation of the process of enabling the split-screen casting function. The following section provides an illustrative explanation of the split-screen casting process.

[0110] 1. The process of mirroring the screen content of mobile phone 1 to smart TV 2

[0111] Taking live streaming media resources as an example, mobile phone 1 opens a live streaming APP and enters the live streaming room of live streaming 1. Then, mobile phone 1 initiates screen mirroring to smart TV 2. After mobile phone 1 and smart TV 2 negotiate to cast the screen, smart TV 2 displays the mirrored content in split-screen area 1.

[0112] For example, such as Figure 7As shown in (a), mobile phone 1 can display interface 71, which is the desktop of mobile phone 1. Desktop 1 can include icons of multiple apps installed on mobile phone 1, such as icons for clock apps, calendar apps, weather apps, and live streaming apps (icon 711). In response to the user's click on the icon 911 of the live streaming application, mobile phone 1 displays... Figure 7 Interface 72, shown in (b) above, is the homepage of the live streaming app. For example, interface 72 includes multiple page options for switching between different pages, such as "Home," "Friends," "Messages," and "Me." The "Home" option is highlighted, indicating that interface 72 is currently the homepage of the live streaming app. Interface 72 also includes multiple theme tags for switching between different themes, such as "Experience," "Qingdao," "Following," "Shop," and "Recommended." Interface 72 also includes a "Search" control 721. Additionally, interface 72 displays the live stream of Live 1, and a "Click to Enter Live Room" control is displayed within the live stream of Live 1.

[0113] Then, mobile phone 1 initiates screen mirroring to smart TV 2. For example, in response to a user's pull-down gesture on interface 72, mobile phone 1 can display... Figure 7 Interface 73, shown in (c), is the control center interface. Interface 73 includes multiple function control switches used to turn corresponding functions of mobile phone 1 on or off. For example, interface 73 may include a "Mute" control switch, a "Do Not Disturb" control switch, an "Airplane Mode" control switch, and a "Wireless Screen Casting" control switch 731. The "Wireless Screen Casting" control switch 731 is used to enable the phone's screen mirroring function. In response to the user's click on the "Wireless Screen Casting" control switch 731, mobile phone 1 enables the screen mirroring function and searches for target devices on the same local area network as mobile phone 1. When mobile phone 1 finds a smart TV 2 on the same local area network as mobile phone 1, mobile phone 1 can display... Figure 7 Interface 74, shown in (d), is a device selection interface. Interface 74 includes a device selection panel 741, which displays multiple device options, such as "Smart TV 2" device option 742 and "Smart TV 3" device option. In response to the user's click on the "Smart TV 2" device option 742, mobile phone 1 establishes a wireless connection with Smart TV 2 and exchanges information, negotiating whether mobile phone 1 and Smart TV 2 support screen mirroring. Figure 7 As shown in (e), during the negotiation process between mobile phone 1 and smart TV 2, smart TV 2 can display interface 75, which can display prompt information such as "casting in progress".

[0114] After the connection and negotiation between mobile phone 1 and smart TV 2 are established, mobile phone 1 can display... Figure 7 The interface 76 shown in (f) indicates that mobile phone 1 cancels the display of the device selection panel, and the content displayed on mobile phone 1 is projected onto smart TV 2 for display. For example, as shown in... Figure 7 As shown in (g), the smart TV 2 can display interface 77, which is a split-screen live streaming interface. Interface 77 can include two projection areas, such as projection area 1 and projection area 2, which are used to display the multi-channel projection content of mobile phone 1. Projection area 1 displays the mirrored projection content 771 of mobile phone 1, i.e., the main interface of the live streaming app. Interface 77 can also include a "search projection" control 772. The "search projection" control 772 is used to trigger the search for media resources corresponding to the target search information and control mobile phone 1 to play the media resources corresponding to the target search information in projection area 2.

[0115] Additionally, a guide animation 773 is displayed in projection area 2 to guide users through the search and projection process.

[0116] Of course, in some examples, mobile phone 1 can also initiate screen mirroring to smart TV 2 first. After mobile phone 1 and smart TV 2 successfully establish a screen mirroring connection, in response to the user's operation, mobile phone 1 opens the live streaming APP. At this time, the content in the screen mirroring area 1 of smart TV 2 is synchronously updated to the main interface of the live streaming APP.

[0117] 2. Search and screen mirroring process

[0118] During the mirroring process between mobile phone 1 and smart TV 2, the user can point to the remote control and click the "Search and Cast" control to control mobile phone 1 to search for and display the media resources corresponding to the target search information. Then, smart TV 2 controls mobile phone 1 in reverse, and casts the media resources corresponding to the target search information in the casting area 2 via DLNA, thus realizing the mirrored screen of mobile phone 1 and the split-screen casting of other media resources.

[0119] For example, after the smart TV 2 enters interface 77, such as Figure 8 As shown in (a), interface 77 includes a "Search and Cast" control 772, which displays the target search information, such as Live 2. The user points the remote control toward the smart TV 2 to display cursor 81 in interface 77. The user adjusts the position of cursor 81 on the screen (i.e., display area) of the smart TV 2 by moving the pointer on the remote control. When cursor 81 is on the "Search and Cast" control 772, the user presses the designated button on the remote control (such as the OK button), thus triggering the search and cast command by pointing at the remote control.

[0120] In response to the search and casting command (i.e., the target control command), Smart TV 2 takes a screenshot of its currently displayed content, identifies and analyzes the screenshot image, generates a reverse control command, and sends this command to Mobile Phone 1 to control Mobile Phone 1 to search for and display the target media resource corresponding to the search information. Mobile Phone 1, responding to the reverse control command, searches for and displays the target media resource corresponding to the search information, i.e., Live 2. Mobile Phone 1 also sends its currently displayed screen content (the live stream of Live 2) to Smart TV 2 via screen mirroring connection. After receiving the currently displayed screen content (the live stream of Live 2) from Mobile Phone 1, Smart TV 2 can display... Figure 8 The interface 82 shown in (b) displays the screen content of the mobile phone 1 in the projection area 1 in sync with the interface 77, that is, the live room interface of the searched live 2.

[0121] After displaying the screen content of mobile phone 1, the process of sending reverse control commands is repeated. This reversely controls mobile phone 1 to initiate a DLNA screen mirroring connection to smart TV 2, and through this DLNA screen mirroring connection, sends DLNA screen mirroring command 1 to smart TV 2. DLNA screen mirroring command 1 carries the resource address of live stream 2 corresponding to the screen content currently displayed on mobile phone 1. Upon receiving DLNA screen mirroring command 1, smart TV 2 retrieves live stream 2 based on the resource address. After retrieving live stream 2, smart TV 2 can display... Figure 8 As shown in (c) of the diagram, interface 83, compared to interface 82, plays live stream 2 in the projection area 2 of interface 83, that is, displays the live stream interface of live stream 2. At this time, the projection area 1 still displays the mirrored content corresponding to mobile phone 1 (i.e., the live stream room interface of live stream 2).

[0122] It is understandable that after the split-screen live stream is completed, the mirrored content displayed in projection area 1 and the DLNA projection content displayed in projection area 2 are both Live 2. Since the DLNA projection content is related to the mirrored image displayed in projection area 1 when the DLNA projection is triggered, the live stream in projection area 2 and the live stream in projection area 1 can be the same or different at this time.

[0123] The above is an illustrative explanation of the process of searching for "Live 2" and then projecting it in the split screen of projection area 2. The following is a detailed explanation of the jump process from interface 77 to interface 82.

[0124] like Figure 9As shown in (a), the smart TV 2 displays the screen mirroring interface A1 in the screen mirroring area 1. At this time, the smart TV 2 takes a screenshot of the currently displayed content, obtaining screenshot image 1. It then identifies and analyzes screenshot image 1, determining that the currently displayed screen mirroring interface A1 jumps to the main interface of the live streaming app (i.e., Page 1), and determines that the control to be triggered is the "search" control A11. It then generates the corresponding reverse control command 1 for the "search" control A11, i.e., the reverse control command to click the "search" control A11. The smart TV 2 sends reverse control command 1 to the mobile phone 1, and the mobile phone 1 responds to reverse control command 1 by displaying... Figure 9 The interface 901 shown in (b) is the search interface 1. Furthermore, mobile phone 1 connects via screen mirroring and sends the screen content corresponding to interface 901 to smart TV 2. For example... Figure 11 As shown in (c), the smart TV 2 receives the screen content corresponding to the interface 901 and displays the casting interface A2 in the casting area 1. The casting interface A2 corresponds to the interface 901 currently displayed on the mobile phone 1.

[0125] After the smart TV 2 displays the screen mirroring interface A2, it takes a screenshot of the currently displayed content, resulting in screenshot image 2. The smart TV 2 then identifies and analyzes screenshot image 2, determining that the current screen mirroring interface A2 redirects to search interface 1 (i.e., Page 2). It further determines that the target search information, such as "Live 2," needs to be entered into the "search box" A21, thus generating a reverse control command 2, which is the command to enter "Live 2" into the "search box" A21. The smart TV 2 sends reverse control command 2 to mobile phone 1, and mobile phone 1 responds to reverse control command 2 by displaying... Figure 9 As shown in (d) above, interface 902 is search interface 2. Furthermore, mobile phone 1 connects via screen mirroring and sends the screen content corresponding to interface 902 to smart TV 2. For example... Figure 9 As shown in (e), the smart TV 2 receives the screen content corresponding to the interface 902 and displays the casting interface A3 in the casting area 1. The casting interface A3 corresponds to the interface 902 currently displayed on the mobile phone 1.

[0126] After the smart TV 2 displays the screen mirroring interface A3, it takes a screenshot of the currently displayed content, resulting in screenshot image 3. The screenshot image 3 is then identified and analyzed to determine that the current screen mirroring interface A3 redirects to search interface 2 (i.e., Page 3), and the control to be triggered is the "search" control A31. A corresponding control command 3 is then generated for the "search" control A31, i.e., the control command to click the "search" control A31. The smart TV 2 sends control command 3 to mobile phone 1, and mobile phone 1 responds to control command 3 by displaying... Figure 9Interface 903, shown in (f), is the search completion interface 2. Furthermore, mobile phone 1 connects via screen mirroring and sends the screen content corresponding to interface 903 to smart TV 2. For example... Figure 9 As shown in (g), the smart TV 2 receives the screen content corresponding to the interface 902 and displays the screen casting interface A4 in the screen casting area 1. The screen casting interface A4 corresponds to the interface 903 currently displayed on the mobile phone 1.

[0127] After the smart TV 2 displays the screen mirroring interface A4, it takes a screenshot of the currently displayed content, resulting in screenshot image 4. The screenshot image 4 is then identified and analyzed to determine that the current screen mirroring interface A4 redirects to the search completion interface 2 (i.e., Page 4), and the control to be triggered is the "Live Stream 2" control A41. Therefore, a counter-control command 4 is generated corresponding to the "Live Stream 2" control A41, i.e., the counter-control command to click the "Live Stream 2" control A41. The smart TV 2 sends the counter-control command 4 to the mobile phone 1, and the mobile phone 1 responds to the counter-control command 4, displaying... Figure 9 Interface 904, shown in (h), is the live streaming interface. Furthermore, mobile phone 1 connects via screen mirroring and sends the screen content corresponding to interface 904 to smart TV 2. For example... Figure 10 As shown in (b), the smart TV 2 receives the screen content corresponding to the interface 904 and displays the corresponding screen projection interface of the interface 904 in the projection area 1, that is, displays the live broadcast room interface of the live broadcast 2.

[0128] The following provides a detailed explanation of the transition process from interface 82 to interface 83.

[0129] like Figure 10 As shown in (a), the smart TV 2 displays the live broadcast interface A5 of Live Broadcast 2 in the projection area 1. At this time, the smart TV 2 takes a screenshot of the currently displayed content, obtaining screenshot image 5. It then identifies and analyzes screenshot image 5, determining that the currently displayed live broadcast interface A5's interface jump location is the live broadcast room interface, and identifies the control to be triggered as the "control panel" control A51. It then generates the corresponding reverse control command 5 for the "control panel" control A51, i.e., the reverse control command to click the "control panel" control A51. The smart TV 2 sends the reverse control command 5 to the mobile phone 1, and the mobile phone 1 responds to the reverse control command 5 by displaying... Figure 10 The interface 1001 shown in (b) is a control panel interface. Furthermore, mobile phone 1 connects via screen mirroring and sends the screen content corresponding to interface 1001 to smart TV 2. For example... Figure 10 As shown in (c), the smart TV 2 receives the screen content corresponding to the interface 1001 and displays the casting interface A6 in the casting area 1. The casting interface A6 corresponds to the interface 1001 currently displayed on the mobile phone 1.

[0130] After the smart TV 2 displays the screen mirroring interface A6, it takes a screenshot of the currently displayed content, resulting in screenshot image 6. The screenshot image 6 is then identified and analyzed to determine that the current screen mirroring interface A6 leads to the control panel interface, and the control to be triggered is the "screen mirroring" control A61. A corresponding control command 6 is then generated for the "screen mirroring" control A61, i.e., clicking the "screen mirroring" control A61. The smart TV 2 sends the control command 6 to the mobile phone 1, and the mobile phone 1 responds to the control command 6 by displaying... Figure 12 Interface 1002, shown in (d) above, is a device selection interface. Furthermore, mobile phone 1 connects via screen mirroring and sends the screen content corresponding to interface 1002 to smart TV 2. For example... Figure 10 As shown in (e), the smart TV 2 receives the screen content corresponding to the interface 1002 and displays the casting interface A7 in the casting area 1. The casting interface A7 corresponds to the interface 1002 currently displayed on the mobile phone 1.

[0131] After Smart TV 2 displays the screen mirroring interface A7, it takes a screenshot of the currently displayed content, resulting in screenshot image 7. The screenshot image 7 is then identified and analyzed to determine that the current screen mirroring interface A7 is the device selection interface, and the control to be triggered is the "Smart TV 2" option A71. Therefore, a reverse control command 7 is generated corresponding to option A71, i.e., clicking option A71. Smart TV 2 sends the reverse control command 7 to mobile phone 1, and mobile phone 1 responds to the reverse control command 7 by displaying... Figure 10 The interface 1003 shown in (f) is the screen mirroring completed interface. Furthermore, mobile phone 1 sends the screen content corresponding to interface 1003 to smart TV 2 via screen mirroring connection. For example... Figure 10 As shown in (g), the smart TV 2 receives the screen content corresponding to the interface 1003 and displays the casting interface A8 in the casting area 1. The casting interface A8 corresponds to the interface 1003 currently displayed on the mobile phone 1.

[0132] After the smart TV 2 displays the casting interface A8, it takes a screenshot of the currently displayed content, resulting in screenshot image 8. The smart TV 2 then identifies and analyzes screenshot image 8 to determine that the current screen transition position of the casting interface A8 is the casting completion screen. This means the smart TV 2 has confirmed that the live stream 1 currently displayed on the phone 1 has been cast to the smart TV 2 via DLNA. Afterwards, the smart TV 2 can send a preset reverse control command 8 to the phone 1. This command controls the phone 1 to cancel the display of the casting pop-up window 10031. The phone 1 responds to the reverse control command 8, cancels the display of the casting pop-up window 10031, and returns to its previous state. Figure 10Interface 1004, shown in (h), is the live stream interface of Live Stream 2. Furthermore, mobile phone 1 sends the screen content corresponding to interface 1004 to smart TV 2 via screen mirroring. For example... Figure 8 As shown in (c), the smart TV 2 receives the screen content corresponding to the interface 1004 and displays the screen projection interface corresponding to the interface 1004 currently displayed on the mobile phone 1 in the projection area 1.

[0133] The process of identifying and analyzing the screenshot image, determining the interface jump position of the currently displayed projection interface, and generating corresponding counter-control commands can be found in the relevant explanation of the knowledge graph below, and will not be repeated here.

[0134] After controlling mobile phone 1 to cast the found live stream 2 to the screen area 2 of smart TV 2 via a DLNA screen mirroring connection, smart TV 2 can send preset control commands to mobile phone 1 to control mobile phone 1 to switch the currently displayed screen content.

[0135] For example, on the Smart TV 2 display Figure 11 After the screen mirroring interface A9 shown in (a) appears, the smart TV 2 can send a reverse control command 9 to the mobile phone 1. The reverse control command 9 is used to control the mobile phone 1 to execute the swipe-up command. In response to the reverse control command 9 (such as the swipe-up command) sent by the smart TV 2, the mobile phone 1 switches the currently displayed screen content, that is, the mobile phone 1 displays... Figure 11 The interface 1101 shown in (b) includes the live stream footage of Live Stream 3. Furthermore, mobile phone 1 sends the screen content corresponding to interface 1101 to smart TV 2 via screen mirroring connection. For example... Figure 11 As shown in (c), the smart TV 2 receives the screen content corresponding to interface 1101 and displays the casting interface A10 in the casting area 1. The casting interface A10 corresponds to the interface 1101 currently displayed on the mobile phone 1, meaning that the smart TV 2 displays the live broadcast of live broadcast 3 in the casting area 1. In this way, while watching live broadcast 2 on the smart TV 2, the user can also watch other screen content on the mobile phone 1.

[0136] The above is an illustrative description of the application scenarios of the screen projection method provided in the embodiments of this application.

[0137] In this embodiment, when generating anti-control instructions based on screenshot images, the screenshot images can be matched with a pre-constructed knowledge graph to generate anti-control instructions. The knowledge graph is illustrated below with reference to the accompanying drawings. The knowledge graph can also be referred to as preset path data. The knowledge graph may include interface information for multiple preset interfaces, as well as information indicating the interface jump relationships between the multiple preset interfaces. The multiple preset interfaces may include multiple preset interfaces that allow jumps from a preset homepage interface to a preset completion interface.

[0138] The interface information of a preset interface can include the interface jump location, element information, and preset action type, where the interface jump location is used to uniquely identify the preset interface. For example, when constructing a knowledge graph, a single interface jump location can be set for each preset interface.

[0139] Element information can include multiple key elements in the preset interface and the preset coordinates of each key element. Key elements can be text elements, such as the text "Cast Screen" or "Click to Enter Live Room". Key elements can also be icon elements, such as the "Cast Screen" icon or the icon of the "Control Panel" control. It should be noted that key elements in a preset interface can include all interface elements in that preset interface, or only some of the interface elements in that preset interface. There are no specific limitations on the number and type of key elements.

[0140] For a preset interface, one of the multiple key elements in the preset interface is designated as the key element. The designated key element is the key element corresponding to the interactive control to be triggered when the preset interface is displayed on the terminal device. For example, when the preset interface is a control panel interface, the designated key element is the "cast" icon corresponding to the "cast" control.

[0141] The preset coordinates of a key element refer to the coordinates of the key element on the screen of the smart TV within the display coordinate system. For example, the preset coordinates of a key element can be the coordinates of the center point of the key element, or it can include the coordinates of the four corner points of the key element.

[0142] The element information may also include weight parameters for each key element. In this embodiment, different preset interfaces may include the same interface elements. Therefore, weight parameters can be set for key elements in a preset interface based on their correlation with the preset interface. The larger the weight parameter corresponding to a key element (e.g., close to 1), the greater the correlation between the key element and the corresponding preset interface. For example, if a key element belongs to only one preset interface, it indicates a greater correlation between the key element and that preset interface, and a larger weight parameter (e.g., 0.8) can be set. Conversely, the smaller the weight parameter corresponding to a key element (e.g., close to 0), the less correlated the key element is with the corresponding preset interface. For example, if multiple preset interfaces include a certain key element, it indicates a less correlated correlation between the key element and the preset interface, and a smaller weight parameter (e.g., 0.8) can be set.

[0143] The preset action type indicates the type of touch action to be triggered for the interactive control when the interface displayed on the terminal device is a preset interface. For example, the preset action type can be a click or a swipe. It can be understood that there is a correspondence between the preset action type and the interface jump position corresponding to the preset interface.

[0144] For example, taking a live streaming application as an example, knowledge graph 1 is the knowledge graph corresponding to the drag-to-split-screen command. For example... Figure 12 As shown, Knowledge Graph 1 includes eight preset interfaces that navigate from the live stream interface to the screen mirroring completion interface. The interface information for each preset interface in Knowledge Graph 1 can be found in Table 1 below.

[0145] Table 1 shows the interface information corresponding to Knowledge Graph 1.

[0146]

[0147]

[0148] It should be noted that the coordinates and weight parameters corresponding to each key element are not shown in Table 1.

[0149] From Table 1 above and Figure 12 As can be seen, when the screen mirroring interface in the screenshot is the main interface, the corresponding interface jump position is Page1, the specified key element to be triggered is the "search" icon, the touch position corresponding to Page1 is the coordinate of the "search" icon, and the action type corresponding to Page1 is click.

[0150] When the screen projection interface in the screenshot is search interface 1, the corresponding interface jump position is Page 2, the specified key element to be triggered is the "search box" icon, the touch position corresponding to Page 2 is the coordinates of the "search box" icon, and the action type corresponding to Page 2 is inputting target search information.

[0151] When the screen mirroring interface in the screenshot is search interface 2, the corresponding interface jump position of the screen mirroring interface is Page 3, the specified key element to be triggered is the text "search", the touch position corresponding to Page 3 is the coordinates of the text "search", and the action type corresponding to Page 3 is click.

[0152] When the screen mirroring interface in the screenshot is the search completion interface, the corresponding interface jump position is Page4, the specified key element to be triggered is the "Live 2" icon, the touch position corresponding to Page4 is the coordinate of the "Live 2" icon, and the action type corresponding to Page4 is click.

[0153] When the screen mirroring interface in the screenshot is the live streaming interface, the corresponding interface jump position is Page 5, the specified key element to be triggered is the "Control Panel" icon, the touch position corresponding to Page 5 is the coordinate of the "Control Panel" icon, and the action type corresponding to Page 5 is click.

[0154] When the screen mirroring interface in the screenshot is the control panel interface, the corresponding interface jump position is Page 6, the specified key element to be triggered is the "screen mirroring" icon, the touch position corresponding to Page 6 is the coordinate of the "screen mirroring" icon, and the action type corresponding to Page 6 is click.

[0155] When the screen mirroring interface in the screenshot is the screen mirroring completion interface, the corresponding interface jump position is Page7, the specified key element to be triggered is the "Smart TV 2" option icon, the touch position corresponding to Page7 is the coordinate of the "Smart TV 2" option icon, and the action type corresponding to Page7 is click.

[0156] The above is an illustrative description of the application scenarios and knowledge graph of the screen projection method provided in the embodiments of this application. The screen projection method provided in the embodiments of this application will be described below.

[0157] Figure 13 This is a flowchart illustrating a screen mirroring method provided in an embodiment of this application. Figure 13 As shown, the screen mirroring method may include the following steps:

[0158] S101, in the multi-channel projection mode, the receiving terminal device sends the first screen content currently displayed on the terminal device through the mirror projection connection, and displays the first interface. The first interface includes a first projection area (i.e. projection area 1 mentioned above). The first projection area displays the first projection interface corresponding to the first screen content. The first projection interface includes a first designated element, and the first designated element corresponds to the first control in the first screen content.

[0159] In this embodiment, under multi-channel projection mode, the display device can simultaneously support one mirror projection and at least one DLNA projection push. The mirror projection connection is established based on a mirror projection protocol that supports a reverse control protocol, including the Miracast protocol. For example, the display device can simultaneously support one mirror projection and one DLNA projection push. Alternatively, the display device can simultaneously support one mirror projection and two DLNA projection pushes.

[0160] After the terminal device establishes a connection with the display device based on the mirroring protocol, the terminal device collects screen frame data (such as audio and video streams) in real time and sends the screen frame data to the display device. The first screen content refers to the screen frame data currently received by the display device. After receiving the first screen content, the display device renders and displays a first interface, which may include a first projection area and a second projection area, and presents the first screen content in the first projection area. It can be understood that the first projection area (i.e., projection area 1 above, also known as the mirroring projection area) is used to display the screen content currently displayed by the terminal device received through the mirroring connection. (i.e., projection area 2 above, also known as the first projection area) is used to display other projection content besides mirroring, such as media resources pushed by the terminal device via DLNA projection.

[0161] For example, such as Figure 8 As shown in (a), interface 77 (i.e., the first interface) includes projection area 1 and projection area 2. The first projection interface corresponding to the first screen content of the terminal device is displayed in projection area 1, such as the live broadcast screen of live broadcast 1.

[0162] The first projection interface is the screen displayed when the display device decodes and renders the first screen content (such as an audio stream) transmitted from the terminal device. The first projection interface includes a first designated element, which is the interface element corresponding to a first control. The first control is a designated interactive control within the first screen content displayed on the terminal device. Through the first control, the terminal device can be triggered to execute a target action, which is to send a DLNA projection command to the display device.

[0163] In some examples, before receiving the first screen content currently displayed on the terminal device sent via the mirroring connection, the method further includes: in response to a multi-channel mirroring mode start command, starting the multi-channel mirroring mode and displaying a guide interface, the guide interface being used to guide the split-screen display of media resources corresponding to the screen content sent by the terminal device via the mirroring connection and the screen content currently displayed on the terminal device. S102, in response to a target control command sent by the control device, taking a screenshot of the content currently displayed on the display device to obtain a first screenshot image.

[0164] The target control command is used to trigger the display device to send multiple counter-control commands to the terminal device. These multiple counter-control commands are used to control the terminal device to search for and display the first media resource corresponding to the target search information.

[0165] In some examples, a first functional control is displayed on the first interface, where the target control instruction is a selection instruction for the first functional control. The first functional control is used to trigger the display device to send multiple counter-control instructions to the terminal device. These counter-control instructions control the terminal device to search for and display the first media resource corresponding to the target search information.

[0166] For example, such as Figure 8 As shown in (a), the first functional control is the "Search and Cast" control 772. Correspondingly, the target control command can be to move the cursor to the "Search and Cast" control 772 and press the designated button on the remote control.

[0167] For example, a first prompt message is displayed on the first functional control, and the first prompt message includes target search information. For example, such as... Figure 8 As shown in (a), the first prompt message is "Live 2".

[0168] In this way, when the display device displays the first interface, the first functional control is displayed on the first interface. Through the first functional control, the display device can be triggered to send multiple control commands to the terminal device with one click, so as to control the terminal device to search for and display the first media resource corresponding to the target search information. The operation is convenient and the user experience is better.

[0169] In this embodiment, in response to a target control command sent by a control device, a screenshot of the content currently displayed on the display device is taken. The screenshot image is then identified and analyzed, and a counter-control command is sent to the terminal device. After sending this counter-control command, the screenshot of the currently displayed content on the display device is taken again. The screenshot image is then identified and analyzed, and a next counter-control command is sent to the terminal device. This process is repeated, allowing multiple counter-control commands to be sent to the display device to control the terminal device to search for and display the first media resource corresponding to the target search information.

[0170] In other words, in response to the target control command sent by the control device, a screenshot is taken of the content currently displayed on the display device to obtain a first screenshot image, and a counter-control command is sent to the terminal device based on the first screenshot image.

[0171] S103, obtain the first action information corresponding to the first specified element (i.e., action information 1 mentioned above). The first action information is determined by identifying the first specified element and the first position information of the first specified element in the first screenshot image, and based on the first position information, the action type corresponding to the first specified element, and the target search information.

[0172] Specifically, after acquiring the first screenshot image, the display device sends the first screenshot image to the intelligent agent, enabling the agent to recognize and analyze the first screenshot image to obtain the first action information. The display device receives the first action information sent by the intelligent agent. The generation process of the first action information is detailed later in the section on... Figure 15 The explanation will not be repeated here.

[0173] S104, based on the first action information, generate a first reverse control command, and send the first reverse control command to the terminal device through a mirroring connection. The first reverse control command is used to control the terminal device to search for and display the first media resource corresponding to the target search information.

[0174] In some examples, the first position information is transformed into third position information based on a preset coordinate transformation relationship. The preset coordinate transformation relationship is used to indicate the coordinate transformation relationship between the display device and the terminal device. Based on the event rules corresponding to the terminal device, a first reverse control instruction is generated according to the third position information and the action type corresponding to the first specified element.

[0175] The event rules corresponding to the terminal device refer to the interaction event rules of the terminal device. The interaction events of the terminal device can include MotionEvent and KeyEvent.

[0176] For example, taking MotionEvent as an example,

[0177] Create and dispatch touch press events:

[0178] MotionEvent downEvent=createMotionEvent(MotionEvent.ACTION_DOWN,x,y);

[0179] dispatchTouchEvent(downEvent).

[0180] Create and dispatch touch lift events

[0181] MotionEvent upEvent=createMotionEvent(MotionEvent.ACTION_UP,x,y);

[0182] dispatchTouchEvent(upEvent).

[0183] S105, the receiving terminal device responds to the second screen content sent by the first reverse control command, and displays the second projection interface corresponding to the second screen content in the first projection area. The second screen content is the resource content of the first media resource.

[0184] It should be noted that the specific implementation methods of S101 to S105 can be found in the above text regarding... Figure 9 The explanation will not be repeated here.

[0185] In some examples, such as Figure 14 As shown, the second projection interface includes a second designated element, which corresponds to a second control in the second screen content. After displaying the second projection interface corresponding to the second screen content in the first projection area, the projection method may further include the following steps:

[0186] S201, Take a screenshot of the content currently displayed on the display device to obtain a second screenshot image.

[0187] S202, obtain the second action information (i.e., action information 2 mentioned above) corresponding to the second specified element. The second action information is determined by recognizing the second specified element in the second screenshot image and the second position information of the second specified element, and based on the action type corresponding to the second position information and the second specified element.

[0188] Here, the second specified element is the interface element corresponding to the second control, and the second control is the specified interactive control in the second screen content displayed on the terminal device. Through the second control, the terminal device can be triggered to execute a specified action, which is to control the terminal device to initiate a first DLNA screen projection connection to the display device.

[0189] Specifically, after acquiring the second screenshot image, the display device sends the second screenshot image to the intelligent agent, enabling the agent to recognize and analyze the second screenshot image to obtain the second action information. The display device receives the second action information sent by the intelligent agent. The generation process of the second action information is detailed later in the section on... Figure 15 The explanation will not be repeated here.

[0190] S203, based on the second action information, generate a second reverse control instruction. The second reverse control instruction is used to control the terminal device to initiate a first DLNA screen projection connection to the display device, and send a first screen projection instruction to the display device through the first DLNA screen projection connection. The first screen projection instruction carries a first resource address, which is the resource address of the first media resource corresponding to the second screen content currently displayed by the terminal device.

[0191] The specific implementation of generating the second counter-control command based on the second action information can be found in the specific implementation of S104 above, and will not be repeated here.

[0192] In this embodiment, when the first control is not a preset projection control, one or more reverse control commands are sent to the terminal device before sending the first reverse control command to the terminal device, until the display device displays a projection interface including the preset projection control. The preset projection control is used to trigger the terminal device to initiate a first DLNA projection connection to the display device.

[0193] For example, generating a second counter-control command based on the second action information may include the following steps:

[0194] S301 generates a third reverse control command based on the second action information.

[0195] S302 sends third-party control commands to terminal devices via screen mirroring connection.

[0196] S303, the terminal device receives the third screen content sent in response to the third reverse control command, and displays the third projection interface corresponding to the third screen content in the first projection area. The third projection interface includes a third specified element, which corresponds to the preset projection control in the third screen content.

[0197] S304, take a screenshot of the content currently displayed on the display device to obtain the third screenshot image.

[0198] S305, obtain the third action information corresponding to the third specified element. The third action information is determined by recognizing the third specified element and the third position information of the third specified element in the third screenshot image, and based on the third position information and the action type corresponding to the third specified element.

[0199] S306, based on the third action information, generates a second reverse control command.

[0200] It should be noted that the specific implementation methods of S301 to S306 can be found in the above text regarding... Figure 10 The explanation will not be repeated here.

[0201] S204 sends the first reverse control command to the terminal device via screen mirroring connection.

[0202] S205, in response to the first screen projection instruction sent by the terminal device, obtains the first media resource according to the first resource address, and plays the first media resource in the second screen projection area (i.e., screen projection area 2 mentioned above).

[0203] In some examples, after playing the first media resource in the second projection area, the method further includes: sending a fourth reverse control command to the terminal device via a mirror projection connection, so that the terminal device responds to the fourth reverse control command and displays fourth screen content, the fourth screen content including a third control, the fourth screen content corresponding to the second media resource; receiving the fourth screen content sent by the terminal device via the mirror projection connection, and displaying the fourth projection interface corresponding to the fourth screen content in the first projection area, the fourth projection interface including a fourth specified element, the fourth specified element corresponding to the third control.

[0204] Optionally, both the first and second media resources are live streaming media resources.

[0205] The fourth reverse control command is used to control the terminal device to switch the currently displayed screen content. For example, the fourth reverse control command is... Figure 11 The reverse control command 9 shown is, for example, the swipe-up command. The content of the fourth screen is... Figure 11 The live stream screen of Live Stream 3 is shown in (b) above. The fourth screen mirroring interface can be... Figure 11 The screen mirroring interface A10 is shown in (c) in the image.

[0206] It should be noted that for details on how the terminal device switches the currently displayed screen content via the fourth reverse control command, please refer to the information on... Figure 11 The explanation will not be repeated here.

[0207] In some examples, before playing the first media resource in the second projection area, the method further includes: displaying a guide animation in the second projection area, the guide animation being used to guide the execution of target control instructions.

[0208] Figure 15 This is a flowchart illustrating another screen mirroring method provided in an embodiment of this application. Figure 15 As shown, this screen projection method is applied to an intelligent agent, and the screen projection method may include the following steps:

[0209] S401, receive a first screenshot image sent by the display device. The first screenshot image is a screenshot image obtained by the display device in response to the target control command. The first screenshot image includes a first projection interface displayed by the display device in the first projection area. The first projection interface corresponds to the first screen content currently displayed by the terminal device. The first projection interface includes a first designated element. The first designated element corresponds to a first control in the first screen content.

[0210] S402, identify the first specified element and the first position information of the first specified element in the first screenshot image.

[0211] Here, the first designated element is the interface element corresponding to the first control, and the first control is the designated interactive control in the first screen content displayed by the terminal device. Through the first control, the terminal device can be triggered to execute a target action, which is to control the terminal device to search for and display the first media resource corresponding to the target search information.

[0212] like Figure 16 As shown, the process of recognizing a screenshot image may include the following steps:

[0213] S501, Obtain preset path data; wherein, the preset path data includes the interface information of each of the multiple preset interfaces, and data indicating the jump relationship between the multiple preset interfaces; the interface information includes multiple preset elements in the preset interface, the preset position information of each preset element, and the preset action type corresponding to a specified preset element among the multiple preset elements.

[0214] The preset path data (also known as a knowledge graph) includes interface information for multiple preset interfaces, as well as data indicating the navigation relationships between these preset interfaces. It can be understood that the multiple preset interfaces may include multiple preset interfaces that navigate from a preset homepage interface to a preset completion interface.

[0215] The interface information includes multiple preset elements within a preset interface and the preset position information of each preset element. A preset element (i.e., the key element mentioned above) within a preset interface can include all interface elements in that preset interface, or it can include only a portion of the interface elements. For example, a preset element can be a text element, such as the text "Cast Screen" or "Click to Enter Live Room". Key elements can also be icon elements, such as the "Cast Screen" icon or the icon for the "Control Panel" controls.

[0216] The preset position information of a preset element (i.e., the preset coordinates mentioned above) refers to the coordinate position of the preset element on the screen of the display device in the display coordinate system. For example, the preset position information of a preset element can be the coordinates of the center point of the preset element, or it can include the coordinates of the four corner points of the preset element.

[0217] For a given preset interface, one of the multiple preset elements within that interface is designated as a specific preset element (i.e., the designated key element mentioned above). The designated preset element is the preset element corresponding to the interactive control to be triggered when the preset interface is displayed on the terminal device. For example, when the preset interface is used as a control panel interface, the designated preset element is the "cast" icon corresponding to the "cast" control.

[0218] Specifying the preset action type corresponding to a preset element indicates the type of touch action to be triggered for the interactive control when the interface displayed on the terminal device is a preset interface. For example, the preset action type can be a click or a swipe. It can be understood that there is a correspondence between the preset action type and the interface jump position corresponding to the preset interface.

[0219] The interface information also includes weight parameters corresponding to preset elements. In this embodiment, different preset interfaces may include the same interface elements. Therefore, weight parameters can be set for preset elements in a preset interface based on the degree of association between the preset element and the preset interface. The larger the weight parameter corresponding to a preset element (e.g., close to 1), the greater the degree of association between the preset element and the corresponding preset interface. For example, if a preset element belongs to only one preset interface, it indicates that the preset element is more associated with that preset interface, and a larger weight parameter (e.g., 0.8) can be set. The smaller the weight parameter corresponding to a preset element (e.g., close to 0), the less associated with the preset element and the corresponding preset interface. For example, if multiple preset interfaces include a certain preset element, it indicates that the preset element is less associated with the preset interface, and a smaller weight parameter (e.g., 0.8) can be set.

[0220] In this embodiment, different preset path data can be pre-set for different applications and different split-screen casting scenarios. Different split-screen casting scenarios can be triggered by different control commands. For example, taking a live streaming application as an example, for a search casting scenario, the target control command is the selection command for the "search casting" control, and the preset path data can be... Figure 12 The knowledge graph 1 is shown. Based on this, after establishing multiple preset path data, the preset path data, the instruction type of the control command corresponding to the preset path data, and the application identifier of the application corresponding to the preset path data can be associated. In this way, the preset path data can be obtained according to the instruction type of the control command and the application identifier of the application to which the projection interface in the first screenshot belongs.

[0221] In some examples, S501 may include: obtaining the instruction type of the target control instruction; determining the application identifier of the application to which the first projection interface belongs based on the first screenshot image; and obtaining preset path data based on the instruction type and application identifier of the target control instruction.

[0222] The instruction type is used to distinguish different control instructions. For example, the target control instruction is a selection instruction for the "Search and Cast" control, and the instruction type of the target control instruction is search and cast.

[0223] Different applications have different interfaces, resulting in different screen-sharing interfaces in screenshots. Therefore, by performing image recognition on the first screenshot, the application identifier of the first screen-sharing interface can be determined. Furthermore, based on the command type of the target control instruction and the application identifier, preset path data can be obtained. In this way, based on the command type of the target control instruction and the application identifier of the first screen-sharing interface, preset path data can be obtained more accurately, enabling automated split-screen projection based on the preset path data.

[0224] S502, identify the first projection interface in the first screenshot image, and obtain at least one first interface element and the position information of each first interface element.

[0225] The first interface element refers to the interface element identified from the first projection interface of the first screenshot image. The position information of the first interface element refers to the coordinate position of the first interface element in the first interface.

[0226] S503, for each preset interface, based on the first interface element and the position information of the first interface element, at least one first interface element is matched with multiple preset elements in the preset interface to determine the first preset interface corresponding to the first projection interface, and the specified preset element in the first preset interface is the first specified element.

[0227] Specifically, S503 may include:

[0228] S5031, for each preset interface, compare the first interface element with the first preset element in the preset interface, and compare the position information of the first interface element with the preset position information of the preset element in the preset interface to determine the second preset element.

[0229] The first preset element is any preset element in the preset interface. The second preset element is the preset element among multiple preset elements in the preset interface that matches the first preset element. For example, there may be one or more second preset elements.

[0230] For example, comparing a first interface element with a first preset element and comparing the position information of the first interface element with the preset position information of the first preset element to determine a second preset element includes: if the first interface element matches the first preset element and the coordinates of the first interface element match the coordinates of the first preset element, then the first preset element is determined as the second preset element.

[0231] In other words, after identifying the first interface element, it is determined whether there is a preset element in the preset path data that matches the first interface element. Specifically, taking any preset interface as an example, the first interface element is compared with each preset element (i.e., the first preset element) in the preset interface. When comparing the first interface element with the first preset element, the element itself (i.e., text or icon) of the first interface element is compared with the element itself of the first preset element, and the position information of the first interface element is compared with the preset position information of the first preset element. If the element itself of the first interface element matches the element itself of the first preset element, and the position information of the first interface element is the same as the preset position information of the first preset element, then the first preset element is determined to match the first interface element, and this first preset element is identified as the second preset element.

[0232] Of course, when identifying multiple first interface elements from the first screenshot image, for each first interface element, the first interface element is compared with each preset element.

[0233] For example, taking the main interface of a live streaming application as a preset interface, the preset elements include the text elements "Click to enter the live room," "Home," "Friends," "Messages," "Me," "Experience," "Following," "Shop," "Recommendations," "Click to enter the live room" control icon, "Search," and "Post." The first interface element is compared with each of the above elements, and the position information of the first interface element is compared with the preset position information of each of the above elements to determine the second preset element that matches the main interface.

[0234] S5032, determine the matching degree of the preset interface based on the weight parameter corresponding to the second preset element and the number of preset elements in the preset interface.

[0235] Specifically, for each preset interface, after identifying the second preset element from the elements of the first interface, the matching degree of that preset interface is determined based on the weight parameter corresponding to the second preset element and the number of preset elements in the preset interface. The matching degree represents the degree of association between the second preset element and different preset interfaces.

[0236] For example, when determining the matching degree corresponding to the preset interface, the weight parameters of each second preset element are summed to obtain the total weight; the ratio of the total weight to the number of preset elements in the preset interface is determined as the matching degree corresponding to the preset interface.

[0237] S5033, determine the first preset interface based on the matching degree corresponding to each preset interface, wherein the matching degree corresponding to the first preset interface is the maximum value among the matching degrees corresponding to multiple preset interfaces, and the matching degree corresponding to the first preset interface is greater than the matching degree threshold corresponding to the first preset interface.

[0238] For each preset interface, a matching degree threshold can be set. When the calculated matching degree is greater than the matching degree threshold, it means that the preset interface is a preset interface that matches the first interface element, that is, the preset interface matches the first projection interface in the first screenshot image.

[0239] In this embodiment of the application, different preset interfaces may include the same preset elements. Therefore, when determining the first preset interface corresponding to the first projection interface, for each preset interface, the second preset element is determined by comparing the first interface element with the first preset element in the preset interface. The matching degree of the preset interface is determined according to the weight parameter corresponding to the second preset element and the number of preset elements in the preset interface. Then, the first preset interface is determined according to the matching degree of each preset interface. In this way, the accuracy of determining the first preset interface can be improved.

[0240] S504, the preset position information of the specified preset element in the first preset interface is determined as the first position information, and the preset action type corresponding to the specified preset element in the first preset interface is determined as the action type corresponding to the first specified element.

[0241] After determining the first preset interface that matches the first projection interface, the specified preset element in the first preset interface is indicated to be the interface element corresponding to the first control triggered by the terminal device to be controlled. The specified preset element corresponding to the first preset interface is determined as the first specified element, and the preset position information of the specified preset element is used to determine the first position of the first specified element. The preset action type corresponding to the specified preset element is determined as the action type corresponding to the first specified element.

[0242] For example, when determining the preset action type corresponding to a specified preset element in the first preset interface, the preset action type corresponding to the specified preset element can be determined based on the interface jump position corresponding to the first preset interface. The interface jump position is used to uniquely identify the preset interface. For example, when constructing a knowledge graph, an interface jump position can be set for each preset interface.

[0243] For example, when the first preset interface matched by the first screen projection interface is the main interface of the live streaming application, the interface jump position corresponding to the screen projection interface is Page1, the specified preset element to be triggered is the "Click to enter the live room" control icon, and the action type corresponding to Page1 is click. Then the first position information of the first specified element is the coordinates corresponding to the "Click to enter the live room" control icon, and the action type corresponding to the first specified element is click.

[0244] For example, if the first preset interface matched by the first screen projection interface is the search interface 1 of the live streaming application, the interface jump position corresponding to the screen projection interface is Page 2, the specified preset element to be triggered is the "search box" icon, and the action type corresponding to Page 2 is inputting target search information, then the first position information of the first specified element is the coordinates corresponding to the "search box" icon, and the action type corresponding to the first specified element is inputting target search information.

[0245] For example, if the first preset interface matched by the first screen projection interface is the search interface 2 of the live streaming application, the interface jump position corresponding to the screen projection interface is Page 3, the specified preset element to be triggered is the text "search", and the action type corresponding to Page 3 is click, then the first position information of the first specified element is the coordinates corresponding to the text "search", and the action type corresponding to the first specified element is click.

[0246] For example, if the first preset interface matched by the first screen projection interface is the search completion interface of the live streaming application, the interface jump position corresponding to the screen projection interface is Page 4, the specified preset element to be triggered is the "Live 2" icon, and the action type corresponding to Page 4 is click, then the first position information of the first specified element is the coordinates corresponding to the "Live 2" icon, and the action type corresponding to the first specified element is click.

[0247] S403, determine the first action information based on the target search information, the first position information of the first specified element, and the action type corresponding to the first specified element.

[0248] For example, the first action information can be represented using JSON data.

[0249] For example, the JSON data is:

[0250]

[0251] The "action" field represents the first action information, where "CLICK" indicates that the action type is click, and "x" and "y" represent the first position information, i.e., the touch coordinates.

[0252] S404, send first action information to the display device so that the display device generates a first counter-control instruction based on the first action information. The first counter-control instruction is used to control the terminal device to search for and display the first media resource corresponding to the target search information.

[0253] In some examples, the screen mirroring method may also include:

[0254] S601, receive a second screenshot image sent by the display device. The second screenshot image is an image captured by the display device after receiving the second screen content sent by the terminal device. The second screenshot image includes a second projection interface displayed by the display device in the first projection area. The second projection interface includes a second specified element. The second specified element corresponds to a second control in the second screen content. The second screen content is the resource content corresponding to the first media resource.

[0255] S602, identify the second specified element and the second position information of the second specified element in the second screenshot image.

[0256] S603, determine the second action information based on the second position information of the second specified element and the action type corresponding to the second specified element.

[0257] S604, send second action information to the display device so that the display device generates a second reverse control command based on the second action information. The second reverse control command is used to control the terminal device to initiate a first DLNA screen projection connection to the display device and send a first screen projection command to the display device through the first DLNA screen projection connection. The first screen projection command carries a first resource address, which is the resource address of the first media resource corresponding to the second screen content currently displayed by the terminal device.

[0258] For details on the specific implementation methods of S601 to S604, please refer to the specific implementation methods of S401 to S404, which will not be repeated here.

[0259] In this embodiment, the intelligent agent can recognize a screenshot image sent by the display device to obtain a first specified element and its first position information. Based on the first position information and the corresponding action type, it determines first action information, which is used to generate a first counter-control command. Thus, based on the screenshot image, the intelligent agent can identify the control to be triggered and generate corresponding action information. The display device can then send a counter-control command to the terminal device based on the action information, thereby controlling the terminal device to perform automated search and screen projection. Furthermore, using an intelligent agent for screenshot image recognition and analysis improves recognition accuracy and processing efficiency.

[0260] The above is a schematic description of the knowledge graph provided in the embodiments of this application. The following description continues, using a smart TV 2 as the display device, a mobile phone 1 as the terminal device, a remote control as the control device, and the Miracast protocol as the screen mirroring protocol, to illustrate the screen mirroring method provided in the embodiments of this application.

[0261] Figure 16 This is a flowchart illustrating a screen mirroring method provided in an embodiment of this application. Figure 16 As shown, the screen mirroring method may include the following steps:

[0262] S701, in response to the open operation 1, the remote control sends an open command 1 to the converged screen mirroring application of the smart TV 2.

[0263] Specifically, the "open operation 1" is used to trigger the sending of an "open command 1" to the remote control. For example, the split-screen casting function can be enabled through the entry point provided by the integrated casting application in the smart TV 2. For instance, "open operation 1" can be used to... Figure 6 The click operation of the "Trial Lab" control 621 shown in (b) is illustrated. Of course, the above-described opening operation 1 is only one example, and the embodiments of this application do not specifically limit the type of opening operation 1.

[0264] Command 1 is used to trigger the smart TV 2 to start the split-screen casting function, that is, to start the multi-channel casting mode.

[0265] S702, in response to the open command 1, Smart TV 2 starts the multi-screen casting mode and displays the guide page.

[0266] In multi-screen casting mode, Smart TV 2 can simultaneously support one-way mirroring and at least one-way DLNA casting push. Specifically, in multi-screen casting mode, when Smart TV 2 receives screen content sent by Mobile Phone 1, Smart TV 2 displays the corresponding screen content of Mobile Phone 1 in a designated casting area (such as casting area 1) within its display area.

[0267] For example, a landing page could be, for instance, like this Figure 6 Interface 63 is shown in (c).

[0268] The default onboarding screen CastPlayerActivity is launched using the native Android launch Activity method. After the application starts, the View is initialized, and the default onboarding screen will be displayed when the application is entered. It will display some device information, such as the name of the current large-screen device and the corresponding network information. The onboarding screen is displayed in a symmetrical manner from left to right using standard Android controls TextView and ImageView and their corresponding Layout.

[0269] Meanwhile, CastPlayerActivity initializes an internal NetAgentProgress object to prepare for automated screen mirroring. The NetAgentProgress object mainly communicates with the Agent over the network, then parses the Agent's instructions and sends them to the terminal device through the screen mirroring control protocol channel.

[0270] S703, in response to the opening operation 2, mobile phone 1 opens the live streaming APP and displays screen content 01.

[0271] The "open" operation 2 can be used to trigger mobile phone 1 to open the live stream of media resource 1 in the live streaming app. For example, the "open" operation 2 may include... Figure 7 The click operation of the live streaming APP icon 711 shown in (a) is illustrated. For example, the opening operation 1 can also be an operation of opening the live screen of the media resource 1 of the live streaming APP by giving a voice instruction to the mobile phone 1. In this embodiment of the application, the type of opening operation 2 is not specifically limited.

[0272] S704, In response to screen casting operation 1, mobile phone 1 sends a Miracast screen casting request to the integrated screen casting application.

[0273] Screen mirroring operation 1 can be used to trigger a screen mirroring connection between mobile phone 1 and smart TV 2. For example, combined with... Figure 7 (b) to Figure 7 As shown in (d), screen mirroring operation 1 may include a pull-down operation on interface 72, a click operation on the "Wireless Screen Mirroring" control switch 731, and a click operation on the "Smart TV 2" device option 742. For example, screen mirroring operation 1 may also be an operation of initiating a mirroring connection from mobile phone 1 to smart TV 2 via voice instruction. The type of screen mirroring operation 1 is not specifically limited in this embodiment.

[0274] S705, in response to the Miracast casting request, Smart TV 2 and Mobile Phone 1 negotiate to establish a mirror casting connection (i.e., one Miracast connection).

[0275] Specifically, after the application page (i.e. the screen casting playback interface) starts, the corresponding screen casting service CastServer will be started, and some initialization preparation work will be done, such as starting the listening image or pushing the corresponding protocol service; after CastServer starts, the listeners UseSessionConnectionListener, UseSessionPrepareListener, UseSessionDisplayListener and UseSessionEventListener will be registered.

[0276] The screen mirroring service is always ready for new sessions to connect. Once phone 1 is ready to connect, the service listens for the new session's `onConnect` call via the `UsessionConnectionListener()` method. Then, it listens for the current screen mirroring content type, such as MiraCast mirroring, and notifies the screen mirroring application `CastPlayerActivity` to create the playback control `SurfaceView / TextureView` via the `onConnect` callback message at the protocol layer. After successful creation, the `ViewModule` class is started to set and manage the display position of the current session. To facilitate the management of the View and Session, this application defines a management object `MultiCastPlayView`, which associates information such as Session, `SurfaceView / TextureView`, and `mViewPosition`. Once CastPlayerActivity has prepared the View, it notifies the mobile device via UseSessionPrepareListener that it can record or send the video stream. When the mobile device is ready to send video information, the server learns that the large screen is about to play the video through UseSessionDisplayListener's onFirstFrame() method. It then obtains the current video width and height of Client1 from the Session, determines the current video stream direction based on the width and height, and finally presents the current projected content through the ViewModule class.

[0277] S706, Mobile Phone 1 connects via screen mirroring and sends the screen content 01 of Mobile Phone 1 to the integrated screen mirroring application.

[0278] S707. In multi-screen projection mode, the smart TV 2 displays a first interface, which includes projection area 1 and projection area 2, and displays the first projection interface corresponding to screen content 01 in projection area 1.

[0279] For example, the first interface can be a split-screen live streaming interface of the display device, such as... Figure 7 Interface 77 is shown in (g).

[0280] S708, in response to screen mirroring operation 2, points the remote control to send a search screen mirroring command to the converged screen mirroring application.

[0281] Among them, the search casting command (i.e. the target control command below) can be used to control the smart TV 2 to search for and display the media resource 1 corresponding to the target search information, so as to cast the media resource 1 to the smart TV 2 in split screen.

[0282] For example, the first interface includes a first functional control, such as... Figure 8 When using the "Search and Cast" control 772 shown in (a) of the first interface, the search and cast command can also be a selection command for the "Search and Cast" control 772 in the first interface. For example, move the cursor to the remote control, move the cursor to the "Search and Cast" control 772, and press the OK button on the remote control.

[0283] S709, Responding to the search casting command, integrate the casting application to obtain device information 1. Target search information and the command type of the search casting command 1.

[0284] Device information 1 is used to uniquely identify smart TV 2. For example, device information 1 is the name of the smart TV, such as smart TV 2.

[0285] The instruction type identifies the triggered split-screen casting instruction. Different types of split-screen casting instructions correspond to different preset path data (i.e., knowledge graphs).

[0286] The target search information is used to indicate the media resources that the mobile phone 1 wants to search. The target search information can be pre-acquired or it can be search information entered by the user via voice. For example, the target search information is "Live Stream 2". This application embodiment does not limit the specific content of the target search information.

[0287] The S710 and the integrated screen mirroring application take a screenshot of the content currently displayed on the smart TV 2 to obtain the first screenshot image.

[0288] This is understandable; it involves taking a screenshot of the content currently displayed on the smart TV 2.

[0289] S711, the integrated screen projection application sends an action generation request 1 to the intelligent agent. The action generation request 1 carries the first screenshot image, instruction type 1, target search information and device information 1.

[0290] S712. In response to action generation request 1, the agent uses the recognition model to recognize the first screenshot image and obtains recognition result 1.

[0291] The recognition result 1 includes multiple interface elements 1 in the first projection interface and the first coordinates (i.e., first position information) of each interface element 1. The first coordinates are used to indicate the position of the interface element 1 in the first screenshot image, that is, the coordinates of the interface element 1 in the display coordinate system of the smart TV 2. For example, the first coordinates may include the coordinates of the center point of the interface element 1, or the coordinates of the four corner points of the interface element 1.

[0292] S713, the agent obtains knowledge graph 1 according to instruction type 1, and generates action information 1 based on knowledge graph 1, target search information and recognition result 1.

[0293] In this embodiment, multiple knowledge graphs are pre-established. The knowledge graphs may include interface information for multiple preset interfaces, as well as information indicating the interface navigation relationships between these preset interfaces. The multiple preset interfaces may include multiple preset interfaces that navigate from a preset homepage interface to a preset completion interface. The interface information of a preset interface includes the interface navigation position (e.g., Page 1), multiple key elements within the preset interface, and the preset coordinates of each key element. Among the multiple key elements is a designated key element, which is the key element corresponding to the interactive control to be triggered when the terminal device's display interface is the preset interface, such as the "cast" icon.

[0294] Key elements can include text elements and icon elements, such as the text "Cast Screen" and the "Cast Screen" icon. Preset coordinates can include the coordinates of the center point of the key element, or the coordinates of its four corner points. It can be understood that the coordinates of key elements in the interface information are also coordinates in the display coordinate system of the smart TV.

[0295] In some examples, different knowledge graphs can be built for different applications. Therefore, the agent can determine the application identifier 1 of the application to which the first projection interface belongs based on the recognition result 1, and obtain the knowledge graph 1 based on the instruction type 1 and the application identifier 1.

[0296] S714, the intelligent agent sends action information 1 to the integrated screen projection application. Action information 1 includes touch position 1 and action type 1.

[0297] S715, the integrated screen mirroring application generates a first reverse control command based on action information 1. This first reverse control command controls mobile phone 1 to search for and display media resource 1 corresponding to the target search information.

[0298] Specifically, the smart TV 2 analyzes the action information 1 and performs coordinate transformation on the touch position 1. Based on the transformed touch position 1, action type 1, and the event rules corresponding to the mobile phone 1, it generates the first reverse control command.

[0299] S716, the integrated screen mirroring application sends the first reverse control command to mobile phone 1.

[0300] S717, In response to the first reverse control command, mobile phone 1 searches for and displays the resource content of media resource 1 corresponding to the target search information.

[0301] S718, mobile phone 1 connects via screen mirroring and sends screen content 02 of mobile phone 1 to the converged screen mirroring application. Screen content 02 includes the content corresponding to media resource 1.

[0302] S719, The integrated screen mirroring application displays screen content 02 in screen mirroring area 1.

[0303] The above provides an illustrative explanation of the process by which a smart TV controls a mobile phone to search for and display media resources corresponding to the target search information. The following provides an illustrative explanation of the process by which a smart TV controls a mobile phone to split-screen and project the media resources corresponding to the currently displayed screen content onto the smart TV.

[0304] Figure 17 This is a flowchart illustrating a screen mirroring method provided in an embodiment of this application. Figure 17 As shown, the screen mirroring method may include the following steps:

[0305] S801. When screen content 02 is displayed in the projection area 1, the integrated projection application takes a screenshot of the content currently displayed on the smart TV 2 to obtain a second screenshot image.

[0306] S802, the integrated screen projection application sends an action generation request 2 to the intelligent agent. The action generation request 2 carries the second screenshot image, instruction type 1 and device information 1.

[0307] S803 In response to action generation request 2, the agent uses the recognition model to recognize the second screenshot image and obtain recognition result 2.

[0308] S804: The agent obtains knowledge graph 1 according to instruction type 1, and generates action information 2 based on knowledge graph 1 and recognition result 2.

[0309] S805, the intelligent agent sends action information 2 to the integrated screen projection application, the action information 2 including touch position 2 and action type 2.

[0310] S806, the integrated screen mirroring application generates a second reverse control command based on the action information 2. The second reverse control command is used to control the mobile phone 1 to initiate a DLNA screen mirroring connection 1 to the smart TV 2, and send the screen mirroring command 1 to the smart TV 2 through the DLNA screen mirroring connection 1.

[0311] Specifically, the smart TV 2 analyzes the action information 2 and performs coordinate transformation on the touch position 2. Based on the transformed touch position 2, action type 2, and the event rules corresponding to the mobile phone 1, it generates a second reverse control command.

[0312] S807, the integrated screen mirroring application sends a second reverse control command to mobile phone 1.

[0313] S808, in response to the second reverse control command, mobile phone 1 sends a DLNA screen mirroring request 1 to the converged screen mirroring application.

[0314] S809, In ​​response to DLNA screen mirroring request 1, Smart TV 2 and Mobile Phone 1 establish DLNA screen mirroring connection 1.

[0315] S810 and mobile phone 1 send a screen casting command 1 to the converged screen casting application via DLNA screen casting connection 1. The screen casting command 1 carries the resource address 1 of media resource 1. Media resource 1 is the media resource corresponding to the screen content 01 currently displayed on mobile phone 1, that is, the media resource searched and displayed according to the target search information, such as live broadcast 2.

[0316] S811, the integrated screen mirroring application obtains media resource 1 based on resource address 1, and plays media resource 1 in the screen mirroring area 2.

[0317] S812, in response to the second reverse control command, mobile phone 1 displays screen content 03.

[0318] S813, Mobile Phone 1 connects via screen mirroring and sends the screen content 03 of Mobile Phone 1 to the integrated screen mirroring application.

[0319] S814, The integrated screen mirroring application displays screen content 03 in screen mirroring area 1.

[0320] S815, the integrated screen mirroring application sends a fourth reverse control command to mobile phone 1, which is used to control mobile phone 1 to switch screen content.

[0321] For example, the fourth reverse control command could be, for instance, Figure 11 The reverse control command 9 in the middle.

[0322] S816, in response to the fourth reverse control command, mobile phone 1 displays screen content 04, screen content 04 is the live content corresponding to media resource 2 (such as live broadcast 3).

[0323] S817, Display screen content 04 in projection area 1.

[0324] The above is a schematic description of the overall process of the screen casting method provided in the embodiments of this application. The process of generating a reverse control command (i.e., S711 to S716 above) by the integrated screen casting application will be described in detail below, taking the first reverse control command as an example.

[0325] Figure 18 This is a flowchart illustrating a screen mirroring method provided in an embodiment of this application. Figure 18 As shown, the screen mirroring method may include the following steps:

[0326] S901. In response to action generation request 1, the management service module in the intelligent agent sends recognition request 1 to the recognition service module, and the recognition request 1 carries the first screenshot image.

[0327] S902, The recognition service module sends recognition request 2 to the recognition model. Recognition request 1 carries the first screenshot image.

[0328] S903. The recognition model recognizes the first screenshot image and obtains recognition result 1. Recognition result 1 includes multiple interface elements 1 and the coordinates 1 of each interface element 1.

[0329] S904, The recognition model sends the recognition result 1 to the management service module.

[0330] S905, the management service module sends a page matching request to the page matching module in the intelligent agent. The page matching request carries the recognition result 1, instruction type 1 and target search information.

[0331] S906. The page matching module obtains knowledge graph 1 according to instruction type 1.

[0332] S907. The page matching module determines, based on knowledge graph 1, whether there is a first preset interface that matches the first projection interface.

[0333] S908, when a first preset interface that matches the first projection interface exists, the page matching module determines the interface jump position 1 corresponding to the first preset interface and the specified key element 1 in the first preset interface.

[0334] S909, the page matching module sends a path planning request to the path planning module in the agent. The path planning request carries the interface jump position 1, the specified key element 1 and the target search information.

[0335] S910, the path planning module obtains the action type 1 corresponding to the interface jump position 1 based on the knowledge graph 1.

[0336] S911 The path planning module generates action information 1 based on action type 1, target search information, and the preset coordinates corresponding to the specified key element 1.

[0337] S912, The path planning module sends action information 1 to the management service module.

[0338] S913, The management service module sends action information 1 to the converged screen casting application of the smart TV 2.

[0339] S914 The integrated screen projection application parses and converts action information 1 to generate the corresponding control command.

[0340] S915, the integrated screen mirroring application sends the corresponding reverse control command of action information 1 to mobile phone 1.

[0341] When the interface jump position 1 indicates that the current screen projection interface is the search completed interface, the reverse control command corresponding to action information 1 is used to control mobile phone 1 to enter the resource interface of the media resource corresponding to the target search information, that is, the reverse control command corresponding to action information 1 is the first reverse control command.

[0342] In some examples, when the interface jump position 1 indicates that the current screen mirroring interface is the screen mirroring completion interface, the path planning module can obtain a pre-stored preset reverse control command corresponding to the preset jump position. This preset reverse control command can be used to control phone 1 to cancel the screen mirroring pop-up and to control phone 1 to switch screen content.

[0343] In some examples, if no first preset interface matches the first projection interface, the page matching module sends an error message to the management service module. The management service module then sends the error message to the integrated projection application. In response to the error message, the integrated projection application stops taking screenshots.

[0344] S916 The management service module obtains the interface jump position 1 from the page matching module and determines whether the interface jump position 1 is the preset jump position.

[0345] Among them, the preset jump position indicates that the screen content of the terminal device jumps to the last preset interface in Knowledge Graph 1, such as the screen projection completion interface.

[0346] S917. If the interface jump position 1 is not the preset jump position, the management service module sends notification information 1 to the integrated screen casting application. Notification information 1 is used to notify the integrated screen casting application 2 that the split screen casting command has not been completed.

[0347] S918, In response to notification message 1, the integrated screen mirroring application takes a screenshot of the content currently displayed on the smart TV 2.

[0348] After S918, the integrated screen mirroring application sends new screenshot images to the agent to obtain new action information from the agent, that is, repeating S902 to S916.

[0349] S919, when the interface jump position 1 is the preset jump position, the management service module determines whether there is an interface element in the recognition result 1 that matches the device information of the smart TV 1.

[0350] S920: If the identification result 1 contains an interface element that matches the device information of the smart TV 1, the management service module sends notification information 2 to the integrated screen casting application. Notification information 2 is used to notify the integrated screen casting application 2 that the split-screen casting command has been executed.

[0351] S921, In response to notification message 2, the integrated screen mirroring application stops taking screenshots.

[0352] This disclosure provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the Bluetooth connection method described above and achieves the same technical effects. To avoid repetition, further details are omitted here. The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0353] This disclosure provides a computer program product that includes a computer program that, when run on a computer, enables the computer to implement the Bluetooth connection method described above.

[0354] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments above is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of the embodiments suitable for specific application considerations.

[0355] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0356] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0357] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0358] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0359] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0360] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0361] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0362] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A screen projection method, characterized in that, Applied to a display device, the method includes: In multi-channel projection mode, the receiving terminal device sends the first screen content currently displayed on the terminal device through the mirror projection connection, and displays a first interface. The first interface includes a first projection area, and displays a first projection interface corresponding to the first screen content in the first projection area. The first projection interface includes a first designated element, and the first designated element corresponds to a first control in the first screen content. In response to a target control command sent via a control device, a screenshot is taken of the content currently displayed on the display device to obtain a first screenshot image; Obtain the first action information corresponding to the first specified element. The first action information is determined by identifying the first specified element and the first position information of the first specified element in the first screenshot image, and based on the first position information, the action type corresponding to the first specified element, and the target search information. Based on the first action information, a first counter-control instruction is generated and sent to the terminal device through the mirroring connection. The first counter-control instruction is used to control the terminal device to search for and display the first media resource corresponding to the target search information. The terminal device receives the second screen content sent in response to the first reverse control command, and displays the second projection interface corresponding to the second screen content in the first projection area. The second screen content is the resource content of the first media resource.

2. The method according to claim 1, characterized in that, The second projection interface includes a second designated element, which corresponds to a second control in the second screen content; after displaying the second projection interface corresponding to the second screen content in the first projection area, the method further includes: A screenshot is taken of the content currently displayed on the display device to obtain a second screenshot image; The second action information corresponding to the second specified element is obtained. The second action information is determined by recognizing the second specified element and the second position information of the second specified element in the second screenshot image, and based on the second position information and the action type corresponding to the second specified element. Based on the second action information, a second reverse control command is generated. The second reverse control command is used to control the terminal device to initiate a first DLNA screen projection connection to the display device, and to send a first screen projection command to the display device through the first DLNA screen projection connection. The first screen projection command carries a first resource address, which is the resource address of the first media resource corresponding to the second screen content currently displayed by the terminal device. The first reverse control command is sent to the terminal device via the screen mirroring connection. In response to the first screen projection command sent by the terminal device, the first media resource is obtained according to the first resource address, and the first media resource is played in the second screen projection area.

3. The method according to claim 2, characterized in that, After playing the first media resource in the second projection area, the method further includes: Through the screen mirroring connection, a fourth reverse control command is sent to the terminal device, so that the terminal device responds to the fourth reverse control command and displays fourth screen content, the fourth screen content including a third control, and the fourth screen content corresponding to a second media resource; The terminal device receives the fourth screen content sent through the mirroring connection and displays the fourth projection interface corresponding to the fourth screen content in the first projection area. The fourth projection interface includes a fourth designated element, which corresponds to the third control.

4. The method according to claim 2, characterized in that, Before playing the first media resource in the second projection area, the method further includes: A guide animation is displayed in the second projection area to guide the execution of the target control command.

5. The method according to any one of claims 1-4, characterized in that, The first interface includes a first functional control, on which a first prompt message is displayed. The first prompt message includes the target search information. The method further includes: In response to the trigger command of the control device to the first functional control, the target control command is obtained.

6. The method according to any one of claims 1-4, characterized in that, Before the receiving terminal device sends the content of the first screen currently displayed on the terminal device via the screen mirroring connection, the method further includes: In response to the start command of the multi-screen projection mode, the multi-screen projection mode is started and a guide interface is displayed. The guide interface is used to guide the split-screen display of the media resources corresponding to the screen content sent by the terminal device through the mirror projection connection and the screen content currently displayed by the terminal device.

7. A screen projection method, characterized in that, Applied to intelligent agents, the method includes: The first screenshot image sent by the display device is a screenshot image obtained by the display device in response to the target control command. The first screenshot image includes a first projection interface displayed by the display device in the first projection area. The first projection interface corresponds to the first screen content currently displayed by the terminal device. The first projection interface includes a first designated element, which corresponds to a first control in the first screen content. Identify the first specified element and the first position information of the first specified element in the first screenshot image; The first action information is determined based on the target search information, the first position information of the first specified element, and the action type corresponding to the first specified element. The first action information is sent to the display device so that the display device generates a first counter-control instruction based on the first action information. The first counter-control instruction is used to control the terminal device to search for and display the first media resource corresponding to the target search information.

8. The method according to claim 7, characterized in that, The method includes: The display device receives a second screenshot image sent by the display device. The second screenshot image is an image captured by the display device after receiving the second screen content sent by the terminal device. The second screenshot image includes a second projection interface displayed by the display device in the first projection area. The second projection interface includes a second specified element. The second specified element corresponds to a second control in the second screen content. The second screen content is the resource content corresponding to the first media resource. Identify the second specified element and the second position information of the second specified element in the second screenshot image; The second action information is determined based on the second position information of the second specified element and the action type corresponding to the second specified element; The second action information is sent to the display device so that the display device generates a second reverse control command based on the second action information. The second reverse control command is used to control the terminal device to initiate a first DLNA screen projection connection to the display device and send a first screen projection command to the display device through the first DLNA screen projection connection. The first screen projection command carries a first resource address, which is the resource address of the first media resource corresponding to the second screen content currently displayed by the terminal device.

9. The method according to claim 8, characterized in that, The step of identifying the first designated element and the first position information of the first designated element in the first screenshot image includes: Obtain preset path data; wherein, the preset path data includes interface information of each of the multiple preset interfaces, and data indicating the jump relationship between the multiple preset interfaces; the interface information includes multiple preset elements in the preset interface, preset position information of each preset element, and preset action type corresponding to a specified preset element among the multiple preset elements; The first projection interface in the first screenshot image is identified to obtain at least one first interface element and the position information of each first interface element. For each of the preset interfaces, based on the first interface element and the position information of the first interface element, the at least one first interface element is matched with multiple preset elements in the preset interface to determine the first preset interface corresponding to the first projection interface, and the specified preset element in the first preset interface is the first specified element. The preset position information of the specified preset element in the first preset interface is determined as the first position information, and the preset action type corresponding to the specified preset element in the first preset interface is determined as the action type corresponding to the first specified element.

10. A display device, characterized in that, include: monitor; A controller connected to the display is configured to: In multi-channel projection mode, the receiving terminal device sends the first screen content currently displayed on the terminal device through the mirror projection connection, and displays a first interface. The first interface includes a first projection area, and displays a first projection interface corresponding to the first screen content in the first projection area. The first projection interface includes a first designated element, and the first designated element corresponds to a first control in the first screen content. In response to a target control command sent via a control device, a screenshot is taken of the content currently displayed on the display device to obtain a first screenshot image; Obtain the first action information corresponding to the first specified element. The first action information is determined by identifying the first specified element and the first position information of the first specified element in the first screenshot image, and based on the first position information, the action type corresponding to the first specified element, and the target search information. Based on the first action information, a first counter-control instruction is generated and sent to the terminal device through the mirroring connection. The first counter-control instruction is used to control the terminal device to search for and display the first media resource corresponding to the target search information. The terminal device receives the second screen content sent in response to the first reverse control command, and displays the second projection interface corresponding to the second screen content in the first projection area. The second screen content is the resource content of the first media resource.

Images