Control method, control device, equipment and medium

Abstract

The embodiment of the present disclosure provides a control method, a control device, equipment and a medium, which belong to the technical field of communication, and aims to simplify the operation of a user in screen projection control. The control method comprises the following steps: based on a pre-established screen projection connection with a second terminal, receiving screen projection data sent by the second terminal; generating a screen projection area corresponding to the second terminal, and displaying the screen projection data in the screen projection area; and in response to a first operation occurring in the screen projection area, interacting with the second terminal to share target data corresponding to the first operation with the second terminal, and / or enabling the second terminal to execute a second operation corresponding to the first operation on the screen projection data based on the target data.

Description

Technical Field

[0001] This disclosure relates to the field of communication technology, and in particular to a control method, control device, equipment and medium. Background Technology

[0002] With the continuous development of mobile communication, smart devices have become an indispensable part of people's daily lives. Smart devices allow users to share content, such as text, images, music, or videos, and also enable remote control, such as remotely controlling an application on another smart device.

[0003] In related technologies, the process of sharing content and remotely controlling devices via smart devices is cumbersome and inconvenient. Summary of the Invention

[0004] Based on the background technology, this disclosure proposes a control method to facilitate content sharing between smart devices.

[0005] In a first aspect, a control method is provided, applied to a first terminal, the method comprising:

[0006] Based on the pre-established screen mirroring connection with the second terminal, receive screen mirroring data sent by the second terminal;

[0007] Generate a projection area corresponding to the second terminal, and display the projection data in the projection area;

[0008] In response to a first operation that occurs wholly or partially in the projection area, the system interacts with the second terminal to exchange target data corresponding to the first operation, so as to share the target data with the second terminal and / or allow the second terminal to perform a second operation corresponding to the first operation on the projection data based on the target data.

[0009] In a second aspect, a control method is provided, applied to a second terminal, the method comprising:

[0010] Based on the pre-established screen projection connection with the first terminal, data is sent to the first terminal so that the first terminal can display the screen projection data in the screen projection area;

[0011] The target data corresponding to the first operation is interacted with the first terminal; wherein, the first operation occurs partly or entirely in the projection area on the first terminal;

[0012] Store the target data, and / or perform a second operation corresponding to the first operation on the projected data based on the target data.

[0013] In a third aspect, a control system is provided, including a first terminal and a second terminal, wherein the first terminal is configured to execute the control method described in the first aspect, and the second terminal is configured to execute the control method described in the second aspect.

[0014] In a fourth aspect, a control method is provided, applied to a control system, the method comprising:

[0015] The first terminal displays the projection data of the second terminal in the projection area; wherein the projection data is sent by the second terminal to the first terminal based on a pre-established projection connection;

[0016] The second terminal responds to a first operation that occurs partly or entirely on the projection area of ​​the first terminal, shares the target data corresponding to the first operation with the first terminal, and / or performs a second operation corresponding to the first operation on the projection data.

[0017] In a fifth aspect, a control device is provided for use in a first terminal, the device comprising:

[0018] The data receiving module is used to receive screen projection data sent by the second terminal based on a pre-established screen projection connection with the second terminal;

[0019] The display module is used to display the projection data in the projection area corresponding to the second terminal;

[0020] An interaction module is used to respond to a first operation that occurs wholly or partially in the projection area, to interact with the second terminal with target data corresponding to the first operation, so as to share the target data with the second terminal, and / or to allow the second terminal to perform a second operation on the projection data corresponding to the first operation based on the target data.

[0021] In a sixth aspect, a control device is provided for use in a second terminal, the device comprising:

[0022] The data sending module is used to send data to the first terminal based on a pre-established screen projection connection with the first terminal, so that the first terminal can display the screen projection data in the screen projection area;

[0023] The data interaction module is used to interact with the first terminal to exchange target data corresponding to the first operation; wherein, the first operation occurs partly or entirely in the projection area on the first terminal;

[0024] An execution module is used to store the target data and / or perform a second operation on the projected data based on the target data, corresponding to the first operation.

[0025] This disclosure also discloses an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the control method as described in any of the embodiments of the first to third aspects.

[0026] This disclosure also discloses a computer-readable storage medium storing a computer program that causes a processor to perform the control method as described in any of the embodiments of the first to third aspects of this disclosure.

[0027] In this embodiment of the disclosure, the first terminal can generate a projection area corresponding to the second terminal based on a pre-established projection connection with the second terminal, and display the projection data sent by the second terminal in the projection area; then, in response to a first operation that occurs in whole or in part in the projection area, the first terminal can interact with the second terminal to exchange target data corresponding to the first operation, so as to share the target data with the second terminal, and / or allow the second terminal to perform a second operation corresponding to the first operation on the projection data based on the target data.

[0028] On the one hand, as the second terminal shares its own projection data with the first terminal, the first terminal can also share its target data with the second terminal based on the projection area where the projection data is played. Thus, the user only needs to perform the first operation in the projection area to share the target data with the second terminal while projecting the data on the first terminal. The user does not need to use channels other than the projection area to share the data on the first terminal with the second terminal, thereby simplifying the user operation.

[0029] On the other hand, since the user of the first terminal can also perform a first operation on the projection area during the process of sharing the projection data of the second terminal, the first operation can be converted into a second operation on the second terminal. Thus, the user can remotely control the projection data on the second terminal by operating on the first terminal, which further simplifies the user operation.

[0030] The control method disclosed herein can be combined with remote control to enable data interaction between smart devices using a projection area, thereby improving the user's ease of operation. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1aThis illustrates a scenario where a mobile phone screen is projected onto a computer for content sharing, according to an embodiment of this disclosure.

[0033] Figure 1b This illustration depicts a scenario where two users share their screens via their respective computers, as described in an embodiment of this disclosure.

[0034] Figure 1c This illustration shows a screen projection scenario between a mobile phone and a computer during a remote conference in an embodiment of this disclosure;

[0035] Figure 2 A flowchart illustrating the steps of a control method according to an embodiment of this disclosure is shown;

[0036] Figure 3 This illustration shows a schematic diagram of a first terminal remotely controlling a second terminal through a projection area in an embodiment of this disclosure;

[0037] Figure 4a This illustration shows a schematic diagram of interface changes between two terminals under a first operation that occurs in part of the projection area in an embodiment of this disclosure;

[0038] Figure 4b This illustration shows a schematic diagram of interface changes between two terminals under a first operation that occurs in part of the projection area in an embodiment of this disclosure;

[0039] Figure 4c This illustration shows a scenario in which a first terminal corresponds to multiple second terminals in an embodiment of this disclosure;

[0040] Figure 4d This illustration shows another scenario in which the first terminal corresponds to multiple second terminals in an embodiment of this disclosure;

[0041] Figure 5 A schematic diagram of the data structure indicated by the service call in an embodiment of this disclosure is shown;

[0042] Figure 6 This illustration shows a schematic diagram of the first terminal and the second terminal establishing a screen mirroring connection in an embodiment of this disclosure;

[0043] Figure 7 This illustration shows a schematic diagram of the first terminal and the second terminal receiving screen projection data in an embodiment of this disclosure;

[0044] Figure 8a This illustration shows a schematic diagram of a first terminal sending target data to a second terminal in an embodiment of this disclosure;

[0045] Figure 8b This illustration shows a schematic diagram of a second terminal sending target data to a first terminal in an embodiment of this disclosure;

[0046] Figure 9A flowchart illustrating the steps of a control method according to Embodiment 2 of this disclosure is shown;

[0047] Figure 10 This illustration shows a schematic diagram of the process of establishing a screen mirroring connection by the second terminal in an embodiment of this disclosure;

[0048] Figure 11 This invention illustrates the structure of the request data packet reported by the first terminal and the structure of the service information data packet returned by the second terminal in an embodiment of this disclosure.

[0049] Figure 12 A flowchart illustrating the steps of a control method according to Embodiment 3 of this disclosure is shown;

[0050] Figure 13 A schematic diagram of the control device located at the first terminal in an embodiment of this disclosure is shown;

[0051] Figure 14 A schematic diagram of the control device located at the second terminal in an embodiment of this disclosure is shown. Detailed Implementation

[0052] To make the above-mentioned objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0053] In related technologies, smart devices generally include devices such as mobile phones, computers, tablets, and televisions. When using smart devices for information sharing, the content of a small-screen smart device is usually projected onto a large-screen smart device, or vice versa. It is generally limited to the projection of screen content.

[0054] As the number of applications increases, the scenarios requiring sharing also grow, along with the amount of content to be shared. Simply using screen projection for screen sharing is insufficient to meet user needs, requiring users to perform additional operations alongside projection. For example, users might send shared content from their phone to their computer via instant messaging software (such as QQ or WeChat) or email, or copy data from their phone to their computer using a portable storage device (such as a USB flash drive). This increases the complexity of the user experience.

[0055] In view of this, this disclosure proposes a control method that can integrate screen sharing, remote control and data sharing. Specifically, the control method can realize the sharing of other data besides the screen-projected data and the remote control of the screen-projected data on another terminal based on the user's operation on the projection area, so as to conveniently realize the sharing of other content in screen sharing.

[0056] To fully illustrate the application scenarios of this disclosure, refer to Figures 1a-1c The diagram shows an application scenario of the control method of this disclosure.

[0057] like Figure 1a This illustrates a scenario where a mobile phone screen is projected onto a computer for content sharing. The user shares desktop or application data from their phone to the computer, such as projecting a playing video onto the computer. If the user also needs to download content from the computer, such as documents or images, to their phone, related technologies only allow users to send documents and images from the computer to the phone through applications that can be linked to the computer, such as WeChat or QQ. In this case, the user needs to log in to WeChat or QQ on both the phone and computer simultaneously, which is cumbersome and may require minimizing the projection area, thus affecting the viewing experience of the projected video. However, using the control method provided in this disclosure, the user can directly send files and documents from the computer to the phone by manipulating the projection area, such as... Figure 1a As mentioned above, files can be directly dragged to the projection area to be sent to the mobile phone. In this way, users do not need to share files from their computer to their mobile phone through other applications, and it does not affect the user's viewing of videos shared in the projection area.

[0058] like Figure 1b The diagram illustrates a scenario where two users share their screens using their respective computers. User 1 shares the desktop of computer 1 to user 2's computer 2 via screen sharing. The shared data can be desktop data or application data; for example, in QQ screen sharing, the shared data could be the current screen data of computer 1. During the sharing process, user 1's actions on computer 1 are shared in real-time with user 2 on computer 2. If user 1 needs additional materials from user 2 for demonstration, in related technologies, user 2 minimizes or reduces the screen sharing area and then sends the materials from computer 2 to user 1 via a dialog box (such as a QQ dialog box). This process relies on the stability of the network between the two users; if the network is unstable, the dialog box may disconnect and fail to send, and the user's operation is also cumbersome. However, using the control method disclosed herein, user 2 only needs to send the files required by user 1 through the projection area, without opening a dialog box, and without affecting the continuity of screen analysis.

[0059] like Figure 1cThe diagram illustrates a screen-sharing scenario between a mobile phone and a computer during a remote meeting. The user projects the page of the meeting application from their mobile phone to the computer. In this scenario, the user still participates in the meeting by using the microphone on their phone, while the data analyzed by the participants can be viewed on the computer with its larger screen. If the user needs to share data located on the computer with the participants, the following steps are typically taken: In related technologies, the user switches the meeting from their mobile phone to their computer, making the computer the participating device, and then shares files on the computer with the participants. Alternatively, the user uses a third-party application, such as WeChat or QQ, to send files from the computer to the mobile phone, which then forwards them to the participants. This process is cumbersome. However, the control method disclosed in this paper... Figure 1c As shown, users only need to send the files they need through the projection area on their computer, without switching meeting devices or transferring files through third-party applications, which does not affect the continuity of the meeting and improves the user's ease of operation.

[0060] Among them, the above Figure 1a and Figure 1c The scenarios shown can be local area network (LAN) scenarios, for example, where the mobile phone and computer are on the same LAN, or they can communicate at close range via Bluetooth, Wi-Fi, etc.; of course, in other examples, they can also communicate through a server; the above Figure 1b In the scenario shown, the two computers can communicate through a server or be on the same local area network.

[0061] The above scenarios are illustrative examples provided for ease of understanding of this disclosure and do not represent a limitation on the application scenarios of the methods proposed in this disclosure. In practice, the control methods of this disclosure can be applied to various scenarios of content sharing between smart devices, such as the Internet of Things (IoT) scenario.

[0062] In some implementations, in addition to simplifying user operations, improved solutions are provided to enhance the security of data shared between devices, and solutions to simplify the development and design of the control methods proposed in this disclosure. Specifically, secure data transmission can be improved through security verification mechanisms, such as carrying a security identification code with each interaction between smart devices. Specific details of these solutions are described in the relevant parts of Embodiments 1, 2, and 3. Furthermore, the specific services required by the viewing device (first terminal) can be provided on the sharing device (second terminal) to simplify development on the first terminal and improve development efficiency. Specific details of these solutions are also described in the relevant parts of Embodiments 1, 2, and 3.

[0063] The control method proposed in this disclosure can be embedded in a computer storage medium and configured into the system of a smart device as part of the system, such as the Android system or Windows system of a mobile phone or computer. Users can use the method through the system control bar. In this case, users do not need to open a specific application to use the method by opening the system control bar, and it can be adapted to various applications installed on the smart device.

[0064] Alternatively, it can be configured into a specific application installed on a smart device as a function of that application. For example, it can be configured into a meeting software on a mobile phone and used by the user as a function of the meeting software. In this case, it can be applied in a local area network environment, or it can be applied beyond the local area network environment, and can achieve content sharing over a longer distance through the backend server that serves the application, thereby expanding the user's usage scenarios.

[0065] Below, based on the above description, the control method of this disclosure will be explained from the perspectives of the viewer (first terminal), the sharer (second terminal), and the combination of the two (first terminal and second terminal).

[0066] Example 1

[0067] This first embodiment describes the control method proposed in this disclosure from the perspective of the first terminal, referring to... Figure 2 As shown, a flowchart illustrating the steps of a control method according to Embodiment 1 is presented. Figure 2 As shown, the specific steps may include:

[0068] Step S201: Based on the pre-established screen mirroring connection with the second terminal, receive screen mirroring data sent by the second terminal;

[0069] Step S202: Generate a projection area corresponding to the second terminal and display the projection data in the projection area.

[0070] In this embodiment, the first terminal can be a mobile phone, computer, tablet computer, television, etc., and the second terminal can also be a smart device such as a mobile phone, computer, tablet computer, television, etc. Specifically, the first terminal and the second terminal can also be IoT devices. The screen mirroring connection between the two can be a wireless connection such as Bluetooth, WIFI, or 5G signal, or a wired network connection. The first terminal and the second terminal can be located on the same or different local area networks. The physical distance between the first terminal and the second terminal can be relatively short, such as being in the same room, or the first terminal and the second terminal can be located in different geographical locations, such as the first terminal being located in city A and the second terminal being located in city B.

[0071] In this scenario, the first terminal can establish a screen mirroring connection with the second terminal upon request, or the second terminal can establish a screen mirroring connection with the first terminal upon request. In other words, the first terminal can actively or passively establish a screen mirroring connection with the second terminal.

[0072] After establishing a screen mirroring connection, the first terminal can generate a screen mirroring area, which can be understood as a playback control used to play the screen mirroring data sent by the second terminal. Specifically, this screen mirroring area can be located on the first terminal's system desktop for user viewing. The screen mirroring area can be maximized, minimized, and its size can be adjusted by the user to suit their viewing needs and facilitate subsequent operations.

[0073] The first terminal can receive screen projection data sent by the second terminal based on the screen projection connection. This screen projection data can be obtained by the second terminal recording its own desktop, or by the second terminal recording the page of a specified application, or it can be the resource location of the media data to be projected sent by the second terminal, such as the URL (Uniform Resource Locator) address of the audio and video to be projected. In this case, the first terminal can obtain the media data from the resource location and play it.

[0074] Step S203: In response to a first operation that occurs wholly or partially in the projection area, send target data corresponding to the first operation to the second terminal to share the target data with the second terminal, and / or allow the second terminal to perform a second operation corresponding to the first operation on the projection data based on the target data.

[0075] When two terminals are projecting their screens, in order to facilitate remote control of the second terminal and data interaction by the user, the first terminal can respond to the first operation performed inside or outside the projection area and interact with the second terminal to exchange the target data corresponding to the first operation.

[0076] In some examples, the first operation may partially occur within the projection area. In this case, the target data corresponding to the first operation can be shared between the first and second terminals. This target data can be data other than the projection data. The first operation partially occurring within the projection area can mean that the first part of the operation is located in the non-projection area, and the second part is located within the projection area; or, the first part of the first operation is located within the projection area, and the second part is located in the non-projection area. For example, dragging a file from the non-projection area onto the projection area on the first terminal, or dragging a file from the projection area onto the non-projection area.

[0077] In the case where the first part of the operation is located in the non-projection area and the second part of the operation can be located in the projection area, the target data of the interaction can come from the first terminal. This target data can be called the first file, which can represent the first terminal sharing its own first file with the second terminal. In this case, the first terminal can send the first file to the second terminal.

[0078] In the case where the first part of the operation is located within the projection area and the second part of the operation is located outside the projection area, the target data for the interaction can come from the second terminal. This target data can be called the second file, which can represent that the first terminal wants to share the second file on the second terminal. In this case, the second file on the second terminal can be sent to the first terminal through control in the projection area.

[0079] In some implementations, the first operation performed by the user can occur entirely within the projection area. In this case, the user can remotely control the second terminal, and the target data corresponding to the first operation can be understood as operation instruction data used to control the second terminal. In this scenario, the first terminal can send the target data generated by the first operation to the second terminal. This target data specifically describes the first operation, instructing the second terminal to perform a second operation with the same meaning as the first operation. Here, "same meaning" can be understood as the second operation being of the same type as the first operation, such as both being double-clicks, both being single-clicks, both being right-clicks, etc.

[0080] In this scenario, the second operation performed by the second terminal can target an object located at the position where the first operation was triggered within the projection area. In some examples, the first terminal can send the location coordinates and operation type of the first operation to the second terminal along with the target data. The location coordinates can be coordinates within the projection area's coordinate system. Upon receiving the target data, the second terminal can convert the location coordinates to the coordinates of the page containing the projection content, determine the target object of the first operation, and then perform the second operation on that object.

[0081] For example, refer to Figure 3 As shown, this diagram illustrates how a first terminal remotely controls a second terminal via a projection area. Figure 1c For example, the first terminal is a computer, the second terminal is a mobile phone, and the screen projection data is the page data of the conferencing software. The screen projection area on the computer displays the conferencing interface. When the user clicks the "microphone icon" in the screen projection area to mute themselves in the meeting, the computer sends the target data to the mobile phone based on this operation (the first operation). The mobile phone then clicks the microphone icon in its own conferencing interface based on the target data, thereby turning off the mobile phone's microphone and achieving mute.

[0082] Of course, in some implementations, the first terminal can also receive control commands sent by the second terminal for the projection area. These control commands are sent by the user of the second terminal when operating on the projection content. The first terminal can then execute control operations based on these commands. For example, the first terminal is a computer, the second terminal is a mobile phone, and the projection data is audio and video data. The computer obtains the audio and video data based on its URL address and plays it in the projection area. The mobile phone clicks pause, fast forward, or rewind on the audio and video data. These pause, fast forward, and rewind commands are sent to the computer. The computer pauses the playback of the audio and video data according to the pause command, fast forwards according to the fast forward command (e.g., requesting more resources or fast forwarding from buffering), and rewinds according to the rewind command. Thus, the computer can accept remote control from the mobile phone during projection.

[0083] By adopting the technical solution of this embodiment, when the first operation partially occurs within the projection area, the first terminal can share its own target data with the second terminal. Furthermore, through this first operation, the first terminal can share target data on the second terminal, excluding the projection data. Thus, the user only needs to perform the first operation within the projection area to share target data with the second terminal while simultaneously sharing the projection data on the first terminal. The user does not need to use channels outside the projection area to share data from the first terminal with the second terminal, thereby simplifying user operation. Moreover, during the sharing of projection data from the second terminal, the first terminal can also control the second terminal to perform a second operation based entirely on the first operation occurring within the projection area, thereby enabling remote control of the second terminal and further simplifying user operation.

[0084] In some examples, the screen-casting data may include desktop data of the second terminal and / or page data of applications configured on the second terminal; wherein the applications include at least one of office applications, media applications, and social applications.

[0085] Desktop data can refer to the recording of the desktop of the second terminal by the second terminal. In this case, the user operation of the second terminal and the changes in the desktop of the second terminal can be displayed in the projection area, and the recording can be sent to the first terminal.

[0086] The page data can refer to the recording of the page of the specified application by the second terminal. In this case, the screen projection data may not include data from other applications besides the specified application. For example, if the specified application is an office application, the screen projection data will only include the document page opened on the desktop, and the application icons on the desktop will not be displayed in the projection area.

[0087] In this scenario, since the first terminal and the second terminal can exchange target data following the first operation, in one case, for the first operation that partially occurs within the projection area, the target data for the exchange can be a corresponding file; in another case, for the first operation that entirely occurs within the projection area, the target data for the exchange can be operation data describing the first operation. Therefore, the target data can include data for sharing and data for remote control. The shared data can be understood as the data selected in the first operation, and the data for remote control can be understood as the instruction data generated along with the first operation, which is used for remote control of the second terminal.

[0088] Accordingly, in response to a first operation in which part of the operation occurs in the projection area and the other part occurs in the non-projection area, the target data targeted by the first operation can be shared with the second terminal. The non-projection area is an area located outside the projection area. The target data targeted by the first operation can come from either the first terminal or the second terminal. The first operation in which part of the operation occurs in the projection area and the other part occurs in the non-projection area can be understood as a first operation that occurs both in the projection area and in the non-projection area.

[0089] Furthermore, in response to the first operation that occurs entirely within the projection area, operation data can be generated, and the operation data can be encapsulated into target data and sent to the second terminal so that the second terminal can perform a second operation on the projection data.

[0090] In some implementations, sharing target data can include two methods. One method is to share data based on a first operation that partially operates within the projection area. In this case, the target data targeted by the first operation can be data on the first terminal or data on the second terminal. The other method is to share data through the initiative of the second terminal, such as the second terminal actively uploading its own target file to the first terminal via projection.

[0091] In cases where target data sharing is achieved through a first operation partially occurring within the projection area, the first terminal can determine whether to share the target data located on the first terminal or the first terminal to share the target data located on the second terminal, based on the first operation that partially occurs within the projection area. Specifically, if the first operation is performed from a non-projection area towards the projection area, the first terminal shares the target data on the first terminal with the second terminal; if the first operation is performed from the projection area towards a non-projection area, the first terminal can receive the target data shared by the second terminal.

[0092] In practice, in response to the operation of dragging target data from the non-projection area into the projection area, the target data can be sent to the second terminal for the second terminal to store the target data.

[0093] Additionally, it can receive target data sent by the second terminal and store the target data in response to a drag operation from the target data of the projection area to a non-projection area.

[0094] In this embodiment, the shared target data can be files, such as image files and other file formats. Taking a file as an example, when a user on the first terminal selects a file on the first terminal and drags it into the projection area, it indicates that a first operation of dragging from a non-projection area into the projection area has occurred. In this case, the first terminal can send the file to the second terminal based on the projection connection. For example, if a user selects a file on the desktop of the first terminal and holds down the left mouse button to drag it into the projection area, a data sending thread will be triggered, sending the file to the second terminal through the projection connection. Accordingly, the second terminal will receive the file, and in one case, it can automatically store it in a preset address, as detailed in Embodiment 2.

[0095] Taking a file as an example, when a user on the first terminal selects a file on the second terminal within the projection area and performs a drag operation representing dragging it to a non-projection area, it signifies that the user on the first terminal wants to access the file located on the second terminal. This drag operation representing dragging to a non-projection area can mean that the file is not actually moved within the projection area; that is, the file does not move with the projection. However, this operation symbolizes dragging the file to a non-projection area. For example, specific operations can be designed to represent dragging a file from the projection area to a non-projection area, such as a mouse drag operation by holding down the left mouse button, or a finger swipe operation after touching the file.

[0096] For example, if a user selects an image file in the projection area and holds down the left mouse button while dragging the cursor to a non-projection area, a data sending thread will be triggered. This data sending thread will send the first operation back to the second terminal in the form of an instruction. Based on this instruction, the second terminal will send the selected file to the first terminal via the projection connection. At this time, the first terminal will receive the file it selected in the projection area. In this way, users can easily obtain files from different terminals by dragging the file from the projection area to the non-projection area.

[0097] Reference Figure 4a and Figure 4b As shown, this diagram illustrates the interface changes of the two terminals during the first operation that occurs in the projection area. Figure 4aAs shown, when a user drags an image from the computer's desktop into the computer's projection area, the phone will display a notification that the image from the computer has been received. Figure 4b As shown, the user selects a file in the projection area of ​​computer 2, and drags the cursor to the desktop of the computer while holding down the left mouse button. Computer 1 will be prompted to send the file to the computer. Clicking OK will display the image received from computer 1 on computer 2.

[0098] In the case where data interaction is initiated by the second terminal, the first terminal can receive the target file uploaded by the second terminal for the projection area. The target file is sent by the second terminal after performing an upload operation on the target file.

[0099] In this case, the first terminal will be notified that it has received the target file sent by the second terminal. The file can be sent via screen mirroring. The process of the second terminal sending the target file can be referred to in Embodiment 2.

[0100] In some implementations, for the first operation in which all operations are located in the projection area, the first terminal can provide the user with the function of remotely controlling the second terminal. Accordingly, the first operation that occurs entirely in the projection area can be called a trigger operation. The trigger operation can include some common functional operations, such as left and right mouse clicks, finger touch, and sliding operations with the left mouse button held down.

[0101] In practice, it can respond to all trigger operations performed in the projection area, generate operation data, and send the operation data to the second terminal so that the second terminal can perform a second operation corresponding to the operation type on the projection data located at the operation position;

[0102] The operation data is used to indicate the operation location and operation type of the first operation.

[0103] Accordingly, in some implementations, the operation type of the triggering operation may include at least one of the following: single click, double click, touch, right click, and drag. Of course, other types may also be included in practice. Specifically, the scenario for this remote control can be referred to... Figure 3 As shown, and refer to Figure 3 The description is sufficient and will not be elaborated further here.

[0104] In some implementations, as described above, this disclosure can also improve the secure transmission of data through a security verification mechanism.

[0105] One type of security verification mechanism involves the first terminal initiating the process. From the moment the first terminal establishes a connection with the second terminal, each interaction uses a security identification code for verification to prevent data leakage. For example, the data sent from the first terminal to the second terminal may include a first security identification code generated by the first terminal to request verification from the second terminal, thus preventing unauthorized terminals from requesting data from the second terminal and from obtaining data from the first terminal.

[0106] In practice, the screen mirroring connection can be established based on a security verification mechanism. The specific process can be as follows: the first terminal responds to the connection request sent by the second terminal, initiates a preset security verification mechanism, and then establishes a screen mirroring connection based on the security verification mechanism. The security verification mechanism is used to instruct that a first security identification code be carried each time data is sent to the second terminal for the second terminal to verify.

[0107] During the process of establishing a screen mirroring connection, each interaction between the first terminal and the second terminal can carry a first security identification code, so that the second terminal can verify the identity of the first terminal and prevent an unauthorized terminal from becoming the screen mirroring terminal of the second terminal.

[0108] When interacting with the second terminal to exchange target data corresponding to the first operation, the target data can also be exchanged with the second terminal based on a security verification mechanism; this includes the target data sent to the second terminal carrying a first security identification code, and the operation data sent to the second terminal carrying a first security identification code.

[0109] In this embodiment, the first security identification code can be a pre-agreed fixed code, such as the identification codes of both parties, which can be in string form. Of course, the first security identification code can also change dynamically, such as multiple security identification codes changing in a certain order, so that the first security identification code sent by the first terminal to the second terminal each time can present a pre-defined change, thereby improving security.

[0110] The first security identification code can be carried in the header of the data packet so that the second terminal can extract and verify it.

[0111] In some implementations, the first security identification code can be dynamically generated in real time by the first terminal to ensure the security of the first security code. Specifically, the first security identification code can be obtained by encrypting the first target information using a preset encryption algorithm; wherein, the first target information includes at least one of the first terminal's identifier, the current time, and the screen projection data.

[0112] In this embodiment, a key mechanism can be used for security verification. For example, the first terminal stores the corresponding private key, and the second terminal stores the public key corresponding to the private key. The first terminal encrypts the first target information according to the private key to obtain the first security identification code, and the second terminal decrypts and verifies it using the public key. The first target information may include the identifier of the first terminal and the current time, or it may include the identifier of the first terminal, the current time, and the projection data. The projection data may be a URL address in some cases, or the identifier of the application from which the projection data originates in others, such as the identifier of the conferencing software; or, if the projection data is desktop data, it may be the identifier of the desktop.

[0113] Another security verification mechanism involves the first terminal passively verifying the data sent by the second terminal. This allows the first terminal to verify the security of the data sent by the second terminal, ensuring the security of the data shared by the second terminal. Alternatively, in this mechanism, the first terminal can also actively employ a security verification method to communicate with the second terminal; for example, the data sent from the first terminal to the second terminal can carry a first security identification code.

[0114] Under this security verification mechanism, the first terminal can perform security verification on the screen projection data sent by the second terminal. The screen projection data can carry a second security identification code, which is generated by the second terminal based on a preset security verification mechanism. When displaying the screen projection data sent by the second terminal in the screen projection area, the second security identification code can be verified first. After successful verification, the screen projection data can be displayed in the screen projection area.

[0115] Similarly, when verifying the second security identifier, the first terminal can use its private key to decrypt the screen projection data and then perform security verification. If the verification is successful, the screen projection data will be played. The second security identifier can also be located in the header of the screen projection data packet. The generation of the second security identifier can be the same as the generation of the first security identifier, and will not be elaborated here.

[0116] It should be noted that if the verification fails, it indicates that the screen projection data sent by the second terminal has not been authorized for projection. In this case, a prompt message can be sent to the second terminal to inform the user that the screen projection data cannot be projected and that permission needs to be obtained before projection. At this time, the user can operate the second terminal to apply for screen projection permission from the first terminal. If the application is successful, an authorization code can be sent to the first terminal. After receiving the authorization code, the first terminal can display the projection data in the projection area.

[0117] By adopting the above implementation scheme, the connection security between the first terminal and the second terminal can be ensured during the establishment of the screen projection connection, thus preventing unauthorized terminals from becoming the screen projection terminal of the second terminal.

[0118] In some examples, this disclosure also provides another solution for maintaining data security and remote control security. In some examples, the user's remote control of the second terminal can be prohibited by locking the projection area on the first terminal, and the non-projection area can be locked to prevent the first terminal and the second terminal from exchanging non-projection data.

[0119] Specifically, before interacting with the second terminal on the target data corresponding to the first operation in response to a first operation that occurs wholly or partially in the projection area, the interaction permissions corresponding to the second terminal can be determined first. These interaction permissions can be used to characterize the scope of permissions for remote control and data interaction between the first terminal and the second terminal during the process of sharing the projection data of the second terminal, such as whether the first terminal is allowed to remotely control the second terminal, whether the target data on the first terminal is allowed to be interacted, and whether the first terminal is allowed to obtain the target data on the second terminal through the first operation.

[0120] Specifically, when the interaction permission only includes operation control rights, it means that the first terminal is only allowed to remotely control the second terminal, but not to exchange target data with each other during remote control; while when the interaction permission only includes file interaction rights, it means that the first terminal and the second terminal are only allowed to exchange target data, but the first terminal is not allowed to remotely control the second terminal, that is, the first terminal is not allowed to send operation data to the second terminal.

[0121] Specifically, when the interaction permissions only include operation control, the area outside the projection area can be locked to prevent the response to the first operation from the projection area to the non-projection area, or to prevent the response to the first operation from the non-projection area to the projection area.

[0122] If the interaction permissions only include file interaction rights, lock the projection area to prevent all operations from responding to the first operation that occurs in the projection area.

[0123] If the interaction permissions only include operational control rights, then only the first terminal is allowed to remotely control the second terminal. This allows locking the non-projection area, meaning the first terminal cannot detect the portion of the first operation located outside the projection area. Since the first operation, partially located within the projection area, is used to share target data with the second terminal, locking the non-projection area prevents this data sharing. For example, dragging target data from the projection area to the non-projection area will not trigger the second terminal to send the target data to the first terminal, and dragging target data from the non-projection area to the projection area will not trigger the first terminal to send the target data to the second terminal. This prevents the target data from being shared by other terminals without permission.

[0124] If the interaction permissions only include file interaction rights, then only the first and second terminals are allowed to share target data, but the first terminal is not allowed to remotely control the second terminal. In this case, the projection area can be locked. This locking means that operations performed by the first terminal within this projection area are restricted. In practice, the non-projection area can be unlocked to respond to the first operation performed between the projection area and the non-projection area. For example, operation data such as single-clicking or double-clicking performed within the projection area cannot be sent to the second terminal.

[0125] Interaction permissions can include file interaction rights and operation control rights. In this case, the projection area and the non-projection area are unlocked, so that when the first operation occurs between the projection area and the non-projection area, the target data can be shared, and when the first operation occurs in all projection areas, operation data can be generated and sent to the second terminal.

[0126] In this process, the acquisition of operational control rights within the interaction permissions is initiated by the second terminal. File interaction rights can be determined by preset data sharing permissions on the first terminal and interaction right information sent by the second terminal. The preset data sharing permissions determine whether target data on the first terminal is allowed to be shared to the second terminal, while the interaction right information determines whether target data on the second terminal is allowed to be shared to the first terminal. Thus, file interaction rights can be further divided into a first interaction right based on the first terminal and a second interaction right based on the second terminal.

[0127] In the case of obtaining the first interaction right but not the second interaction right, the first terminal can respond to the first operation from the projection area to the non-projection area, that is, the first terminal can send the target data located on the first terminal to the second terminal, while it is prohibited to respond to the first operation from the non-projection area to the projection area, that is, the first terminal cannot obtain the target data on the second terminal.

[0128] In the case where the second interaction right is obtained but the first interaction right is not obtained, the response to the first operation from the projection area to the non-projection area can be prohibited, while the response to the first operation from the non-projection area to the projection area can be allowed. That is, the first terminal can obtain the target data on the second terminal, while the second terminal cannot obtain the target data on the first terminal.

[0129] Of course, if neither party acquires the second interaction right and the first interaction right, they cannot share the target data with each other. However, if both parties acquire the second interaction right and the first interaction right, they can share each other's target data.

[0130] With this implementation scheme, data interaction can only be performed when the corresponding permissions are obtained, which can further improve the security of data interaction between the first terminal and the second terminal.

[0131] In some embodiments of this disclosure, the specific services required by the first terminal can also be provided to the second terminal to simplify development on the first terminal and thus improve development efficiency. Specifically, when the first terminal remotely controls the second terminal through a first operation, the first terminal can directly send target data with a standardized data structure to the second terminal, so that the second terminal can execute the second operation through its control logic by parsing the data structure of the target data. In this way, there is no need to develop an instruction architecture adapted to the operation control of the second terminal on the first terminal, and the development can be carried out on the second terminal, thereby reducing the amount of development work and improving efficiency.

[0132] In practice, after establishing a screen mirroring connection and before generating a screen mirroring area, the first terminal can obtain service information sent by the second terminal based on the screen mirroring connection, and activate at least one service call that matches at least one screen mirroring control service based on the service information.

[0133] The service information includes at least one screen projection control service provided by the second terminal; wherein different screen projection control services are used to perform different second operations.

[0134] Accordingly, when the first operation occurs entirely in the projection area, the process of responding to the first operation can be as follows: triggering a service call corresponding to the operation type of the first operation, and encapsulating the operation data of the first operation based on the data structure indicated by the triggered service call to obtain target data. This target data can be understood as instruction data used to control the second terminal, which can be used by the second terminal to call the corresponding projection control service to execute the second operation corresponding to the first operation.

[0135] In this embodiment, the first terminal maintains multiple service calls, and the second terminal maintains multiple screen casting control services. The service calls and screen casting control services are matched one-to-one. For example, for a mouse click, the first terminal maintains a service call 1, and the second terminal also maintains a screen casting control service 1. When a mouse click is performed on the first terminal, the operation data is encapsulated using service call 1 and sent to the second terminal. The second terminal then calls screen casting control service 1 to perform the mouse click operation.

[0136] Reference Figure 5 As shown, a schematic diagram of the data structure indicated by a service call is illustrated, such as... Figure 5 As shown, the top diagram illustrates the data structure of the operation data sent by the first terminal, while the bottom diagram shows the data structure of the data fed back from the second terminal to the first terminal based on the operation data. Specifically, the data packet mainly includes a header and a body. Figure 5The system mechanism identification string mentioned above is the security identification code. Its header structure includes packet length, protocol version, system mechanism identification string (first security identification code or second security identification code), operation type, operation location, etc. The operation type is further divided into service categories and service subcategories, such as... Figure 5 The "MouseClick" service shown is the main service category, representing a mouse click; "SingleClick" is the sub-service category, representing a single click, separated by "\0". This service call allows data packets to be encapsulated according to this data structure when the first operation is triggered, obtaining the operation data, which is then sent to the second terminal.

[0137] For the second terminal, its data structure can be parsed to obtain service categories and service subcategories. Then, according to the service subcategories, the corresponding screen projection control service is triggered, which is its own control logic for the screen projection data. Following this control logic, the second operation is executed. Because service calls are managed according to service categories and service subcategories, the second terminal can quickly parse out the required screen projection control service, thereby efficiently executing the second operation and achieving efficient remote control.

[0138] The matching between the service calls on the first terminal and the screen mirroring control service on the second terminal is as follows:

[0139] In some cases, A, the service calls on the first terminal can correspond one-to-one with the screen casting control service. That is, the types and number of service calls on the first terminal are consistent with the types and number of screen casting control services on the second terminal. In this case, the two are completely matched, and the first terminal activates all service calls.

[0140] In some other cases (B), the service calls on the first terminal may correspond only to a portion of the screen mirroring control services on the second terminal. For example, the types of service calls configured on the first terminal may only be included in the screen mirroring control services on the second terminal. In other words, the service calls on the first terminal are fewer than those on the screen mirroring control services on the second terminal. In this case, the first terminal still activates all service calls.

[0141] In some other cases, C, the service calls on the first terminal may exceed the screen mirroring control service on the second terminal. In this case, the first terminal only activates the call service corresponding to the screen mirroring control service on the second terminal.

[0142] In some other cases, D, the service calls on the first terminal may be partially the same as the screen mirroring control service on the second terminal, but partially different from the screen mirroring control service on the second terminal. In this case, the first terminal only activates the call service that matches the screen mirroring control service on the second terminal.

[0143] In cases where the service call of the first terminal does not match the screen casting control service on the second terminal, such as cases B, C, and D, the first terminal can first identify the target service call that does not match the screen casting control service, and download and / or upgrade the target service call, as well as activate the target service call.

[0144] The target service call includes: non-existent service calls and / or mismatched service calls. A non-existent service call may mean that a screen casting control service exists on the second terminal, but there is no corresponding service call for the screen casting control service on the first terminal, which is considered as the first terminal lacking a corresponding service call; a mismatched service call may mean that the version of the service call on the first terminal is different from the version of the screen casting control service on the second terminal.

[0145] For mismatched target service calls, the target service call can be downloaded, i.e., a new target service call can be downloaded directly; alternatively, a service call that does not exist in the target service call can be downloaded to upgrade the mismatched service call. The download and upgrade processes can be performed according to relevant technologies, which will not be elaborated upon here.

[0146] In some embodiments, for a mismatched target service call, the first terminal may immediately log the call and output a prompt message to remind the user whether to upgrade the service.

[0147] In some implementations, the first terminal can also share screen projection data from multiple second terminals. In some examples, the first terminal can generate multiple screen projection sub-regions, with different screen projection sub-regions corresponding to different second terminals to display screen projection data sent by different second terminals.

[0148] In some other examples, the first terminal can generate multiple projection sub-areas based on the projection area of ​​the second terminal. Different projection sub-areas can display different projection content on the second terminal. For example, one projection sub-area can display the movie or TV show that needs to be played on the second terminal, while another projection sub-area can display the remote meeting interface held on the second terminal. Different projection sub-areas can interact with each other through independent projection connections.

[0149] In some examples, the first terminal can generate multiple projection sub-regions based on the projection area of ​​the second terminal. Different projection sub-regions can display the same projection content from the second terminal, but with different projection functions. For instance, multiple projection sub-regions may all display the remote conference interface on the second terminal, but one sub-region might respond to a portion of the first operation occurring there, sharing target data with the second terminal, while another sub-region might respond to all of the first operation occurring there, remotely controlling the second terminal to execute a second operation. This approach allows the first terminal to monitor different operations in different regions, and different regions can be bound to one or a few listening threads, improving the independence and effectiveness of monitoring.

[0150] Specifically, in this embodiment, projection data sent by each of the multiple second terminals can be displayed in multiple projection sub-areas. Correspondingly, when the first terminal sends target data corresponding to the first operation to the second terminal in response to a first operation that occurs entirely or partially in the projection area, it can also send target data to the second terminal corresponding to the target projection sub-area in response to a first operation that occurs entirely or partially in the target projection sub-area among the multiple projection sub-areas.

[0151] In this embodiment, for each projection sub-area, the first terminal monitors the first operation for that projection sub-area. Depending on whether the first operation occurs partially or entirely in that projection sub-area, the first terminal sends corresponding target data to the second terminal. If the first operation occurs partially in that projection sub-area, the first terminal sends the target data located on the first terminal to the second terminal corresponding to that projection sub-area according to the operation direction of the first operation, or receives the target data located on the second terminal sent by the second terminal. If the first operation occurs entirely in that projection sub-area, the first terminal sends the operation data encapsulated by the triggered service call to the second terminal corresponding to that projection sub-area according to the operation type and location of the first operation.

[0152] Among them, the projection sub-areas corresponding to different second terminals can be located in different display areas of the first terminal, or the projection sub-areas corresponding to different second terminals can have overlapping parts.

[0153] In one example of this implementation, a user of the first terminal can share data among multiple second terminals through a simplified operation. A sharing area can be set within the projection area, thus including not only multiple projection sub-areas but also a shared area for these sub-areas to share. The first terminal can use this shared area to send the shared data to each of the multiple second terminals. Furthermore, the first terminal can also use this shared area to remotely control multiple second terminals simultaneously, enabling them to perform the same operation.

[0154] In specific implementation, the first terminal may respond to the third operation that occurs in whole or in part in the shared area by sending the shared data corresponding to the third operation to multiple second terminals, so that the multiple second terminals can share the shared data and / or all of them can execute the fourth operation corresponding to the third operation.

[0155] The shared area can be a region different from any other projection sub-region, or the shared area can be an overlapping region of multiple projection sub-regions. Of course, in order to more easily distinguish the sharing operation, the shared area can be a region different from any other projection sub-region.

[0156] In this embodiment, when the first terminal detects a third operation that occurs partially or entirely in the shared area, it can share the shared data located on the first terminal with multiple second terminals, or it can instruct multiple second terminals to perform the same fourth operation.

[0157] In the case where the third operation occurs in the shared area, the first terminal sends shared data to multiple second terminals. Specifically, the third operation can be the operation of dragging shared data from the non-projection area to the shared area. The shared data can be a file located on the first terminal, which can be a document, an image, or even a program file. The shared data can be understood as a file that is needed by multiple second terminals.

[0158] In the case of the third operation, which occurs entirely in the shared area, the operation data can be sent to multiple second terminals simultaneously, thereby enabling the same action to be performed on multiple second terminals. In other words, the fourth operation and the third operation can be of the same type.

[0159] Reference Figure 4c and 4d As shown, a schematic diagram of a scenario where a first terminal corresponds to multiple second terminals is illustrated. Figure 4c As shown, computer 2 is the first terminal, and computers 12 and 11 are the second terminals. The first terminal is equipped with multiple screen projection sub-areas. When the first operation occurs in screen projection sub-area 1, the corresponding computer 11 will respond accordingly. For example, as shown in 4c, the operation for logging into QQ includes: receiving target data, performing the second operation, and sending target data to computer 2.

[0160] like Figure 4d As shown, computer 2 is the first terminal, and computers 12 and 11 are the second terminals. Multiple projection sub-areas and sharing areas are set up on the first terminal. When a new creation operation occurs in the sharing area, computers 11 and 12 will simultaneously execute the creation action; when shared data, such as files, is dragged into the sharing area, computers 11 and 12 will simultaneously receive the shared data. Figure 4d The process is shown by the dashed line.

[0161] In this implementation, for third operations that all occur within the shared area, it can include a "create" operation by right-clicking the mouse. For example, if a user right-clicks in the shared area, a pop-up menu appears, and then clicks "create," the operation data for this "create" operation is simultaneously sent to multiple second terminals, and these second terminals simultaneously create new folders, enabling one-click creation operations across multiple second terminals. As another example, if a user right-clicks in the shared area, a pop-up menu appears, and then clicks "refresh," the operation data for this "refresh" operation is simultaneously sent to multiple second terminals, and these second terminals simultaneously execute the refresh operation.

[0162] Thus, by sharing a region, simultaneous control of multiple second terminals is achieved, and the operation is made simpler.

[0163] To fully illustrate the scheme of Embodiment 1, a specific example 1 is used to explain the control method executed by the first terminal, referring to... Figure 6 , Figure 7 , Figure 8a and Figure 8b As shown, the flowcharts illustrate the process of establishing a screen mirroring connection, receiving screen mirroring data, and interacting with target data by the first terminal and the second terminal, respectively. Specifically, Example 1 uses the above-mentioned... Figure 1a Taking the screen mirroring scenario between a mobile phone and a computer as an example, it specifically includes:

[0164] S1: The first terminal receives a connection request sent by the second terminal, wherein the connection request can be... Figure 6 The broadcast data packet shown carries a second security identification code, which is obtained by encrypting the identifier of the second terminal using a preset encryption algorithm.

[0165] S2: The first terminal parses the broadcast data packet and verifies the second security identification code to determine whether it is applicable to the local machine. The first terminal uses a preset decryption algorithm to decrypt and verify the data.

[0166] S3: If applicable, the first terminal sends a reply data packet to the second terminal. The reply data packet carries the first security identification code. The second terminal will verify the first security identification code. After successful verification, the network card information is set to support P2P connection. In this case, the second terminal acts as the server device of the first terminal, and the first terminal acts as the client device of the second terminal. Thus, the screen mirroring connection is established.

[0167] If the connection establishment times out during this process, the connection will be re-established.

[0168] S4: The first terminal requests service information from the second terminal, and the service information also carries the first security identification code;

[0169] S5: The first terminal registers and activates its own configured service call that matches the screen projection control service based on the service information sent by the second terminal;

[0170] S6: Generate a projection area and receive projection data sent by the second terminal through the projection connection;

[0171] S7: The first terminal verifies the second security identification code carried in the screen projection data. After successful verification, the screen projection data is displayed in the screen projection area, thus realizing the screen data sharing between the two terminals.

[0172] S8: If the first terminal detects an operation of clicking on target data 1 outside the projection area and dragging it into the projection area, it sends target data 1 to the second terminal via the projection connection, and the second terminal receives it directly; target data 1 also carries the first security identification code; the process is as follows. Figure 8a As shown, Figure 8a The boundary crossing event is the FileMoveout event that occurs when a device moves from a non-projection area to the projection area.

[0173] S9: If the first terminal detects an operation of dragging target data 2 from the projection area to a non-projection area, it encapsulates the operation data packet of target data 2 and sends it to the second terminal, and also receives target data 2 sent by the second terminal; the operation data packet also carries the first security identification code; the process is as follows: Figure 8b As shown, Figure 8b The boundary crossing event refers to the FileMovein event where the projected area moves to a non-projected area.

[0174] S10: If the first terminal detects that the left mouse button has double-clicked on file 3 in the projection area, it encapsulates the operation data according to the corresponding service category and service subclass and sends it to the second terminal. The second terminal parses the operation data and calls the projection control service corresponding to the service subclass according to the service category and service subclass to execute the operation of opening file 3. The operation data packet carries the first security identification code.

[0175] S11: The first terminal periodically obtains the service information of the second terminal and upgrades or downloads the service for any mismatched targets.

[0176] Example 2

[0177] This second embodiment describes the control method proposed in this disclosure from the perspective of the second terminal, referring to... Figure 9 As shown, a flowchart of the control method described in Embodiment 2 is illustrated. Figure 9 As shown, the specific steps may include:

[0178] Step S301: Based on the pre-established screen projection connection with the first terminal, send data to the first terminal so that the first terminal can display the screen projection data in the screen projection area;

[0179] The screen mirroring connection can be a local area network (LAN) connection or an internet connection. In this way, the first terminal and the second terminal can be in the same LAN. The process of establishing a screen mirroring connection can be as follows: In response to a target operation triggered on the second terminal, the second terminal scans multiple terminals to be mirrored in the same LAN as the first terminal and displays the terminal identifiers corresponding to each of the multiple terminals to be mirrored; then, in response to a selection operation of the multiple terminal identifiers performed on the second terminal, a screen mirroring connection can be established with the first terminal corresponding to the selection operation.

[0180] like Figure 6 The diagram illustrates the process of establishing a screen mirroring connection. In this process, the second terminal can send a broadcast data packet to the first terminal. Then, after the first terminal parses the broadcast data packet and grants permissions, a screen mirroring connection can be established between the two terminals.

[0181] Step S302: Interact with the first terminal to obtain the target data corresponding to the first operation; wherein, the first operation may occur partially or entirely in the projection area on the first terminal;

[0182] Step S303: Store the target data, and / or perform a second operation on the projected data corresponding to the first operation based on the target data.

[0183] In this embodiment, the second terminal can receive target data sent by the first terminal. The target data can be target data on the first terminal or operation data when performing the first operation. For the target data, the second terminal can store it in a preset address. For the operation data, the second terminal needs to perform a second operation corresponding to the first operation on the screen projection data.

[0184] The second terminal can also send target data to the first terminal. Specifically, if the first operation occurs partially in the projection area and is performed from the projection area to the non-projection area, the second terminal will receive the operation data packet sent by the first terminal. Based on the operation data packet, the second terminal will send the target data located on the second terminal to the first terminal. The target data is the file targeted by the first operation.

[0185] The second operation performed by the second terminal can target an object located at the position in the projection area where the first operation was triggered. Upon receiving the target data, the second terminal can convert the position coordinates into coordinates of the page containing the projection content, then determine the object targeted by the first operation, and finally perform the second operation on that object. For details, see [link to details]. Figure 3As shown, the second terminal follows the action of clicking the "microphone icon" in the projection area on the first terminal, and clicks the microphone icon in its own meeting interface to mute the phone's microphone.

[0186] Using the technical solution of this embodiment two, the second terminal can send its own screen projection data to the first terminal for sharing, and can execute the second operation corresponding to the first operation to accept the remote control of its own screen projection data by the first terminal. At the same time, based on the first operation that occurs in part of the screen projection area, it can interact with the first terminal to obtain target data, thereby enabling remote control and file sharing between the two terminals.

[0187] The process of establishing a screen mirroring connection will be explained first, in order:

[0188] In some implementation schemes of this second embodiment, as described above, the control method of this second embodiment can be solidified into a computer-readable storage medium and configured into an application of a smart device or a system of a smart device. In the case of being configured into an application of a smart device, a screen projection connection between the second terminal and the first terminal can be triggered through the function buttons configured in the application. In this case, the application needs to be pre-installed on the second terminal before it can be used.

[0189] In some implementations of this second embodiment, when a computer-readable storage medium is configured in the system of a smart device, the target operation can be triggered directly on the system desktop of the second terminal to establish a screen mirroring connection. Since this function is directly located in the system of the second terminal, it can be triggered from the system control bar of the second terminal. This reduces the dependence on the applications configured on the second terminal and makes it more convenient for users to operate.

[0190] Specifically, the screen mirroring connection can be established based on a target operation triggered on the system desktop of the second terminal, wherein the target operation includes triggering a target button on the system control bar of the second terminal.

[0191] Reference Figure 10 The diagram illustrates the process of establishing a screen mirroring connection on the second terminal. Figure 10 As shown, in response to the triggering operation of the target button on the system control bar, the second terminal can start scanning for surrounding smart devices. Specifically, it can send broadcast data packets to surrounding devices. Devices within the same local area network will definitely receive the data packets. After receiving the data packets, it parses them. If the parsing is applicable to the local machine, it will return the data packets. The second terminal can display the identifiers of the devices that returned the data packets on the second terminal in the form of icons, indicating that the surrounding devices have been found.

[0192] Next, in response to the user's action of dragging an icon to a designated area, the second terminal can send a connection request to the first terminal corresponding to that icon. After receiving confirmation from the first terminal, a screen mirroring connection is established between the two terminals. During this process, after receiving confirmation from the first terminal, the second terminal can set its direct-connect virtual network adapter to the IP address of the first terminal. At this time, the second terminal's direct-connect virtual network adapter is set to the gateway address of the first terminal's IP address. For example, if the first terminal is set to 192.168.1.12, the gateway can be set to 192.168.1.1. In this case, the second terminal acts as the server device, and the first terminal acts as the client device.

[0193] After establishing a screen mirroring connection, screen mirroring data can be sent to the first terminal. As described in Embodiment 1 above, the screen mirroring data can be page data of an application on the second terminal, or desktop data of the second terminal. Specifically, it can be based on the user's selection of the content to be mirrored.

[0194] In some implementations, when sending screen projection data to the first terminal, the content to be projected can be determined based on the selection operation of the content to be projected; and in response to the display of the content to be projected on the first terminal, the content to be projected can be recorded to obtain screen projection data.

[0195] The selection operation includes selecting an application and selecting a desktop; the application includes at least one of office software applications, media applications, and social applications.

[0196] In practice, if the operation involves selecting an application, the second terminal will start recording a screenshot of the opened application's page when the selected application is opened, and send the recorded data as screen projection data to the first terminal to avoid leaking other data on the second terminal besides the selected application; if the operation involves selecting the desktop, the second terminal will record the desktop and send the recorded data as screen projection data to the first terminal. In this way, the operation process performed by the user on the second terminal can be shared with the first terminal.

[0197] After sending the projection data to the first terminal, the first terminal can play the projection data in the projection area. Based on the interaction permissions, the second terminal can receive remote control of the projection data from the first terminal, or the second terminal can interact with the first terminal to obtain the target data.

[0198] In some implementation schemes of this second embodiment, corresponding to the first embodiment above, in screen sharing, the first terminal and the second terminal can also share data other than the screen projection data. For example, the first terminal can share its own target data with the second terminal, and the second terminal can also share its own files with the first terminal. In addition to responding to the first operation that occurs on the screen projection area of ​​the first terminal and sending the target data of the first operation to the first terminal, the second terminal can also actively send its own target files to the first terminal through the screen projection connection.

[0199] In practice, in response to the upload operation performed on the target file on the second terminal, the target file can be sent to the first terminal; wherein, the upload operation includes any of the following: the control selection operation in the right-click menu bar, or the operation of dragging the target file into the preset icon corresponding to the projection area.

[0200] In this embodiment, after the second terminal establishes a screen mirroring connection with the first terminal, it can generate a preset icon on the second terminal corresponding to the screen mirroring area. This preset icon can be used to help the second terminal upload the target file to the first terminal. In practice, the second terminal can trigger the target file sending thread in response to the operation of dragging the target file into the preset icon. This sending thread is used to send the target file to the first terminal through the screen mirroring connection. Alternatively, it can be sent to the first terminal through Bluetooth communication, near-field communication, etc. between the first and second terminals. In this way, the connection resources of the screen mirroring connection can be avoided.

[0201] In another approach, after the second terminal establishes a screen mirroring connection with the first terminal, the second terminal can add an upload control corresponding to the first terminal. This upload control is located in the menu bar that appears when the mouse is right-clicked, and it is associated with a file sending thread. This sending thread is used to send the target file to the first terminal via the screen mirroring connection or Bluetooth connection. When the second terminal detects that the upload control has been selected for the target file, such as when the user right-clicks the target file and then left-clicks to select the upload control, the target file can be sent to the first terminal via Bluetooth communication, near-field communication, or other means between the first and second terminals.

[0202] Using the above implementation method, the second terminal can actively upload files to the first terminal. This method eliminates the need to drag and drop files into a dialog box to complete the transfer, as is required in related technologies. It can reduce reliance on social communication software and simplify user operations.

[0203] In the interaction between the second terminal and the first terminal, as described in Embodiment 1 above, this disclosure can also improve the secure transmission of data through a security verification mechanism. The second terminal can verify the legitimacy of the first terminal under the security verification mechanism, or it can proactively request the first terminal to perform security verification on itself.

[0204] One type of security verification mechanism may be as follows:

[0205] When a connection request is sent to the first terminal, a preset security verification mechanism is triggered, as described in Embodiment 1 above. This security verification mechanism is used to instruct that a second security identification code be carried in each piece of data sent to the first terminal for the first terminal to verify.

[0206] Next, a screen mirroring connection can be established with the first terminal based on a security verification mechanism;

[0207] Accordingly, when sending screen projection data to the first terminal, a security verification mechanism can be used to send the screen projection data to the first terminal so that the screen projection data carries a second security identification code.

[0208] When the screen-casting data carries a second security identification code, the first terminal can verify the second security identification code. Only when the verification is successful will the screen-casting data of the second terminal be played, thus preventing the screen-casting data of the second terminal from being played by unauthorized terminals.

[0209] In this embodiment, the second security identification code, as described in the first embodiment above, can also be obtained by encrypting the second target information using a preset encryption algorithm. The second target information includes at least one of the following: the identifier of the second terminal, the current time, and the projection data. The relevant aspects of this implementation can be referred to the security verification mechanism described in the first embodiment above, and will not be repeated here.

[0210] The second security identifier can be carried in the data packet header for the first terminal to extract and verify. In some implementations, the second security identifier can also be dynamically changed in real time by the second terminal to ensure its security. For example, the second security identifier can be a pre-agreed fixed code, such as the identification codes of both parties. It can be in string form, such as multiple security identifiers changing in a certain order, so that the second security identifier sent by the second terminal to the first terminal each time can present a pre-defined change, thereby improving security.

[0211] Another security verification mechanism can be as follows: the second terminal can perform security verification on the data sent by the first terminal to ensure the security of data interaction between the two parties.

[0212] Specifically, the second terminal can verify the first security identification code carried in the target data; after the verification is successful, it stores the target data and / or performs the second operation on the projected data based on the target data; wherein, the first security identification code is generated by the first terminal based on a preset security verification mechanism.

[0213] By carrying a security identification code, the security of the data sent by the second terminal each time can be guaranteed.

[0214] It should be noted that if the verification fails, it indicates that the first terminal is not authorized to send the target data, such as not being authorized to remotely control the second terminal. In this case, the second terminal can prohibit the storage of the target data and the execution of the second operation, and can send a prompt message to the first terminal to inform the user that the target data cannot be executed and that permission needs to be obtained before execution. At this time, the user can operate the first terminal to request control permission from the second terminal. If the request is successful, an authorization code can be sent to the second terminal. After receiving the authorization code, the second terminal can store the target data or execute the second operation.

[0215] In some implementation schemes, such as described in Embodiment 1 above, this disclosure proposes another security verification mechanism for maintaining data security and remote control security. Specifically, the second terminal can send interaction permissions to the first terminal to indicate whether the first terminal can remotely control the second terminal and whether the second terminal can upload its target data to the first terminal according to the operation of the first terminal.

[0216] In practice, the second terminal can send interaction permissions to the first terminal based on the screen projection connection, so that the first terminal can lock the projection area based on the interaction permissions.

[0217] Accordingly, when storing target data and / or performing a second operation corresponding to the first operation on the projected data based on the target data, the sending of target data to the first terminal may be prohibited if the interaction permissions only include control rights, and the execution of the second operation may be prohibited if the interaction permissions only include file interaction permissions.

[0218] The interaction permissions sent from the second terminal to the first terminal may include: operation control rights and file interaction rights. The file interaction rights are used to determine whether the target data on the second terminal is allowed to be shared to the first terminal, and the operation control rights are used to determine whether the second terminal allows the first terminal to control it.

[0219] In this scenario, when the interaction permission sent only includes control rights, the second terminal can perform a second operation in response to the operation data sent by the first terminal, while prohibiting the sending of target data to the first terminal. For example, when the second terminal receives an operation data packet sent by the first terminal for obtaining target data (sent by the first terminal in response to an operation from the projection area to a non-projection area), it will lock the operation data packet and not respond. Since the first terminal will not send operation data packets in response to an operation from the projection area to a non-projection area, this method, if the second terminal also prohibits the sending of target data to the first terminal based on interaction permissions, will avoid the problem of target data interaction caused by the first terminal's accidental triggering, thereby improving data security for both parties.

[0220] Of course, when the interaction permissions only include control rights, the second terminal will also prohibit the upload operation of the target file performed on the second terminal.

[0221] When the interaction permissions sent only include file interaction rights, the second terminal is prohibited from responding to operation data sent by the first terminal to execute the second operation, but is allowed to receive target data sent by the first terminal and send its own target data to the second terminal. Since the first terminal will not respond to operations in the projection area and send operation data, this method, if the second terminal also prohibits responding to operation data based on interaction permissions, avoids the problem of remotely controlling the second terminal due to accidental triggering by the first terminal, thereby improving control security for both parties.

[0222] In some implementation schemes, as described in Embodiment 1 above, the specific services required by the first terminal can be provided on the second terminal, thereby simplifying development on the first terminal and improving development efficiency. Specifically, when the first terminal remotely controls the second terminal through a first operation, the first terminal can directly send target data (i.e., operation data) with a standardized data structure to the second terminal. In this case, the second terminal executes the second operation through its screen projection control service by parsing the data structure of the target data. Thus, there is no need to develop an instruction architecture adapted to the operation control of the second terminal on the first terminal, allowing development to be carried out on the second terminal, thereby reducing development workload and improving efficiency.

[0223] In practice, the second terminal can send service information to the first terminal based on the screen projection request, so that the first terminal can activate at least one service call that matches the service information; wherein, the service information includes at least one screen projection control service provided by the second terminal.

[0224] The table below shows the major service categories and subcategories of the screen mirroring control services provided by the second terminal.

[0225] Table - Service Categories and Service Subcategories

[0226]

[0227]

[0228] The service parameters describe the location where the operation is triggered, as shown in the table below. Figure 11 The diagram illustrates the structure of a request data packet from the first terminal requesting the reporting of services supported by the second terminal, and the structure of a data packet containing service information returned by the second terminal. It is evident that the service information returned by the second terminal includes the data packet structure for each screen projection control service. Therefore, the first terminal can be instructed to encapsulate corresponding operation data for the triggered first operation. The data structure of the operation data encapsulated by the first terminal is shown below. Figure 5 The image shows the target data generated by a mouse click.

[0229] Accordingly, when the second terminal performs the second operation corresponding to the first operation on the screen projection data based on the target data, the target data can be parsed to obtain the service call corresponding to the target data and the operation position where the first operation is located; at the operation position, the screen projection control service corresponding to the target data is called to execute the second operation.

[0230] Parsing the target data can involve extracting the service categories and subcategories, such as... Figure 5 As shown, the service class is MouseClick, the service subclass is SingleClick, and the service parameters [int, Int] are obtained. Then, the screen casting control service corresponding to the service subclass is called, which can also be understood as the control logic, to execute the mouse click operation.

[0231] To fully illustrate the scheme of Embodiment 2, the control method executed by the first terminal is described below through a specific example 2: (Refer to...) Figures 6-8b As shown, the flowcharts illustrate the process of establishing a screen mirroring connection, receiving screen mirroring data, and interacting with target data by the first terminal and the second terminal, respectively. Specifically, Example 1 uses the above-mentioned... Figure 1a Taking the screen mirroring scenario between a mobile phone and a computer as an example, it specifically includes:

[0232] S1: The second terminal responds to the pull-down operation of the system control bar on the system desktop, pops up the system control bar, which has a target button. Then, in response to the click operation of the target button, it starts to send broadcast data packets to the surrounding area. Devices within the same local area network can definitely receive the data packet. After receiving the data packet, it parses it. If the parsing is applicable to the local machine, it will return the data packet.

[0233] S2: The second terminal displays the identifier of the device corresponding to the received return data packet on the interface. Different identifiers correspond to different smart devices.

[0234] S3: In response to the selection operation of the identifier, the second terminal sends a connection request to the first terminal corresponding to the selected identifier. The connection request carries a second security identification code. The second security identification code is obtained by the second terminal encrypting the identifier of the second terminal using a preset encryption algorithm.

[0235] S4: The second terminal receives the reply data packet returned by the first column. The reply data packet carries the first security identification code. The second terminal verifies the first security identification code. After successful verification, the strong direct connection virtual network card is set to the IP address of the first terminal, thereby establishing a screen projection connection. In this case, the second terminal becomes the server device of the first terminal.

[0236] S5: The second terminal receives service request information sent by the first terminal. The service request information carries a first security identification code. After the second terminal verifies the first security identification code, it returns service information to the first terminal, which carries a second security identification code. The data structures of the service request information and the returned service information can be referred to... Figure 11 As shown;

[0237] S6: The second terminal records a screenshot of the launched application page or the desktop data based on the user's selection of the screen projection. The recorded data is sent to the first terminal as screen projection data. The screen projection data carries a second security identification code. After the first terminal verifies the second security identification code, it plays the screen projection data in the screen projection area.

[0238] S7: The second terminal receives the target data sent by the first terminal based on the first operation performed from the non-projection area to the projection area, and saves the target data to a preset address to share the data on the first terminal;

[0239] S8: The second terminal receives the operation data packet sent by the first terminal based on the first operation performed from the projection area to the non-projection area, and sends the operation data packet as target data indicating the local area to the second terminal; wherein, the target data carries a second security identification code;

[0240] S9: The second terminal receives the operation data sent by the first terminal based on the first operation performed within the projection area, and verifies the first security identification code in the operation data. After successful verification, the service category, service subclass and service parameters in the operation data are parsed out.

[0241] S10: Call the screen casting control service (control logic) corresponding to the service subclass, and perform a second operation on the control corresponding to the service parameters (such as the operation position), such as clicking the left mouse button, clicking the right mouse button, or clicking the right button to create a new operation.

[0242] Example 3

[0243] This third embodiment describes the control method proposed in this disclosure from the perspective of the first terminal and the second terminal, referring to... Figure 12 As shown, a flowchart of the control method described in Embodiment 3 is illustrated. Figure 12 As shown, the specific steps may include:

[0244] Step S401: The first terminal displays the projection data of the second terminal in the projection area; wherein, the projection data is sent by the second terminal to the first terminal based on a pre-established projection connection;

[0245] Step S402: The second terminal responds to the first operation that occurs part or all of the time in the projection area on the first terminal, shares the target data corresponding to the first operation with the first terminal, and / or performs a second operation corresponding to the first operation on the projection data.

[0246] In this embodiment, the process of establishing the screen mirroring connection and generating the screen mirroring data are described in Embodiment 2 above, and will not be repeated here. The first terminal can display the screen mirroring data sent by the second terminal in the screen mirroring area.

[0247] In this embodiment, the target data includes target data and operation data. The target data is the data corresponding to the first operation that occurs in part of the projection area. Depending on the direction of the first operation, it can originate from either the first terminal or the second terminal. Specifically, in response to the first operation from the non-projection area to the projection area, the first terminal sends the corresponding target data to the second terminal, or in response to the first operation from the projection area to the non-projection area, the first terminal receives the target data sent by the second terminal.

[0248] The operation data refers to the data corresponding to the first operation that occurs entirely in the projection area. This data is generated by the first terminal and is used to describe the first operation. The second terminal can respond to the operation data and execute the second operation. The process of executing the second operation can be referred to the description in Embodiment 2 above, and will not be repeated here.

[0249] In this way, the second terminal can remotely respond to the first operation that occurs in the projection area on the first terminal, share the target data corresponding to the first operation with the first terminal, and remotely respond to the first operation that occurs in the projection area on the first terminal, and perform the second operation corresponding to the first operation on the projection data.

[0250] In one specific implementation, as described in Embodiments 1 and 2 above, the second terminal responds to the operation of dragging target data into the projection area on the first terminal, receives the target data and stores the target data in a preset address; and the second terminal can respond to the trigger operation performed by the first terminal in the projection area, and perform a second operation corresponding to the trigger operation on the target content corresponding to the projection data; the target content is the content corresponding to the location of the trigger operation.

[0251] In another specific implementation, the first terminal can respond to the upload operation performed on the second terminal and receive the target file corresponding to the file upload operation; the upload operation includes any of the following: a control selection operation in the menu bar triggered by right-clicking, or an operation of dragging the target file into a preset icon.

[0252] The process of uploading the target file on the second terminal can be referred to the process described in Embodiment 2 above, and will not be repeated here.

[0253] Using the technical solution of this application embodiment, the first terminal can share its target data with the second terminal based on a first operation that partially occurs in the projection area, and the second terminal can share its target data with the first terminal based on a first operation that partially occurs in the projection area; furthermore, the first terminal can remotely control the second terminal based on the first operation that occurs entirely in the projection area, and the second terminal can perform a second operation in the remote control; thus, during the process of sharing projection data, the first terminal and the second terminal can further share target data based on the user's operation on the projection area, thereby enabling the user to perform remote control and file sharing between the two terminals through simple operations.

[0254] In one embodiment of this third embodiment, as described in the first embodiment above, the first terminal may include multiple projection sub-areas. Thus, in this third embodiment, multiple second terminals may be included, and different second terminals may correspond to different projection sub-areas. When a first operation occurs on the first terminal in a certain projection sub-area, the second terminal corresponding to that projection sub-area can respond to the first operation and perform interactive control with the first terminal.

[0255] Specifically, in one interactive control, there is at least one second terminal responding to a first operation on the first terminal that occurs part or all of the time in the target projection sub-area, storing target data, and / or performing a second operation on the projection data;

[0256] The target projection sub-region can be any one of multiple projection sub-regions. The specific process of this implementation method can be referred to the description in Embodiment 1, and will not be repeated here.

[0257] In a further example, as described in Embodiment 1 above, the user of the first terminal can share data among multiple second terminals through a simpler operation. A sharing area can be set within the projection area. Thus, the projection area not only includes multiple projection sub-areas but also a shared area for these sub-areas to share. The first terminal can use this shared area to send the shared data to each of the multiple second terminals. Of course, the first terminal can also use this shared area to simultaneously remotely control multiple second terminals, enabling them to perform the same operation.

[0258] In specific implementation, the first terminal may respond to the third operation that occurs in whole or in part in the shared area by sending the shared data corresponding to the third operation to multiple second terminals respectively. Accordingly, the multiple second terminals may respond to the third operation that occurs in part or in whole in the shared area on the first terminal by receiving the shared data corresponding to the third operation and / or executing the fourth operation corresponding to the third operation.

[0259] The fourth operation can be the third operation that is entirely within the shared area, such as a file creation operation, as described in Example 1.

[0260] In this embodiment, the operations performed by the first terminal can be referred to the description in Embodiment 1, and the operations performed by the second terminal can be referred to the operations in Embodiment 2.

[0261] It should be noted that the above Figures 6-8b In this context, the client device refers to the first terminal, and the server device refers to the second terminal.

[0262] Based on the same inventive concept, this disclosure provides a control system, including the first terminal and the second terminal described above. The first terminal is used to execute the control method described in Embodiment 1, and the second terminal is used to execute the control method described in Embodiment 2.

[0263] Based on the same inventive concept, this disclosure provides a control device, referring to... Figure 13 The diagram shows the structure of the control device, which is applied to the first terminal and may specifically include the following modules:

[0264] The data receiving module is used to receive screen projection data sent by the second terminal based on a pre-established screen projection connection with the second terminal;

[0265] The display module is used to generate a projection area corresponding to the second terminal and to display the projection data in the projection area.

[0266] An interaction module is used to respond to a first operation that occurs wholly or partially in the projection area, to interact with the second terminal with target data corresponding to the first operation, so as to share the target data with the second terminal, and / or to allow the second terminal to perform a second operation on the projection data corresponding to the first operation based on the target data.

[0267] Optionally, the interaction module includes:

[0268] The first interaction unit is used to share the target data targeted by the first operation with the second terminal in response to a first operation in which part of the operation occurs in the projection area and the other part of the operation occurs in the non-projection area, wherein the non-projection area is an area located outside the projection area.

[0269] The second interaction unit is configured to respond to a first operation occurring entirely within the projection area by sending target data describing the first operation to the second terminal, so that the second terminal can perform the second operation on the projection data.

[0270] Optionally, the first interaction unit is specifically used for:

[0271] When the first operation represents dragging the target data from the non-projection area into the projection area, the target data is sent to the second terminal so that the second terminal can store the target data;

[0272] When the first operation is to drag the target data of the projection area into the non-projection area, the target data sent by the second terminal is received and the target data is stored.

[0273] Optionally, the steps for establishing the screen mirroring connection include:

[0274] In response to a connection request sent by the second terminal, a preset security verification mechanism is initiated. The security verification mechanism is used to instruct that a first security identification code be carried each time data is sent to the second terminal for verification by the second terminal.

[0275] Based on the aforementioned security verification mechanism, the screen projection connection is established;

[0276] The interaction module is specifically used to interact with the second terminal on the basis of the security verification mechanism to obtain the target data.

[0277] Optionally, the first security identification code is obtained by encrypting the first target information using a preset encryption algorithm; wherein, the first target information includes at least one of the identifier of the first terminal, the current time, and the screen projection data.

[0278] Optionally, the projection data carries a second security identification code, which is generated by the second terminal based on a preset security verification mechanism; the display module includes...

[0279] A verification unit is used to verify the second security identification code;

[0280] The display unit is used to display the projection data in the projection area after the verification is successful.

[0281] Optionally, the device further includes:

[0282] The permission determination module is used to determine the interaction permissions corresponding to the second terminal;

[0283] The first locking module is used to lock the non-projection area when the interaction permission only includes operation control rights, so as to prohibit the response to the first operation from the projection area to the non-projection area, or to prohibit the response to the first operation from the non-projection area to the projection area.

[0284] The second locking module is used to lock the projection area when the interaction permissions only include file interaction rights, so as to prevent the response to the first operation that occurs entirely in the projection area.

[0285] Optionally, the first terminal is configured with multiple service calls, and the device further includes:

[0286] The information acquisition module is used to acquire service information sent by the second terminal based on the screen projection connection. The service information includes at least one screen projection control service provided by the second terminal; wherein different screen projection control services are used to perform different second operations.

[0287] An activation module is used to activate at least one service call that matches the at least one screen projection control service, based on the service information.

[0288] The interaction module includes:

[0289] The triggering unit is invoked to trigger a service call corresponding to the operation type of the first operation when all of the first operations occur in the projection area.

[0290] The data encapsulation unit is used to encapsulate the operation data of the first operation based on the data structure indicated by the triggered service call to obtain the target data, so that the second terminal can call the corresponding screen casting control service to execute the second operation corresponding to the first operation.

[0291] Optionally, the device further includes an upgrade module for performing the following steps:

[0292] Identify the target service call to be processed, wherein the target service call is a service call that does not match the screen projection control service;

[0293] Download and / or upgrade the target service call, and activate the target service call.

[0294] Optionally, the system includes multiple second terminals, and the projection area includes projection sub-areas corresponding to the second terminals respectively. The display module is specifically used to display projection data sent by the multiple second terminals in the multiple projection sub-areas respectively.

[0295] The interaction module is specifically used to respond to a first operation that occurs, in whole or in part, in the target projection sub-region among the multiple projection sub-regions, and to send the target data to the second terminal corresponding to the target projection sub-region.

[0296] Optionally, the projection area further includes a shared area shared by multiple projection areas; the device further includes: a sharing module, configured to, in response to a third operation that occurs entirely or partially in the shared area, send shared data corresponding to the third operation to multiple second terminals respectively, so that multiple second terminals share the shared data, and / or all perform a fourth operation corresponding to the third operation.

[0297] Optionally, the device further includes:

[0298] The file receiving module is used to receive the target file uploaded by the second terminal for the projection area, wherein the target file is sent by the second terminal after performing an upload operation on the target file.

[0299] Based on the same inventive concept, this disclosure provides a control device, referring to... Figure 14 The diagram shows the structure of the control device, which is applied to a second terminal and may specifically include the following modules:

[0300] The data sending module is used to send data to the first terminal based on a pre-established screen projection connection with the first terminal, so that the first terminal can display the screen projection data in the screen projection area;

[0301] The data interaction module is used to interact with the first terminal to exchange target data corresponding to the first operation; wherein, the first operation occurs partly or entirely in the projection area on the first terminal;

[0302] An execution module is used to store the target data and / or perform a second operation on the projected data based on the target data, corresponding to the first operation.

[0303] Optionally, the device further includes:

[0304] An upload module is configured to send the target file to the first terminal in response to an upload operation performed on the target file on the second terminal; wherein the upload operation includes any of the following:

[0305] The operation includes selecting controls from the right-click menu and dragging the target file into the preset icon corresponding to the projection area.

[0306] Optionally, the screen mirroring connection is established based on a target operation triggered on the system desktop of the second terminal, wherein the target operation includes triggering a target button on the system control bar of the second terminal.

[0307] Optionally, the steps for establishing the screen mirroring connection include:

[0308] A connection request is sent to the first terminal, and a preset security verification mechanism is triggered. The security verification mechanism is used to indicate that a second security identification code is carried in each data sent to the first terminal for the first terminal to verify.

[0309] Based on the security verification mechanism, a screen projection connection is established with the first terminal;

[0310] Sending screen projection data to the first terminal includes:

[0311] Based on the security verification mechanism, screen projection data is sent to the first terminal so that the screen projection data carries the second security identification code.

[0312] Optionally, the second security identification code is obtained by encrypting the second target information based on a preset encryption algorithm; wherein the second target information includes at least one of the identifier of the second terminal, the current time, and the screen projection data.

[0313] Optionally, the execution module includes:

[0314] The verification unit is used to verify the first security identification code carried in the target data; wherein the first security identification code is generated by the first terminal based on a preset security verification mechanism.

[0315] An execution unit is used to store the target data after verification, and / or perform the second operation on the projection data based on the target data.

[0316] Optionally, the device further includes:

[0317] The information sending module is used to send service information to the first terminal based on the screen projection request, so that the first terminal can activate at least one service call matching the service information; wherein, the service information includes at least one screen projection control service provided by the second terminal;

[0318] The execution module includes:

[0319] The parsing unit is used to parse the target data to obtain the service call corresponding to the target data and the operation location where the first operation is located;

[0320] The calling unit is used to call the screen projection control service corresponding to the target data at the operation location to execute the second operation.

[0321] Optionally, the device further includes:

[0322] The permission sending module is used to send interaction permissions to the first terminal based on the screen projection connection, so that the first terminal can lock the screen projection area based on the interaction permissions;

[0323] The execution module is used for:

[0324] When the interaction permission only includes control rights, sending the target data to the first terminal is prohibited;

[0325] If the interaction permissions only include file interaction permissions, the second operation is prohibited.

[0326] Optionally, the data transmission module includes:

[0327] The content determination unit is used to determine the content to be projected based on the selection operation of the content to be projected, wherein the selection operation includes the selection operation of the application and the selection operation of the desktop;

[0328] The recording unit is used to record the content to be projected in response to the display of the content to be projected on the first terminal, and to obtain the projection data;

[0329] The applications include at least one of office software applications, media applications, and social applications.

[0330] This disclosure also provides an electronic device whose stored computer program causes a processor to execute the control method as described in this disclosure.

[0331] This disclosure also provides a computer-readable storage medium storing a computer program that causes a processor to perform the control method as described in this disclosure.

[0332] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0333] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0334] This disclosure describes embodiments of methods, terminal devices (systems), and computer program products according to embodiments of this disclosure with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0335] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0336] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0337] While preferred embodiments of the present disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0338] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0339] The control method, control device, equipment, and storage medium provided in this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A control method, characterized in that, Applied to a first terminal, the method includes: Based on the pre-established screen mirroring connection with the second terminal, receive screen mirroring data sent by the second terminal; Generate a projection area corresponding to the second terminal, and display the projection data in the projection area; In response to a first operation that occurs wholly or partially in the projection area, the system interacts with the second terminal to exchange target data corresponding to the first operation, so as to share the target data with the second terminal and / or allow the second terminal to perform a second operation corresponding to the first operation on the projection data based on the target data; The establishment of the screen projection connection includes: In response to a connection request sent by the second terminal, a preset security verification mechanism is initiated. The security verification mechanism is used to instruct that a first security identification code be carried each time data is sent to the second terminal for verification by the second terminal. Based on the aforementioned security verification mechanism, the screen projection connection is established; The interaction with the second terminal of the target data corresponding to the first operation includes: Based on the security verification mechanism, the target data is interacted with the second terminal; The first security identification code is obtained by encrypting the first target information using a preset encryption algorithm; the first target information includes at least one of the following: the identifier of the first terminal, the current time, and the screen projection data. The screen projection data carries a second security identification code, which is generated by the second terminal based on a preset security verification mechanism; displaying the screen projection data sent by the second terminal in the projection area includes: Verify the second security identifier; After successful verification, the projection data is displayed in the projection area.

2. The method according to claim 1, characterized in that, The response to a first operation that occurs wholly or partially in the projection area, and the interaction with the second terminal of target data corresponding to the first operation, includes: In response to a first operation in which part of the operation occurs in the projection area and the other part of the operation occurs in a non-projection area, the second terminal shares the target data targeted by the first operation, wherein the non-projection area is an area located outside the projection area; In response to a first operation occurring entirely within the projection area, target data describing the first operation is sent to the second terminal, so that the second terminal can perform the second operation on the projection data.

3. The method according to claim 2, characterized in that, The first operation, which responds to a partial operation occurring in the projection area and a partial operation occurring in a non-projection area, shares the target data targeted by the first operation with the second terminal, including: When the first operation represents dragging the target data from the non-projection area into the projection area, the target data is sent to the second terminal so that the second terminal can store the target data; When the first operation is to drag the target data of the projection area into the non-projection area, the target data sent by the second terminal is received and the target data is stored.

4. The method according to any one of claims 1-3, characterized in that, Before interacting with the second terminal with the target data corresponding to the first operation, which occurs wholly or partially in the projection area, the method further includes: Determine the interaction permissions corresponding to the second terminal; When the interaction permissions only include operation control rights, the non-projection area is locked to prevent the response to the first operation from the projection area to the non-projection area, or to prevent the response to the first operation from the non-projection area to the projection area. If the interaction permissions only include file interaction rights, the projection area is locked to prevent responses to the first operation that occurs entirely within the projection area.

5. The method according to any one of claims 1-3, characterized in that, The first terminal is configured with multiple service calls. Before generating the projection area corresponding to the second terminal, the method further includes: Based on the screen mirroring connection, service information sent by the second terminal is obtained. This service information includes at least one screen mirroring control service provided by the second terminal; wherein different screen mirroring control services are used to perform different second operations. Based on the service information, activate at least one service call that matches each of the at least one screen mirroring control services; Interacting with the second terminal to obtain the target data corresponding to the first operation includes: If all of the first operations occur in the projection area, a service call corresponding to the operation type of the first operation is triggered. Based on the data structure indicated by the triggered service call, the operation data describing the first operation is encapsulated to obtain the target data, which is then used by the second terminal to call the corresponding screen casting control service to execute the second operation corresponding to the first operation.

6. The method according to claim 5, characterized in that, After activating at least one service call that matches the at least one screen casting control service based on the service information, the method further includes: Identify the target service call to be processed, wherein the target service call is a service call that does not match the screen projection control service; Download and / or upgrade the target service call, and activate the target service call.

7. The method according to claim 1, characterized in that, It includes multiple second terminals, and the projection area includes projection sub-areas corresponding to each of the second terminals. The step of displaying the projection data sent by the second terminal in the projection area includes: The projection data sent to each of the multiple second terminals is displayed in multiple projection sub-regions respectively; The response to a first operation that occurs wholly or partially in the projection area, sending target data corresponding to the first operation to the second terminal, includes: In response to a first operation that occurs, in whole or in part, in a target projection sub-region among the multiple projection sub-regions, the target data is sent to a second terminal corresponding to the target projection sub-region.

8. The method according to claim 7, characterized in that, The projection area also includes a shared area shared by multiple projection areas; after the projection data sent by the second terminal is displayed in the projection area, the method further includes: In response to a third operation that occurs wholly or partially in the shared area, shared data corresponding to the third operation is sent to a plurality of second terminals respectively, so that the plurality of second terminals share the shared data, and / or all of them execute a fourth operation corresponding to the third operation.

9. The method according to claim 1, characterized in that, After displaying the projection data sent by the second terminal in the projection area, the method further includes: The system receives a target file uploaded by the second terminal for the projection area, wherein the target file is sent by the second terminal after performing an upload operation on the target file.

10. A control method, characterized in that, Applied to a second terminal, the method includes: Based on the pre-established screen projection connection with the first terminal, data is sent to the first terminal so that the first terminal can display the screen projection data in the screen projection area; The target data corresponding to the first operation is interacted with the first terminal; wherein, the first operation occurs partly or entirely in the projection area on the first terminal; Store the target data, and / or perform a second operation corresponding to the first operation on the projected data based on the target data; The establishment of the screen projection connection includes: A connection request is sent to the first terminal, and a preset security verification mechanism is triggered. The security verification mechanism is used to indicate that a second security identification code is carried in each data sent to the first terminal for the first terminal to verify. Based on the security verification mechanism, a screen projection connection is established with the first terminal; Sending screen projection data to the first terminal includes: Based on the security verification mechanism, screen projection data is sent to the first terminal so that the screen projection data carries the second security identification code. The step of storing the target data and / or performing a second operation corresponding to the first operation on the projected data based on the target data includes: The first security identification code carried in the target data is verified; wherein the first security identification code is generated by the first terminal based on a preset security verification mechanism. After successful verification, the target data is stored, and / or the second operation is performed on the projected data based on the target data; The second security identification code is obtained by encrypting the second target information based on a preset encryption algorithm; the second target information includes at least one of the following: the identifier of the second terminal, the current time, and the screen projection data.

11. The method according to claim 10, characterized in that, After sending the screen projection data to the first terminal, the method further includes: In response to an upload operation performed on a target file on the second terminal, the target file is sent to the first terminal; wherein the upload operation includes any of the following: The menu bar includes the control selection operation and the operation of dragging the target file into the preset icon corresponding to the projection area.

12. The method according to claim 10, characterized in that, The screen mirroring connection is established based on a target operation triggered on the system desktop of the second terminal, wherein the target operation includes triggering a target button on the system control bar of the second terminal.

13. The method according to claim 10, characterized in that, The method further includes: Based on the screen mirroring request, service information is sent to the first terminal so that the first terminal can activate at least one service call matching the service information; wherein, the service information includes at least one screen mirroring control service provided by the second terminal; The second operation, corresponding to the first operation, is performed on the projected data based on the target data, including: The target data is parsed to obtain the service call corresponding to the target data and the operation location of the first operation; At the specified operation location, the screen projection control service corresponding to the target data is invoked to execute the second operation.

14. The method according to claim 10, characterized in that, Before sending data to the first terminal, the method further includes: Based on the screen projection connection, an interaction permission is sent to the first terminal, so that the first terminal can lock the screen projection area based on the interaction permission; Storing the target data, and / or performing a second operation corresponding to the first operation on the projected data based on the target data, including: When the interaction permission only includes control rights, sending the target data to the first terminal is prohibited; If the interaction permissions only include file interaction permissions, the second operation is prohibited.

15. The method according to claim 10, characterized in that, Sending screen projection data to the first terminal, including: Based on the selection operation of the content to be cast, the content to be cast is determined, wherein the selection operation includes the selection operation of the application and the selection operation of the desktop; In response to the display of the content to be projected on the first terminal, the content to be projected is recorded to obtain the projection data; The applications include at least one of office software applications, media applications, and social applications.

16. A control system, characterized in that, It includes a first terminal and a second terminal, wherein the first terminal is configured to execute the control method according to any one of claims 1-9, and the second terminal is configured to execute the control method according to any one of claims 10-15.

17. A control method, characterized in that, Applied to the control system of claim 16, the method comprises: The first terminal displays the projection data of the second terminal in the projection area; wherein the projection data is sent by the second terminal to the first terminal based on a pre-established projection connection; The second terminal responds to a first operation that occurs partly or entirely on the projection area of ​​the first terminal, shares the target data corresponding to the first operation with the first terminal, and / or performs a second operation corresponding to the first operation on the projection data.

18. The method according to claim 17, characterized in that, The projection area includes multiple second terminals, and the projection area includes projection sub-areas corresponding to the multiple second terminals respectively; wherein: At least one of the second terminals responds to a first operation that occurs partly or entirely on the first terminal in the target projection sub-area, shares the target data with the first terminal, and / or performs the second operation on the projection data; The target projection sub-region can be any one of multiple projection sub-regions.

19. The method according to claim 18, characterized in that, The projection area also includes a shared area for multiple projection sub-areas to share; the method further includes: Multiple second terminals respond to a third operation that occurs partially or entirely on the first terminal in the shared area, all of which receive shared data corresponding to the third operation, and / or all execute a fourth operation corresponding to the third operation.

20. An electronic device, characterized in that, include: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the control method as claimed in any one of claims 1-9 or 10-15.

Images