Method and device for reverse control of screen projection

Abstract

The embodiments of the present application disclose a method and device for reverse control of screen projection, which relates to the field of electronic devices and solves the problem of high latency in receiving user operation instructions by the screen projection electronic device during reverse control of screen projection. The specific solution is as follows: a first electronic device sends a projection screen to a second electronic device, and the second electronic device is connected to a peripheral device; the first electronic device establishes a connection with the peripheral device; the first electronic device receives a first instruction sent by the peripheral device, and the first instruction is generated by the peripheral device according to the user operation; the first electronic device performs a corresponding operation according to the first instruction; and the first electronic device sends the projection screen after performing the corresponding operation to the second electronic device.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of electronic devices, and in particular, to a method and device for reverse control of screen projection. Background Art

[0002] When wirelessly mirroring a phone to another electronic device (such as a computer or smart screen), the screen and audio from the phone are typically transmitted to the other device via a direct wireless fidelity (Wi-Fi) connection or a Wi-Fi local area network connection. The data transmitted from the phone to the device during screen mirroring typically includes video and audio streams.

[0003] In order to facilitate users to reversely control their mobile phones through the electronic device being projected during screen projection, a signaling channel, such as a user input backchannel (UIBC), is set between the mobile phone and the electronic device being projected. The instructions that allow users to operate on the electronic device being projected can be transmitted to the mobile phone through the signaling channel so that the mobile phone can respond accordingly. For example, after the user uses a mobile phone to project the screen to a computer, when the user uses the computer's mouse to operate the controls in the screen projected from the mobile phone, the computer can transmit the instructions of the user's operation through the mouse to the mobile phone through the signaling channel between the mobile phone and the computer, so that the mobile phone can respond accordingly according to the received instructions.

[0004] However, after the electronic device whose screen is projected receives the user's operation, it needs to process the user's operation first (for example, parsing, packaging, etc.), and then transmit the user's operation instruction to the mobile phone through the signaling channel, so that the mobile phone can respond accordingly. Since the electronic device whose screen is projected needs to process the user's operation, the delay for the mobile phone to receive the user's operation instruction is relatively high. Summary of the Invention

[0005] The embodiments of the present application provide a method and device for reverse control of screen projection, which solves the problem of high delay in the screen projection electronic device receiving user operation instructions during reverse control of screen projection.

[0006] In order to achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In the first aspect, an embodiment of the present application provides a method for reverse control of screen projection, which can be applied to a first electronic device in a screen projection system, and the screen projection system may also include a second electronic device. The method includes: the first electronic device sends a projection screen to the second electronic device, the second electronic device is connected to a peripheral device, and the peripheral device is used to operate and control the screen displayed by the second electronic device; the first electronic device establishes a connection with the peripheral device; the first electronic device receives a first instruction sent by the peripheral device, and the first instruction is generated by the peripheral device according to the user operation; the first electronic device performs a corresponding operation according to the first instruction; the first electronic device sends the projection screen after performing the corresponding operation to the second electronic device.

[0008] By adopting the above technical solution, when the first electronic device is projecting the screen to the second electronic device, when the user reversely controls the first electronic device on the second electronic device, the instructions corresponding to the operations performed by the user through the peripheral device of the second electronic device can be directly sent by the peripheral device of the second electronic device to the first electronic device, so that the first electronic device can make a corresponding response according to the user's operation. This avoids the second electronic device first receiving the instructions sent by its peripheral device and then processing the instructions before transmitting them to the first electronic device, simplifies the path of control instruction transmission when the user reversely controls the first electronic device through the peripheral device of the second electronic device, and thus improves the reverse control delay in the projection state, improves the response speed of the projection electronic device (i.e., the first electronic device) to the user's operation, and improves the follow-up of the projection screen displayed by the projection electronic device (i.e., the second electronic device) relative to the user's operation.

[0009] In one possible implementation, before the first electronic device establishes a connection with the peripheral device, the method further includes: the first electronic device receives connection information of the peripheral device sent by the second electronic device, where the connection information is information used by the second electronic device to connect with the peripheral device; the first electronic device establishes a connection with the peripheral device, including: the first electronic device establishes a connection with the peripheral device according to the connection information.

[0010] In this way, the first electronic device can directly connect to the peripheral device based on the connection information when the second electronic device is connected to the peripheral device, thereby improving the convenience of connecting the first electronic device to the peripheral device and simplifying the connection process when the first electronic device establishes a connection with the peripheral device.

[0011] In another possible implementation, the first electronic device receives connection information of the peripheral device sent by the second electronic device, including: the first electronic device receives the connection information when negotiating protocol parameters in real time with the second electronic device; or, the first electronic device receives the connection information when establishing a trusted connection with the second electronic device and / or logging in with the same account.

[0012] In this way, the first electronic device and the second electronic device can synchronize connection information when they have a trusted connection relationship or when screen projection is to be performed, thereby improving the security of the first electronic device when receiving connection information sent by the second electronic device.

[0013] In another possible implementation, the connection information includes the physical address of the peripheral device and a token used for connection verification.

[0014] This allows the first electronic device to quickly connect to the peripheral device based on its physical address, without having to search for it first. Furthermore, the token allows for convenient and efficient authentication between the first electronic device and the peripheral device, ensuring a secure connection between them.

[0015] In another possible implementation, the second electronic device displays the projection screen of the first electronic device in full screen.

[0016] In another possible implementation, the second electronic device displays the projected screen of the first electronic device in non-full screen; the method also includes: the first electronic device receives the position information of the projected screen sent by the second electronic device in the display screen of the second electronic device; the first electronic device performs a corresponding operation according to the first instruction, including: when the first electronic device determines that the user operation corresponding to the first instruction is within the projected screen of the first electronic device based on the position information of the projected screen in the display screen of the second electronic device, the first electronic device performs a corresponding operation according to the first instruction.

[0017] In this way, when the second electronic device does not display the projection screen of the first electronic device in full screen, it can first determine whether the user operation is within the projection screen of the first electronic device based on the instruction received from the external device, and then respond to the instruction to perform related operations, thereby avoiding the first electronic device from invalidating operations that are not performed on the first electronic device. Correspondingly, when the second electronic device does not display the projection screen of the first electronic device in full screen, the second electronic device can also receive the instruction sent by the external device, and then determine whether the user operation is outside the projection screen of the first electronic device based on the instruction. If so, the second electronic device responds to the instruction to perform related operations.

[0018] In another possible implementation, the second electronic device displays the projection screen of the first electronic device in non-full screen; the method also includes: the first electronic device receives the position information of the projection screen sent by the second electronic device in the display screen of the second electronic device; the first electronic device determines whether the user operation corresponding to the first instruction is outside the projection screen of the first electronic device based on the position information of the projection screen in the display screen of the second electronic device; if the first electronic device determines that the user operation corresponding to the first instruction is outside the projection screen of the first electronic device, the first electronic device does not execute the first instruction.

[0019] In this way, when the second electronic device does not display the projection screen of the first electronic device in full screen, it first determines the user operation within the projection screen of the first electronic device based on the instructions received from the external device, and then responds to the instructions to perform related operations, thereby avoiding the first electronic device from invalidating operations that are not performed on the first electronic device.

[0020] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device, and the first electronic device is wirelessly connected to the peripheral device.

[0021] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi, and the first electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi.

[0022] In another possible implementation, the peripheral device includes any one of the following: a mouse, a keyboard, a stylus, and a handle; the first instruction includes any one of the following: movement information of the mouse or stylus, typing information of the keyboard or handle; the movement information includes the distance and direction of movement.

[0023] In a second aspect, an embodiment of the present application provides a device for reverse control of screen projection, which can be applied to a first electronic device in a screen projection system to implement the method in the first aspect above. The functions of the device can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as a screen projection module, a connection module, and a processing module.

[0024] Among them, the screen projection module can be used to send the projected screen to the second electronic device, the second electronic device is connected to the peripheral device, and the peripheral device is used to operate and control the screen displayed on the second electronic device; the connection module can be used to establish a connection with the peripheral device; receive the first instruction sent by the peripheral device, the first instruction is generated by the peripheral device according to the user operation; the processing module can be used to perform the corresponding operation according to the first instruction; the screen projection module is also used to send the projected screen after performing the corresponding operation to the second electronic device.

[0025] In one possible implementation, the connection module is further used to receive connection information of the peripheral device sent by the second electronic device, where the connection information is information used to connect the second electronic device to the peripheral device; the connection module is specifically used to establish a connection with the peripheral device based on the connection information.

[0026] In another possible implementation, the connection module is specifically used to receive connection information when the first electronic device and the second electronic device negotiate protocol parameters in real time; or, receive connection information when the first electronic device and the second electronic device establish a trusted connection and / or log in with the same account.

[0027] In another possible implementation, the connection information includes the physical address of the peripheral device and a token used for connection verification.

[0028] In another possible implementation, the second electronic device displays the projection screen of the first electronic device in full screen.

[0029] In another possible implementation, the second electronic device displays the projected screen of the first electronic device in non-full screen; the connection module is also used to receive the position information of the projected screen sent by the second electronic device in the display screen of the second electronic device; the processing module is specifically used to determine that the user operation corresponding to the first instruction is within the projected screen of the first electronic device based on the position information of the projected screen in the display screen of the second electronic device, and perform the corresponding operation according to the first instruction.

[0030] In another possible implementation, the second electronic device displays the projected screen of the first electronic device in non-full screen; the connection module is also used to receive the position information of the projected screen sent by the second electronic device in the display screen of the second electronic device; the processing module is also used to determine whether the user operation corresponding to the first instruction is outside the projected screen of the first electronic device based on the position information of the projected screen in the display screen of the second electronic device; if it is determined that the user operation corresponding to the first instruction is outside the projected screen of the first electronic device, the first instruction is not executed.

[0031] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device, and the first electronic device is wirelessly connected to the peripheral device.

[0032] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi, and the first electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi.

[0033] In another possible implementation, the peripheral device includes any one of the following: a mouse, a keyboard, a stylus, and a handle; the first instruction includes any one of the following: movement information of the mouse or stylus, typing information of the keyboard or handle; the movement information includes the distance and direction of movement.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, and a memory for storing instructions executable by the processor. When the processor is configured to execute the instructions, the electronic device implements the method for reverse screen projection control as described in any one of the first aspect or possible implementations of the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by an electronic device, the electronic device implements the method for reverse screen projection control as described in the first aspect or any possible implementation of the first aspect.

[0036] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer-readable code, which, when executed in an electronic device, enables the electronic device to implement a method for reverse control of screen projection as described in any one of the first aspect or possible implementations of the first aspect.

[0037] It should be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0038] In a sixth aspect, an embodiment of the present application provides a method for reverse control of screen projection, which can be applied to a second electronic device in a screen projection system, and the screen projection system may also include a first electronic device. The method includes: the second electronic device receives and displays the projection screen of the first electronic device, the second electronic device is connected to a peripheral device, and the peripheral device is used to operate and control the screen displayed by the second electronic device; the second electronic device displays the projection screen of the first electronic device in a non-full screen manner; the second electronic device receives a first instruction from the peripheral device, and the first instruction is generated by the peripheral device according to the user operation; the second electronic device determines that the user operation corresponding to the first instruction is outside the projection screen of the first electronic device, and the second electronic device performs the corresponding operation according to the first instruction.

[0039] With the above technical solution, when a first electronic device projects a screen onto a second electronic device, when a user reversely controls the first electronic device on the second electronic device, the instructions corresponding to the operations performed by the user through the peripheral device of the second electronic device can be directly sent by the peripheral device of the second electronic device to the first electronic device, so that the first electronic device can respond accordingly according to the user's operation. This avoids the second electronic device first receiving the instructions sent by its peripheral device and then processing the instructions before transmitting them to the first electronic device, simplifying the path for the control instruction transmission when the user reversely controls the first electronic device through the peripheral device of the second electronic device, thereby improving the reverse control delay in the projection state, improving the response speed of the projection electronic device (i.e., the first electronic device) to the user's operation, and improving the tracking of the projection screen displayed by the projection electronic device (i.e., the second electronic device) relative to the user's operation. In addition, when the second electronic device is not displaying the projection screen of the first electronic device in full screen, the second electronic device can receive the instructions sent by the peripheral device and, when it is determined based on the instructions received from the peripheral device that the user operation is outside the projection screen of the first electronic device (i.e., when the user operates within the interface of the second electronic device itself), execute the operation corresponding to the received instructions. This avoids the second electronic device from being unresponsive when the user operates on its own interface.

[0040] In a possible implementation, the method further includes: the second electronic device sending connection information of the peripheral device to the first electronic device, where the connection information is information used by the second electronic device to connect to the peripheral device.

[0041] In this way, the first electronic device can directly connect to the peripheral device based on the connection information when the second electronic device is connected to the peripheral device, thereby improving the convenience of connecting the first electronic device to the peripheral device and simplifying the connection process when the first electronic device establishes a connection with the peripheral device.

[0042] In another possible implementation, the second electronic device sends connection information of the peripheral device to the first electronic device, including: the second electronic device sends connection information to the first electronic device when negotiating protocol parameters in real time with the first electronic device; or, the second electronic device synchronizes connection information to the first electronic device when establishing a trusted connection with the first electronic device and / or logging in with the same account.

[0043] In this way, the first electronic device and the second electronic device can synchronize connection information when they have a trusted connection relationship or when screen projection is to be performed, thereby improving the security of the first electronic device when receiving connection information sent by the second electronic device.

[0044] In another possible implementation, the connection information includes the physical address of the peripheral device and a token used for connection verification.

[0045] This allows the first electronic device to quickly connect to the peripheral device based on its physical address, without having to search for it first. Furthermore, the token allows for convenient and efficient authentication between the first electronic device and the peripheral device, ensuring a secure connection between them.

[0046] In another possible implementation, the method further includes: the second electronic device sending position information of the projected image on the display screen of the second electronic device to the first electronic device.

[0047] In this way, the first electronic device can determine whether the user operation is within the projection screen of the first electronic device based on the position information of the projection screen in the display screen of the second electronic device.

[0048] In another possible implementation, the method further includes: when the user operation corresponding to the first instruction is performed within the projection screen of the first electronic device, the second electronic device receives and displays the projection screen sent by the first electronic device after performing the corresponding operation.

[0049] In this way, when the first electronic device performs a corresponding operation according to the first instruction received from the external device and the screen to be projected changes, the first electronic device can project the changed screen to the second electronic device. Optionally, the first electronic device can send the projected screen to the second electronic device so that the second electronic device can display the screen projected by the first electronic device in real time.

[0050] In another possible implementation, the method further includes not executing the first instruction when the user operation corresponding to the first instruction is within the projection screen of the first electronic device.

[0051] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device, and the first electronic device is wirelessly connected to the peripheral device.

[0052] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi, and the first electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi.

[0053] In another possible implementation, the peripheral device includes any one of the following: a mouse, a keyboard, a stylus, and a handle; the first instruction includes any one of the following: movement information of the mouse or stylus, typing information of the keyboard or handle; the movement information includes the distance and direction of movement.

[0054] In the seventh aspect, an embodiment of the present application provides a device for reverse control of screen projection, which can be applied to a second electronic device in a screen projection system to implement the method in the sixth aspect above. The functions of the device can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as a screen projection module, a connection module, and a processing module.

[0055] Among them, the screen projection module can be used to receive and display the projection screen of the first electronic device. The second electronic device is connected to the peripheral device, and the peripheral device is used to operate and control the screen displayed by the second electronic device; the second electronic device displays the projection screen of the first electronic device in a non-full screen manner; the connection module is used to receive a first instruction from the peripheral device when the second electronic device displays the projection screen of the first electronic device in a non-full screen manner, and the first instruction is generated by the peripheral device according to the user operation; the processing module is used to perform the corresponding operation according to the first instruction when it is determined that the user operation corresponding to the first instruction is outside the projection screen of the first electronic device.

[0056] In a possible implementation, the connection module is further configured to send connection information of the peripheral device to the first electronic device, where the connection information is information used by the second electronic device to connect to the peripheral device.

[0057] In another possible implementation, the connection module is specifically used to send connection information to the first electronic device when the second electronic device negotiates protocol parameters in real time with the first electronic device; or to synchronize connection information to the first electronic device when the second electronic device establishes a trusted connection with the first electronic device and / or logs in with the same account.

[0058] In another possible implementation, the connection information includes the physical address of the peripheral device and a token used for connection verification.

[0059] In another possible implementation, the connection module is further used to send location information of the projected image on the display screen of the second electronic device to the first electronic device.

[0060] In another possible implementation, the screen projection module is further used to receive and display the projection screen sent by the first electronic device after executing the corresponding operation when the user operation corresponding to the first instruction is within the projection screen of the first electronic device.

[0061] In another possible implementation, the processing module is further configured to not execute the first instruction when the user operation corresponding to the first instruction is within the projection screen of the first electronic device.

[0062] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device, and the first electronic device is wirelessly connected to the peripheral device.

[0063] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi, and the first electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi.

[0064] In another possible implementation, the peripheral device includes any one of the following: a mouse, a keyboard, a stylus, and a handle; the first instruction includes any one of the following: movement information of the mouse or stylus, typing information of the keyboard or handle; the movement information includes the distance and direction of movement.

[0065] In an eighth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory for storing instructions executable by the processor. When the processor is configured to execute the above instructions, the electronic device implements the method for reverse screen projection control as described in any one of the sixth aspect or the possible implementations of the sixth aspect.

[0066] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by an electronic device, the electronic device implements the method for reverse screen projection control as described in the sixth aspect or any one of the possible implementations of the sixth aspect.

[0067] In the tenth aspect, an embodiment of the present application provides a computer program product, including a computer-readable code. When the computer-readable code is run in an electronic device, the electronic device implements a method for reverse control of screen projection as described in any one of the sixth aspect or possible implementation methods of the sixth aspect.

[0068] It should be understood that the beneficial effects of the seventh to tenth aspects mentioned above can be found in the relevant description of the sixth aspect mentioned above, and will not be repeated here.

[0069] On the eleventh aspect, an embodiment of the present application provides a method for reverse control of screen projection, which can be applied to a screen projection system, wherein the screen projection system includes a first electronic device and a second electronic device connected to each other, and a peripheral device connected to the second electronic device, and the peripheral device is used to operate and control the screen displayed on the second electronic device; the method includes: the first electronic device sends a projection screen to the second electronic device; the second electronic device receives and displays the projection screen sent by the first electronic device; the first electronic device establishes a connection with the peripheral device; in response to the user operation, the peripheral device generates a first instruction according to the user operation; the peripheral device sends the first instruction to the first electronic device; the first electronic device receives the first instruction sent by the peripheral device; the first electronic device performs the corresponding operation according to the first instruction; the first electronic device sends the projection screen after performing the corresponding operation to the second electronic device; the second electronic device receives and displays the projection screen sent by the first electronic device after performing the corresponding operation.

[0070] In one possible implementation, before the first electronic device establishes a connection with the peripheral device, the method further includes: the second electronic device sends connection information of the peripheral device to the first electronic device, where the connection information is information used to connect the second electronic device with the peripheral device; the first electronic device receives the connection information sent by the second electronic device; the first electronic device establishes a connection with the peripheral device, including: the first electronic device establishes a connection with the peripheral device according to the connection information.

[0071] In another possible implementation, the second electronic device sends connection information of the peripheral device to the first electronic device, including: the second electronic device sends connection information to the first electronic device when negotiating protocol parameters in real time with the first electronic device; or, the second electronic device synchronizes connection information to the first electronic device when establishing a trusted connection with the first electronic device and / or logging in with the same account.

[0072] In another possible implementation, the connection information includes the physical address of the peripheral device and a token used for connection verification.

[0073] In another possible implementation, the second electronic device displays the projection screen of the first electronic device in full screen.

[0074] In another possible implementation, the second electronic device displays the projected screen of the first electronic device in non-full screen; the method also includes: the second electronic device sends the position information of the projected screen in the display screen of the second electronic device to the first electronic device; the first electronic device receives the position information of the projected screen sent by the second electronic device in the display screen of the second electronic device; the first electronic device performs a corresponding operation according to the first instruction, including: when the first electronic device determines that the user operation corresponding to the first instruction is within the projected screen of the first electronic device based on the position information of the projected screen in the display screen of the second electronic device, the first electronic device performs the corresponding operation according to the first instruction; when the first electronic device determines that the user operation corresponding to the first instruction is outside the projected screen of the first electronic device based on the position information of the projected screen in the display screen of the second electronic device, the first electronic device does not execute the first instruction.

[0075] In another possible implementation, the method also includes: the peripheral device sends a first instruction to the second electronic device; the second electronic device receives the first instruction from the peripheral device; when the second electronic device determines that the user operation corresponding to the first instruction is outside the projection screen of the first electronic device, the second electronic device performs a corresponding operation according to the first instruction; when the second electronic device determines that the user operation corresponding to the first instruction is within the projection screen of the first electronic device, the second electronic device does not execute the first instruction.

[0076] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device, and the first electronic device is wirelessly connected to the peripheral device.

[0077] In another possible implementation, the second electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi, and the first electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi.

[0078] In another possible implementation, the peripheral device includes any one of the following: a mouse, a keyboard, a stylus, and a handle; the first instruction includes any one of the following: movement information of the mouse or stylus, typing information of the keyboard or handle; the movement information includes the distance and direction of movement.

[0079] In a twelfth aspect, an embodiment of the present application provides a screen projection system, comprising a first electronic device and a second electronic device connected to each other, and a peripheral device connected to the second electronic device, the peripheral device being used to operate and control the projected screen. The screen projection system can be used to perform the method for reverse control of screen projection as described in any one of the possible implementations of the eleventh aspect or the eleventh aspect.

[0080] It should be understood that the beneficial effects of the above-mentioned eleventh and twelfth aspects can be found in the relevant descriptions of the above-mentioned first and sixth aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A flowchart of a method for reverse control of screen projection provided in related technology;

[0082] Figure 2 A schematic diagram of an application scenario of a method for reverse control of screen projection provided in an embodiment of the present application;

[0083] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0084] Figure 4 A flowchart of a method for reverse control of screen projection provided in an embodiment of the present application;

[0085] Figure 5 A schematic diagram of a process flow for real-time transport protocol related parameter negotiation provided in an embodiment of the present application;

[0086] Figure 6 A schematic diagram of a full-screen projection interface provided in an embodiment of the present application;

[0087] Figure 7 A schematic diagram of another full-screen projection interface provided in an embodiment of the present application;

[0088] Figure 8 A schematic diagram of a non-full-screen projection interface provided in an embodiment of the present application;

[0089] Figure 9 A schematic diagram of another non-full-screen projection interface provided in an embodiment of the present application;

[0090] Figure 10 A schematic diagram of a flow chart of the application of a method for reverse control of screen projection provided in an embodiment of the present application;

[0091] Figure 11 A schematic structural diagram of a device for reverse control of screen projection provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] At present, wireless screen projection technology is becoming more and more mature. People can use wireless screen projection technology to project the content displayed on an electronic device (such as a mobile phone, tablet, etc.) to other electronic devices (such as computers, smart TVs, smart screens, etc.) to achieve multi-screen collaboration or screen sharing functions. In order to facilitate users to reversely control the electronic device that projects the screen through the electronic device being projected during screen projection, a signaling channel will be set between the electronic device that projects the screen and the electronic device being projected, such as a user input back channel (UIBC). The instructions that allow the user to operate on the electronic device being projected can be transmitted to the mobile phone through the signaling channel so that the mobile phone can respond accordingly.

[0093] For example, when a first electronic device projects a screen onto a second electronic device, a user controls the first electronic device in reverse through the second electronic device. Figure 1 The figure shows a flow chart of a method for reverse control of screen projection provided by related technology. Figure 1 As shown, the method for reverse control of screen projection may include: the second electronic device receives an instruction corresponding to the user's operation (or called event injection), and then processes the instruction (or called event processing). After processing the received instruction, the second electronic device can package the processed instruction (or called event transmission packaging), and then transmit the packaged instruction to the first electronic device through a network (such as a signaling channel). When the first electronic device receives the instruction from the second electronic device (or called event injection), the first electronic device can respond according to the received instruction and perform image synthesis. When the image synthesis of the first electronic device is completed, the synthesized image can be encoded and encrypted and packaged, and then the encrypted and packaged image can be transmitted to the second electronic device through a network (such as a link formed by connecting the first electronic device and the second electronic device through a wireless fidelity (Wi-Fi) direct connection or a Wi-Fi local area network connection). When the second electronic device receives the packaged image from the first electronic device, it can unpack and decode the packaged image, and then send the decoded image to the display module for display (or called display).

[0094] For example, a user can project the display interface of a mobile phone to a computer through wireless screen projection technology, and then use a mouse, keyboard and other devices on the computer to operate the display interface of the mobile phone displayed on the computer to reversely control the mobile phone, thereby realizing the multi-screen collaboration function to view and use the mobile phone on the computer. Among them, when the user uses a mouse, keyboard and other devices on the computer to operate the display interface of the mobile phone displayed on the computer (for example, the user uses the mouse to click, drag and other operations on the controls in the display interface of the mobile phone displayed on the computer), after the computer receives the instruction sent by the mouse corresponding to the user's operation, the computer can first process the instruction sent by the mouse (for example, parse the instruction, etc.), and then package the processed instruction and send it to the mobile phone through the signaling channel so that the mobile phone can respond accordingly after receiving the instruction.

[0095] From the above, we can see that when the user uses the electronic device (i.e. Figure 1 The second electronic device shown in FIG) projects the screen to the electronic device (i.e. Figure 1 ), after the electronic device being projected receives the user's operation, it needs to process the user's operation first, and then transmit the user's operation instruction to the electronic device being projected, so that the electronic device being projected responds accordingly. Since the electronic device being projected needs to process the user's operation, the delay in the electronic device being projected receiving the user's operation instruction is relatively high, that is, after the user operates on the electronic device being projected, it takes a long time for the electronic device being projected to respond accordingly. When the user performs reverse control under projection, the tracking performance of the projection image displayed on the electronic device being projected is poor.

[0096] To solve the above problem, an embodiment of the present application provides a method for reverse screen projection control, which can be applied to a scenario where a first electronic device projects a screen to a second electronic device. For example, take the first electronic device as a mobile phone and the second electronic device as a computer. Figure 2 The following is a schematic diagram showing an application scenario of a method for reverse control of screen projection provided by an embodiment of the present application. Figure 2 As shown, the mobile phone can cast its screen to the computer to cast the display screen of the mobile phone (such as the mobile phone desktop (or main interface) including icons of various applications installed on the mobile phone) to the display interface of the computer. The user can control and operate the mobile phone through reverse control on the computer. For example, the user can click on the icon of an application on the cast mobile phone desktop on the computer to reversely control the mobile phone to open the corresponding application.

[0097] The method for reverse control of screen projection may include: when a first electronic device projects a screen to a second electronic device, when the peripheral device of the second electronic device, that is, an accessory (such as a human interface device (HID) device such as a mouse, keyboard, stylus, and game controller, or a custom device, etc.) receives a user operation, the instructions and data corresponding to the user operation can be sent directly to the first electronic device, that is, the peripheral device does not need to be forwarded to the first electronic device through the second electronic device. After the first electronic device receives the instruction sent by the peripheral device, it can make a corresponding response according to the instruction, and then send the display image and / or audio obtained after the response instruction to the second electronic device for screen projection.

[0098] In which, when the first electronic device projects the screen to the second electronic device, the first electronic device and the second electronic device, the peripheral device and the first electronic device, and the second electronic device can be connected through short-range communication (such as Bluetooth connection, wireless fidelity (Wi-Fi) direct connection, etc.), or can also be connected through a wireless local area network-based method, which is not restricted here.

[0099] In this way, when the first electronic device is projecting its screen onto the second electronic device, when the user reversely controls the first electronic device on the second electronic device, the instructions corresponding to the operations performed by the user through the peripheral device of the second electronic device can be directly sent by the peripheral device to the first electronic device, so that the first electronic device can make a corresponding response according to the user's operation. This avoids the second electronic device first receiving the instructions sent by its peripheral device and then processing the instructions before transmitting them to the first electronic device, simplifies the path for controlling the transmission of instructions when the user reversely controls the first electronic device through the peripheral device of the second electronic device, and thus improves the reverse control delay in the projection state, improves the response speed of the projection electronic device (i.e., the first electronic device) to the user's operation, and improves the follow-up of the projection screen displayed by the projection electronic device (i.e., the second electronic device) relative to the user's operation.

[0100] Below, the method of reverse control of screen projection provided by the embodiment of the present application will be described with reference to the accompanying drawings.

[0101] In the embodiment of the present application, the first electronic device and the second electronic device may be a mobile phone, a tablet computer, a handheld computer, a PC, a cellular phone, a personal digital assistant (PDA), a wearable device (such as a smart watch, a smart bracelet), a smart home device (such as a TV), a car computer (such as an onboard computer), a smart screen, a game console, a smart projector, a smart TV box, and an augmented reality (AR) / virtual reality (VR) device. The embodiment of the present application does not impose any special restrictions on the specific device form of the electronic device.

[0102] Optionally, in some embodiments, the first electronic device and the second electronic device may have the same device form factor. For example, both the first electronic device and the second electronic device may be mobile phones.

[0103] Alternatively, in other embodiments, the first electronic device and the second electronic device may have different device forms. For example, the first electronic device may be a mobile phone and the second electronic device may be a smart screen, or the first electronic device may be a mobile phone and the second electronic device may be a computer.

[0104] For example, taking the electronic devices (such as the first electronic device and the second electronic device) as mobile phones, Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown. That is, for example, Figure 3 The electronic device shown may be a mobile phone.

[0105] like Figure 3As shown, the electronic device may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a sensor module 380, a button 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a subscriber identification module (SIM) card interface 395, etc. Among them, the sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, an air pressure sensor 380C, a magnetic sensor 380D, an acceleration sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, an ambient light sensor 380L, a bone conduction sensor 380M, etc.

[0106] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0107] The processor 310 may include one or more processing units. For example, the processor 310 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0108] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0109] Processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 310 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 310. If processor 310 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 310 latency, and thus improves system efficiency.

[0110] In some embodiments, the processor 310 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0111] The charging management module 340 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 can receive charging input from the wired charger via the USB interface 330. In some wireless charging embodiments, the charging management module 340 can receive wireless charging input via the electronic device's wireless charging coil. While charging the battery 342, the charging management module 340 can also provide power to the electronic device via the power management module 341.

[0112] The power management module 341 is used to connect the battery 342, the charging management module 340, and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340 and provides power to the processor 310, the internal memory 321, the external memory, the display 394, the camera 393, and the wireless communication module 360. The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 341 can also be set in the processor 310. In other embodiments, the power management module 341 and the charging management module 340 can also be set in the same device.

[0113] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, modem processor and baseband processor.

[0114] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0115] The mobile communication module 350 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to electronic devices. The mobile communication module 350 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 350 can be set in the processor 310. In some embodiments, at least some of the functional modules of the mobile communication module 350 can be set in the same device as at least some of the modules of the processor 310.

[0116] The wireless communication module 360 ​​can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 360 ​​can be one or more devices that integrate at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 310. The wireless communication module 360 ​​can also receive the signal to be sent from the processor 310, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0117] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 350, and antenna 2 is coupled to wireless communication module 360, so that the electronic device can communicate with a network and other devices via wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0118] The electronic device implements display functionality through a GPU, display screen 394, and an application processor. A GPU is a microprocessor for image processing that connects display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 310 may include one or more GPUs that execute program instructions to generate or modify display information.

[0119] Display screen 394 is used to display images, videos, and the like. Display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLED, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device can include one or N display screens 394, where N is a positive integer greater than one.

[0120] The electronic device can realize the shooting function through the ISP, camera 393, video codec, GPU, display 394 and application processor.

[0121] The ISP processes data fed back by camera 393. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 393.

[0122] The camera 393 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 393, where N is a positive integer greater than 1.

[0123] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency, the DSP performs a Fourier transform on the frequency energy.

[0124] Video codecs are used to compress or decompress digital video. Electronic devices may support one or more video codecs. This allows them to play or record videos in a variety of encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0125] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in electronic devices, such as image recognition, face recognition, speech recognition, and text comprehension.

[0126] The internal memory 321 can be used to store computer executable program codes, which include instructions. The processor 310 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 321. The internal memory 321 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0127] Of course, it is understandable that the above Figure 3 The following is only an example of an electronic device in the form of a mobile phone. If the electronic device is a tablet computer, handheld computer, PC, PDA, wearable device (such as smart watch, smart bracelet), smart home device (such as TV), car computer (such as car computer), smart screen, game console and AR / VR device, the structure of the electronic device may include more than Figure 3 The structure shown in the figure is less than Figure 3 More structures are shown in the figure, which are not limited here.

[0128] The methods in the following embodiments can all be implemented in an electronic device having the above hardware structure.

[0129] Take the example of a first electronic device projecting its screen onto a second electronic device and a user performing reverse control of the first electronic device on the second electronic device. Figure 4 The figure shows a flow chart of a method for reverse control of screen projection provided by an embodiment of the present application. Figure 4 As shown, the method may include the following S401 to S404.

[0130] S401: When a first electronic device projects a screen onto a second electronic device, the first electronic device is connected to a peripheral device of the second electronic device.

[0131] Among them, before the first electronic device projects the screen to the second electronic device, the first electronic device and the second electronic device need to establish a connection. In the process of establishing a connection between the first electronic device and the second electronic device, a real-time negotiation protocol will be performed between the first electronic device and the second electronic device, such as the negotiation of relevant parameters of the real-time streaming protocol (RTSP), so that after the parameter negotiation, the first electronic device sends the audio and video stream to the second electronic device according to the negotiation result, so that the second electronic device displays the display screen of the first electronic device (that is, the second electronic device displays the projection screen of the first electronic device) and plays the sound of the first electronic device. Moreover, through negotiation, the second electronic device and the first electronic device can determine the specific position of the display screen of the first electronic device in the display interface of the second electronic device when the screen is not fully projected (that is, the second electronic device does not display the display screen of the first electronic device (that is, the projection screen) in full screen) (usually, after the screen is projected, the second electronic device can also synchronize the canvas of its display interface to the first electronic device, and the corresponding position (or position information) of the window of the display screen (or projection screen) of the first electronic device on the canvas (or display screen of the second electronic device)).

[0132] For example, Figure 5As shown, the process of negotiating parameters related to the real-time streaming protocol between a first electronic device and a second electronic device may include: the first electronic device sends a real-time streaming protocol option request message (i.e., an M1Req message) to the second electronic device. After receiving the M1Req message, the second electronic device may determine the specific method of the real-time streaming protocol to be adopted. In response, the second electronic device returns a real-time streaming protocol option reply message (i.e., an M1Rsp message) to the first electronic device, which may include the determined specific method of the real-time streaming protocol. The real-time streaming protocol option request message and reply message enable the first and second electronic devices to negotiate and determine the specific method of the real-time streaming protocol to be adopted. Then, the second electronic device may send a media initialization description information request message (i.e., an M2Req message) to the first electronic device. After receiving the M2Req message, the first electronic device may return a media initialization description information reply message (i.e., an M2Rsp message) to the second electronic device, which may include media initialization description information. The media initialization description information request message and reply message enable the first and second electronic devices to negotiate and determine their respective media capabilities. Afterwards, the first electronic device can send a real-time transmission parameter request message (i.e., an M3Req message) to the second electronic device, and accordingly, the second electronic device can return a real-time transmission parameter reply message (i.e., an M3Rsp message) to the first electronic device after receiving the M3Req message, and the message may include real-time transmission parameters. Through the real-time transmission parameter request message and the reply message, the first electronic device and the second electronic device can negotiate to obtain the parameters required for real-time transmission between the two. Then, the first electronic device can send a parameter setting request message (i.e., an M4Req message) to the second electronic device, and the second electronic device can set the real-time transmission parameters after receiving the M4Req message, and then the second electronic device can return a parameter setting reply message (i.e., an M4Rsp message) to the first electronic device, and the first electronic device can set the real-time transmission parameters after receiving the M4Rsp message. By setting the parameter request message and the reply message, the first electronic device and the second electronic device can respectively set the parameters required for real-time transmission. After the first electronic device and the second electronic device have set the parameters, the second electronic device can send a session establishment request message (i.e., a Steup message) to the first electronic device. In response, the first electronic device establishes a session with the second electronic device based on the session establishment request message, and then returns a session establishment reply message (i.e., a Resp message) to the second electronic device. The session establishment reply message may include a session identifier, session-related information, etc.After a session is established between the first electronic device and the second electronic device, the second electronic device can send a streaming data (e.g., audio and video streams displayed and played by the first electronic device) request message (i.e., Play message) to the first electronic device. Correspondingly, the first electronic device can return a reply message (i.e., Resp message) to the second electronic device to inform the second electronic device that streaming data will be sent. Then, the first electronic device can send streaming data to the second electronic device, thereby casting the screen of the first electronic device to the second electronic device.

[0133] In some possible implementations, when a first electronic device projects a screen onto a second electronic device, the second electronic device may send connection information of a peripheral device connected thereto (e.g., the physical address of the peripheral device, a token, etc.) to the first electronic device, so that the first electronic device can connect to the peripheral device according to the connection information of the peripheral device connected to the second electronic device; wherein the token may be determined by negotiation when the peripheral device is connected to the second electronic device. For example, the first electronic device may connect to the peripheral device according to the received physical address of the peripheral device, and the peripheral device may perform connection verification through the token when connected; the verification process may be, for example: the first electronic device sends a random number to the peripheral device, and then the first electronic device and the peripheral device verify whether the random number is consistent with the derived token generated by the token, and if they are consistent, the connection between the peripheral device and the first electronic device is verified.

[0134] As an example, the second electronic device may send the connection information of the peripheral device connected thereto to the first electronic device, as described above. Figure 5 In the real-time transport protocol related parameter negotiation process shown, the second electronic device carries the connection information of the peripheral device connected to the second electronic device in the real-time transport parameter reply message (ie, M3Rsp message) returned by the second electronic device to the first electronic device.

[0135] In other possible implementations, the second electronic device may send the connection information of the peripheral device connected to it to the first electronic device. Before the first electronic device projects the screen to the second electronic device, if the first electronic device and the second electronic device are logged in with the same account, or a trusted connection is established between the first electronic device and the second electronic device (such as a binding relationship between the first electronic device and the second electronic device), the second electronic device may synchronize the device information of the peripheral device connected to it to the first electronic device. This allows the first electronic device to connect to the peripheral device of the second electronic device based on the device information when the first electronic device projects the screen to the second electronic device.

[0136] It should be noted that the connection between the peripheral device and the first electronic device and the second electronic device can be a Bluetooth connection, a Wi-Fi direct connection, etc., which is not limited here.

[0137] When a user reversely controls the first electronic device on the second electronic device through the peripheral device of the second electronic device, the peripheral device of the second electronic device can send an instruction corresponding to the user operation to the first electronic device. For example, the peripheral device of the second electronic device can execute the following S402.

[0138] S402: The peripheral device of the second electronic device receives the user operation and sends an instruction corresponding to the user operation to the first electronic device.

[0139] Among them, the user operation can be an operation such as sliding, clicking, and typing performed by the user through the peripheral device. For example, if the peripheral device is a mouse, the user operation can be a sliding, clicking, or dragging operation performed by the user through the mouse. For another example, if the peripheral device is a keyboard, the user operation can be a typing operation performed by the user through the keyboard. The instruction sent by the peripheral device can include information corresponding to the specific user operation (such as sliding, clicking, typing, dragging, etc.) (such as movement information of the mouse or stylus, typing information of the keyboard or handle, etc. The movement information can include the distance and direction of movement).

[0140] For example, take the peripheral device as a mouse, and the user operation as the user sliding the mouse and clicking. When the first electronic device is not fully screen projected to the second electronic device, the second electronic device can synchronize the canvas of the display interface of the second electronic device to the first electronic device, the corresponding position of the display screen of the first electronic device on the canvas of the display interface of the second electronic device, and the position of the mouse cursor on the canvas of the display interface of the second electronic device (i.e., coordinates). When the user moves the mouse to slide, the mouse can send an instruction including the direction and distance of the mouse movement to the first electronic device. The first electronic device can determine the position of the mouse cursor on the canvas of the display interface of the second electronic device after the user moves the mouse based on the position of the mouse cursor synchronized by the second electronic device on the canvas of the display interface of the second electronic device, combined with the received direction and distance of the mouse movement. When the user clicks the mouse button, the mouse can send an instruction including indication information for indicating that the user has performed a click operation to the first electronic device. The first electronic device can determine that the user has performed a click operation based on the indication information in the instruction. When a first electronic device is projecting its full screen to a second electronic device, the first electronic device can project its own display screen as a full-screen mirror image of the projected screen to the second electronic device, or draw the projected screen according to the size of the display screen of the second electronic device and then project it full-screen to the second electronic device. At this time, the screen displayed by the second electronic device is completely the screen drawn by the first electronic device. Therefore, the mouse cursor can be drawn by the first electronic device in the projected screen of the first electronic device and projected to the second electronic device. When the user moves the mouse to slide, the mouse can send an instruction including the direction and distance of the mouse movement to the first electronic device. The first electronic device can determine the position of the mouse cursor after the user moves the mouse based on the received direction and distance of the mouse movement. Among them, after the first electronic device projects the screen, the mouse cursor can be drawn at a preset initial position. For example, after the first electronic device projects the screen, the mouse cursor can be displayed in the upper left corner of the projected screen, or the mouse cursor can be displayed in the center of the projected screen, etc.

[0141] For another example, take the external device being a keyboard and the user operation being typing a character on the keyboard. When the user types a character on the keyboard, the keyboard may send a command including indication information indicating the character typed by the user to the first electronic device. The first electronic device may determine that the user has typed the corresponding character based on the indication information in the command.

[0142] Optionally, when the peripheral device of the second electronic device sends an instruction corresponding to the user operation to the first electronic device, the instruction may be sent only to the first electronic device, or may be sent to both the first electronic device and the second electronic device.

[0143] For example, when the first electronic device projects the screen to the second electronic device, the first electronic device and the second electronic device can perform the following operations: Figure 5 In the negotiation process shown, it is pre-negotiated whether the screen projection of the first electronic device to the second electronic device is full-screen projection or non-full-screen projection.

[0144] like Figure 6 As shown, when the first electronic device (such as the mobile phone shown in the figure) projects the screen to the second electronic device (such as the computer shown in the figure), the first electronic device can project the full screen of the screen to the second electronic device in equal proportions. Figure 7 As shown, when the first electronic device (such as the mobile phone shown in the figure) projects the screen to the second electronic device (such as the computer shown in the figure), the first electronic device can adaptively project the full screen to the second electronic device according to the display screen size of the second electronic device, so that the projected screen adapts to the display screen size of the second electronic device. Figure 6 or Figure 7 In the full-screen projection shown, the display screen of the first electronic device will be displayed in full screen on the second electronic device, and the display screen of the second electronic device itself will not be displayed. When the second electronic device displays the display screen projected from the first electronic device in full screen (that is, the first electronic device projects the display screen to the second electronic device in full screen), since the second electronic device does not display the display screen generated by itself, the operations performed by the user through the peripheral device of the second electronic device will not be operations performed on the second electronic device. So at this time, the peripheral device of the second electronic device can only send instructions to the first electronic device (such as, the peripheral device directly belongs to the first electronic device). Alternatively, the peripheral device of the second electronic device sends instructions corresponding to the user operation to both the first electronic device and the second electronic device, but the second electronic device does not accept or discard the received instructions.

[0145] like Figure 8As shown, when a first electronic device (such as a mobile phone as shown in the figure) projects its screen to a second electronic device (such as a computer as shown in the figure), it can be a non-full screen projection of the first electronic device to the second electronic device. At this time, the second electronic device displays both its own display screen 801 and the display screen 802 of the first electronic device. When the second electronic device displays the display screen (i.e., the projection screen) projected from the first electronic device in a non-full screen (i.e., the first electronic device projects its screen to the second electronic device in a non-full screen), since the second electronic device also displays its own generated display screen, the operation performed by the user through the peripheral device of the second electronic device may also be an operation performed on the second electronic device. Therefore, at this time, the peripheral device of the second electronic device can send instructions to both the first electronic device and the second electronic device. At this time, after receiving the instruction, the first electronic device and the second electronic device can determine whether to respond to the instruction based on whether the location of the user operation corresponding to the instruction is within their own display screen. For example, if the peripheral device of the second electronic device is a mouse and the user clicks the mouse, then after receiving the instruction sent by the mouse, the first electronic device and the second electronic device can determine the location of the user click based on the instruction to determine whether the user operation is an operation performed on themselves. If the above location is within the display screen of the first electronic device, the first electronic device responds according to the instruction. If the above-mentioned position is within the display screen of the second electronic device, the second electronic device responds according to the instruction. That is, when the user moves the mouse and clicks, the first electronic device can determine the position of the mouse cursor within the canvas of the display screen of the second electronic device after the mouse cursor moves, i.e., the position of the mouse click operation within the display screen of the second electronic device, based on the movement direction and distance sent after the mouse moves, the position of the mouse cursor synchronized by the second electronic device within the canvas of the display screen of the second electronic device, and the corresponding positions of the canvas of the display screen of the second electronic device and the display screen of the first electronic device within the canvas of the display screen of the second electronic device. If the position determined by the first electronic device is within the display screen of the first electronic device, the first electronic device responds to the mouse movement and click operation. The second electronic device can determine the position of the mouse cursor within the canvas of the display screen of the second electronic device after the mouse moves, i.e., the position of the mouse click operation within the display screen of the second electronic device, based on the movement direction and distance sent after the mouse moves. If the position determined by the second electronic device is within the display screen of the second electronic device, the second electronic device responds to the mouse movement and click operation.

[0146] In some possible implementations, the display screen of the first electronic device displayed by the second electronic device is as follows: Figure 8 In addition to the screen currently displayed on the first electronic device, it can also be multiple window screens projected from the first electronic device to the second electronic device. Figure 9As shown, when the first electronic device (such as the mobile phone shown in the figure) projects the screen to the second electronic device (such as the computer shown in the figure), it can be a multi-window non-full-screen projection of the first electronic device to the second electronic device. At this time, the second electronic device displays both its own display screen 901 and multiple window screens projected from the first electronic device. For example, the window screen 902 of the first application and the window screen 903 of the main interface of the first electronic device. Among them, the above-mentioned multiple window screens can be sent to the second electronic device after being drawn in the background of the first electronic device.

[0147] S403: The first electronic device responds according to the received instruction corresponding to the user operation.

[0148] The user operation may be a click, slide, drag, or type operation. For example, if a user clicks a control on the display screen of the first electronic device, the first electronic device may respond to the click of the control according to the received instruction corresponding to the user operation. For another example, if a user drags an icon on the display screen of the first electronic device, the first electronic device may respond according to the received instruction corresponding to the user operation and generate a display screen showing the icon being dragged.

[0149] Typically, the screen projection of the first electronic device to the second electronic device can be full-screen projection or non-full-screen projection.

[0150] Therefore, as an example, such as the related example in S402, when the first electronic device is full-screen projecting to the second electronic device, the peripheral device of the second electronic device may only send instructions corresponding to the user operation to the first electronic device. Alternatively, the peripheral device of the second electronic device sends instructions corresponding to the user operation to both the first electronic device and the second electronic device, but the second electronic device does not accept or discards the received instructions. In this case, when the first electronic device receives the instruction corresponding to the user operation, it can respond directly according to the instruction.

[0151] As another example, such as the related example in S402, when the first electronic device is projecting a non-full screen to the second electronic device, the peripheral device of the second electronic device can send instructions to both the first electronic device and the second electronic device. At this time, since the first electronic device is projecting a screen to the second electronic device, the two can mutually determine the specific position of the display screen of the first electronic device in the display interface of the second electronic device through negotiation. Therefore, after the first electronic device and the second electronic device respectively receive the instructions corresponding to the user operation, the first electronic device can determine whether the position of the user operation (that is, the coordinates of the position corresponding to the user operation in the display interface of the second electronic device) is in the display screen of the first electronic device or the display screen of the second electronic device. If the position of the user operation is in the display screen of the first electronic device, the first electronic device processes the instruction and responds accordingly. If the position of the user operation is not in the display screen of the first electronic device, the first electronic device does not process the instruction. Similarly, the second electronic device can determine whether the location of the user operation (i.e., the coordinates of the location corresponding to the user operation in the display interface of the second electronic device) is in the display screen of the first electronic device or the display screen of the second electronic device. If the location of the user operation is in the display screen of the second electronic device, the second electronic device processes the instruction and responds accordingly. If the location of the user operation is not in the display screen of the second electronic device, the second electronic device does not process the instruction.

[0152] Optionally, when the first electronic device responds to the received instruction corresponding to the user operation, the display screen of the first electronic device may change. Alternatively, the first electronic device may play audio accordingly, or the audio currently being played may change. Therefore, the first electronic device may also send the corresponding display screen and / or audio in response to the instruction corresponding to the user operation to the second electronic device. For example, the first electronic device may execute the following S404.

[0153] S404: The first electronic device sends a corresponding display image and / or audio in response to the instruction corresponding to the user operation to the second electronic device.

[0154] For example, the process of sending a display image and / or audio from a first electronic device to a second electronic device can be the same as the transmission process when the current electronic device is casting a screen. For example, the display image and / or audio of the first electronic device can be encoded by a video codec and / or an audio codec (optionally, the encoded display image and / or audio can also be compressed and converted into a packetized elementary stream (PES) packet format to reduce the data size during subsequent transmission to the second electronic device). Then, the relevant parameters of the high-bandwidth digital content protection technology (HDCP) protocol are added to the encoded display image and / or audio to encrypt the display image and / or audio. The display image and / or audio are then merged in the form of a transport stream (MPEG2-TS). Then add the header of the real-time transport protocol (RTP), and finally send it to the second electronic device via the user datagram protocol (UDP), transmission control protocol (TCP) or internet protocol (IP) using the connection established between the first electronic device and the second electronic device (such as Wi-Fi direct connection or Wi-Fi local area network connection). Accordingly, the second electronic device can follow the reverse process of the above to decapsulate, MPEG2-TS partition, HDCP decrypt, PES decompress and decode the received data for display and / or audio playback.

[0155] S405: The second electronic device displays the display screen of the first electronic device and / or plays the audio of the first electronic device.

[0156] For example, based on the above Figure 4 The method shown takes the first electronic device as a mobile phone and the second electronic device as a computer as an example, and the display screen of the first electronic device changes after reverse control. Figure 10 The following is a flow chart showing the application of a method for reverse control of screen projection provided by an embodiment of the present application. Figure 10As shown, the process of reverse control of screen projection may include: after the computer's peripheral device receives the user's operation, it sends the instruction corresponding to the user's operation to the mobile phone (i.e., event injection), and then the mobile phone can respond according to the received instruction to perform image synthesis. When the mobile phone image synthesis is completed, the synthesized image can be encoded and encrypted and packaged, and then the encrypted and packaged image can be transmitted to the computer through the network (such as a link formed by connecting the mobile phone and the computer in an end-to-end or LAN-based manner). When the computer receives the packaged image from the mobile phone, it can unpack and decode the packaged image, and then send the decoded image to the display module for display (or called sending for display).

[0157] By adopting the method in the above embodiment, when the first electronic device is projecting the screen to the second electronic device, when the user reversely controls the first electronic device on the second electronic device, the instructions corresponding to the operations performed by the user through the peripheral device of the second electronic device can be directly sent by the peripheral device of the second electronic device to the first electronic device, so that the first electronic device can make a corresponding response according to the user's operation. This avoids the second electronic device first receiving the instructions sent by its peripheral device and then processing the instructions before transmitting them to the first electronic device, simplifies the path of control instruction transmission when the user reversely controls the first electronic device through the peripheral device of the second electronic device, and thus improves the reverse control delay in the projection state, improves the response speed of the projection electronic device to the user's operation, and improves the follow-up of the projection screen displayed by the projection electronic device relative to the user's operation.

[0158] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the interaction between the first electronic device and the second electronic device. It is understandable that in order to achieve the above functions, those skilled in the art should easily realize that, in combination with the steps of each example described in the embodiment disclosed herein, it is also possible to implement only the method involved in the first electronic device, or only the method involved in the second electronic device, and the embodiment of the present application does not limit this.

[0159] Corresponding to the method in the aforementioned embodiment, the embodiment of the present application also provides a device for reverse control of screen projection. The device can be applied to the above-mentioned first electronic device to implement the method related to the first electronic device in the aforementioned embodiment. The function of the device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, Figure 11 A schematic diagram of the structure of a device for reverse control of screen projection is shown. Figure 11As shown, the device includes: a screen projection module 1101, a connection module 1102 and a processing module 1103, etc. The screen projection module 1101, the connection module 1102 and the processing module 1103 can be used together to implement the method related to the first electronic device in the above-mentioned embodiment. Optionally, the embodiment of the present application also provides a device for reverse control of screen projection. The device can be applied to the above-mentioned second electronic device to implement the method related to the second electronic device in the above-mentioned embodiment. The function of the device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. For example, the specific structure of the device can refer to the following Figure 10 The structure of the device for reverse control of screen projection is shown.

[0160] It should be understood that the division of units or modules (hereinafter referred to as units) in the above devices is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a single physical entity, or physically separated. Furthermore, the units in the device may be implemented entirely in the form of software called through processing elements; entirely in the form of hardware; or partially in the form of software called through processing elements, and partially in the form of hardware.

[0161] For example, each unit can be a separately established processing element, or it can be integrated into a certain chip of the device for implementation. In addition, it can also be stored in a memory in the form of a program, and called by a certain processing element of the device to execute the function of the unit. In addition, all or part of these units can be integrated together, or they can be implemented independently. The processing element described here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be implemented by the integrated logic circuit of the hardware in the processor element or in the form of software called by the processing element.

[0162] In one example, the units in the above apparatus may be one or more integrated circuits configured to implement the above method, such as one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of at least two of these integrated circuit forms.

[0163] For another example, when the units in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a CPU or other processor that can call programs. For another example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0164] In one implementation, the units implementing the corresponding steps of the above methods in the apparatus described above may be implemented in the form of a processing element scheduling program. For example, the apparatus may include a processing element and a storage element, with the processing element invoking a program stored in the storage element to execute the method described in the above method embodiments. The storage element may be a storage element on the same chip as the processing element, i.e., an on-chip storage element.

[0165] In another implementation, the program for executing the above method may be stored in a memory element on a different chip from the processing element, i.e., an off-chip memory element. In this case, the processing element calls or loads the program from the off-chip memory element onto the on-chip memory element to call and execute the method described in the above method embodiment.

[0166] For example, an embodiment of the present application may also provide a device, such as: an electronic device (such as the first electronic device or the second electronic device in the aforementioned embodiment, etc.), which may include: a processor, a memory for storing instructions executable by the processor. When the processor is configured to execute the above instructions, the electronic device implements the method of reverse control of screen projection implemented by the first electronic device or the second electronic device in the aforementioned embodiment. The memory may be located inside the electronic device or outside the electronic device. And the processor includes one or more.

[0167] In another implementation, the unit of the apparatus implementing each step of the above method may be configured as one or more processing elements, which may be provided on the corresponding electronic device. The processing elements may be integrated circuits, such as one or more ASICs, one or more DSPs, one or more FPGAs, or a combination of these integrated circuits. These integrated circuits may be integrated together to form a chip.

[0168] For example, embodiments of the present application further provide a chip system that can be applied to the above-mentioned electronic devices. The chip system includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected via circuits; the processors receive and execute computer instructions from the memory of the electronic device via the interface circuits to implement the methods related to the first electronic device or the second electronic device in the above method embodiments.

[0169] An embodiment of the present application also provides a computer program product, including computer instructions executed by an electronic device, such as the electronic device described above.

[0170] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0172] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0174] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, such as a program. The software product is stored in a program product, such as a computer-readable storage medium, and includes a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0175] For example, an embodiment of the present application may further provide a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by an electronic device, the electronic device implements the method for reverse control of screen projection as described in the aforementioned method embodiment.

[0176] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for reverse control of screen projection, characterized in that: include: The first electronic device sends a projection screen to the second electronic device, and the second electronic device is connected to a peripheral device, and the peripheral device is used to operate and control the screen displayed by the second electronic device; The first electronic device receives connection information of the peripheral device sent by the second electronic device, where the connection information is information used by the second electronic device to connect to the peripheral device; The first electronic device establishing a connection with the peripheral device includes: the first electronic device establishing a connection with the peripheral device according to the connection information; The first electronic device receives a first instruction sent by the peripheral device, where the first instruction is generated by the peripheral device according to a user operation; The first electronic device performs a corresponding operation according to the first instruction; The first electronic device sends the projection image after performing the corresponding operation to the second electronic device.

2. The method according to claim 1, characterized in that The first electronic device receiving the connection information of the peripheral device sent by the second electronic device includes: The first electronic device receives the connection information when negotiating real-time negotiation protocol parameters with the second electronic device; or The first electronic device receives the connection information when establishing a trusted connection with the second electronic device and / or logging in with the same account.

3. The method according to claim 1 or 2, characterized in that The connection information includes the physical address of the peripheral device and a token used for connection verification.

4. The method according to claim 1 or 2, characterized in that The second electronic device displays the projection screen of the first electronic device in full screen.

5. The method according to claim 1 or 2, characterized in that The second electronic device displays the projection screen of the first electronic device in a non-full screen manner; The method further comprises: The first electronic device receives the position information of the projected image on the display screen of the second electronic device sent by the second electronic device; The first electronic device performs a corresponding operation according to the first instruction, including: When the first electronic device determines, based on the position information of the projection screen on the display screen of the second electronic device, that the user operation corresponding to the first instruction is within the projection screen of the first electronic device, the first electronic device performs a corresponding operation according to the first instruction.

6. The method according to claim 1 or 2, characterized in that The second electronic device displays the projection screen of the first electronic device in a non-full screen manner; and the method further includes: The first electronic device receives the position information of the projected image on the display screen of the second electronic device sent by the second electronic device; The first electronic device determines, based on the position information of the projected screen on the display screen of the second electronic device, whether the user operation corresponding to the first instruction is outside the projected screen of the first electronic device; If the first electronic device determines that the user operation corresponding to the first instruction is outside the projection screen of the first electronic device, the first electronic device does not execute the first instruction.

7. The method according to claim 1 or 2, characterized in that The second electronic device is wirelessly connected to the peripheral device, and the first electronic device is wirelessly connected to the peripheral device.

8. The method according to claim 1 or 2, characterized in that The second electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi, and the first electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi.

9. The method according to claim 1 or 2, characterized in that The peripheral device includes any one of the following: a mouse, a keyboard, a stylus and a handle; the first instruction includes any one of the following: movement information of a mouse or a stylus, typing information of a keyboard or a handle; the movement information includes the distance and direction of movement.

10. A method for reverse control of screen projection, characterized in that: include: The second electronic device receives and displays the screen projection of the first electronic device, the second electronic device is connected to a peripheral device, and the peripheral device is used to operate and control the screen displayed by the second electronic device; the second electronic device displays the screen projection of the first electronic device in a non-full screen manner; The second electronic device receives a first instruction from the peripheral device, where the first instruction is generated by the peripheral device according to a user operation; When the second electronic device determines that the user operation corresponding to the first instruction is outside the projection screen of the first electronic device, the second electronic device performs a corresponding operation according to the first instruction; The second electronic device sends connection information of the peripheral device to the first electronic device, where the connection information is information used by the second electronic device to connect with the peripheral device; the connection information is used for the first electronic device to establish a connection with the peripheral device.

11. The method according to claim 10, characterized in that The second electronic device sending the connection information of the peripheral device to the first electronic device includes: The second electronic device sends the connection information to the first electronic device when negotiating real-time negotiation protocol parameters with the first electronic device; or The second electronic device synchronizes the connection information with the first electronic device when establishing a trusted connection with the first electronic device and / or logging in with the same account.

12. The method according to claim 10 or 11, characterized in that The connection information includes the physical address of the peripheral device and a token used for connection verification.

13. The method according to claim 10 or 11, characterized in that The method further comprises: The second electronic device sends the position information of the projection image on the display screen of the second electronic device to the first electronic device.

14. The method according to claim 10 or 11, characterized in that The method further comprises: When the user operation corresponding to the first instruction is performed within the projection screen of the first electronic device, the second electronic device receives and displays the projection screen sent by the first electronic device after performing the corresponding operation.

15. The method according to claim 10 or 11, characterized in that The method also includes: when the user operation corresponding to the first instruction is within the projection screen of the first electronic device, the second electronic device does not execute the first instruction.

16. The method according to claim 10 or 11, characterized in that The second electronic device is wirelessly connected to the peripheral device, and the first electronic device is wirelessly connected to the peripheral device.

17. The method according to claim 10 or 11, characterized in that The second electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi, and the first electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi.

18. The method according to claim 10 or 11, characterized in that The peripheral device includes any one of the following: a mouse, a keyboard, a stylus and a handle; the first instruction includes any one of the following: movement information of a mouse or a stylus, typing information of a keyboard or a handle; the movement information includes the distance and direction of movement.

19. A method for reverse control of screen projection, characterized in that: Applied to a screen projection system, the screen projection system includes a first electronic device and a second electronic device connected to each other, and a peripheral device connected to the second electronic device, the peripheral device being used to operate and control a screen displayed by the second electronic device; the method includes: The first electronic device sends a projection screen to the second electronic device; The second electronic device receives and displays the projection image sent by the first electronic device; The second electronic device sends connection information of the peripheral device to the first electronic device, where the connection information is information used by the second electronic device to connect to the peripheral device; The first electronic device receives the connection information sent by the second electronic device; The first electronic device establishing a connection with the peripheral device includes: the first electronic device establishing a connection with the peripheral device according to the connection information; In response to a user operation, the peripheral device generates a first instruction according to the user operation; The peripheral device sends the first instruction to the first electronic device; The first electronic device receives the first instruction sent by the peripheral device; The first electronic device performs a corresponding operation according to the first instruction; The first electronic device sends the projection image after performing the corresponding operation to the second electronic device; The second electronic device receives and displays the projection screen sent by the first electronic device after performing the corresponding operation.

20. The method according to claim 19, characterized in that The second electronic device sending the connection information of the peripheral device to the first electronic device includes: The second electronic device sends the connection information to the first electronic device when negotiating real-time negotiation protocol parameters with the first electronic device; or The second electronic device synchronizes the connection information with the first electronic device when a trusted connection is established with the first electronic device and / or the second electronic device logs in with the same account.

21. The method according to claim 19 or 20, characterized in that The connection information includes the physical address of the peripheral device and a token used for connection verification.

22. The method according to claim 19 or 20, characterized in that The second electronic device displays the projection screen of the first electronic device in full screen.

23. The method according to claim 19 or 20, characterized in that The second electronic device displays the projection screen of the first electronic device in a non-full screen manner; The method further comprises: The second electronic device sends, to the first electronic device, information about the position of the projected image on the display screen of the second electronic device; The first electronic device receives the position information of the projected image on the display screen of the second electronic device sent by the second electronic device; The first electronic device performing a corresponding operation according to the first instruction includes: When the first electronic device determines, based on the position information of the projected screen on the display screen of the second electronic device, that the user operation corresponding to the first instruction is within the projected screen of the first electronic device, the first electronic device performs the corresponding operation according to the first instruction; When the first electronic device determines, based on the position information of the projection screen on the display screen of the second electronic device, that the user operation corresponding to the first instruction is outside the projection screen of the first electronic device, the first electronic device does not execute the first instruction.

24. The method according to claim 23, wherein The method further comprises: The peripheral device sends the first instruction to the second electronic device; The second electronic device receives the first instruction from the peripheral device; When the second electronic device determines that the user operation corresponding to the first instruction is outside the projection screen of the first electronic device, the second electronic device performs a corresponding operation according to the first instruction; When the second electronic device determines that the user operation corresponding to the first instruction is within the projection screen of the first electronic device, the second electronic device does not execute the first instruction.

25. The method according to claim 19 or 20, characterized in that The second electronic device is wirelessly connected to the peripheral device, and the first electronic device is wirelessly connected to the peripheral device.

26. The method according to claim 19 or 20, characterized in that The second electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi, and the first electronic device is wirelessly connected to the peripheral device via Bluetooth or Wi-Fi.

27. The method according to claim 19 or 20, characterized in that The peripheral device includes any one of the following: a mouse, a keyboard, a stylus and a handle; the first instruction includes any one of the following: movement information of a mouse or a stylus, typing information of a keyboard or a handle; the movement information includes the distance and direction of movement.

28. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor, wherein when the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 1 to 18.

29. A computer-readable storage medium having computer program instructions stored thereon; characterized in that: When the computer program instructions are executed by an electronic device, the electronic device is caused to implement the method according to any one of claims 1 to 18.

Images