Control method, apparatus, and system, and electronic device

Abstract

The present application relates to the technical field of communications, and provides a control method, apparatus, and system, and an electronic device. The method is applied to a first electronic device. The first electronic device establishes a connection with a second electronic device, and a display interface of the first electronic device comprises a display interface region for the second electronic device. The method comprises: receiving a first image sent by the second electronic device; displaying the first image in the display interface region for the second electronic device; receiving a first signal sent by a peripheral of the first electronic device and having an HID function; and when a target of the first signal is the second electronic device, in response to the first signal, sending to the second electronic device a second signal having an HID function, such that the second electronic device, in response to the second signal, performs a corresponding operation on a target interface on which the first image is displayed. The present application enables wide compatibility with various peripherals of the first electronic device and with various second electronic devices.

Description

Control methods, devices and systems, electronic equipment

[0001] This application claims priority to Chinese Patent Application No. 202411226984.5, filed on August 30, 2024, entitled "Control Method, Apparatus and System, Electronic Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of communication technology, and in particular to a control method, apparatus and system, and electronic equipment. Background Technology

[0003] With the development of electronic technology and terminal products, electronic devices can interconnect and communicate in different scenarios, as well as project and share applications between different electronic devices. For example, users can project their mobile phone screen onto a large-screen device (such as a TV) like a smart screen (which can be understood as an intelligent display device). When the first electronic device projects its screen onto a second electronic device, if the first electronic device can be controlled by manipulating the second electronic device, thereby enabling interactive operation between the two devices and connecting their ecosystems, it can provide diversified information presentation and enhance the user experience.

[0004] The related technology provides a method for enabling interaction between two electronic devices, where a peripheral device establishes channels with both the first and second electronic devices. The peripheral device simultaneously sends peripheral events to both the first and second electronic devices, and both electronic devices can receive peripheral movement and selection commands and execute corresponding operations.

[0005] However, the relevant technologies are usually only applicable to sending mouse-type peripheral events, resulting in poor compatibility. Summary of the Invention

[0006] This application provides a control method, apparatus, and system that solves the problem of poor compatibility in related technologies, and is widely compatible with various peripherals of the first electronic device, as well as with various second electronic devices.

[0007] In a first aspect, this application provides a control method applied to a first electronic device, wherein the first electronic device establishes a connection with a second electronic device, and the display interface of the first electronic device includes a display interface area of ​​the second electronic device; the method includes: receiving a first image sent by the second electronic device; displaying the first image in the display interface area of ​​the second electronic device; receiving a first signal with human interface device (HID) function sent by a peripheral device of the first electronic device; and, if the target of the first signal is the second electronic device, responding to the first signal by sending a second signal with HID function to the second electronic device, so that the second electronic device responds to the second signal and performs a corresponding operation on the target interface displaying the first image.

[0008] The first signal is a signal transmitted by the peripheral device of the first electronic device using the HID standard. For example, the first signal could be a first HID event, and the second signal could be a second HID event. For instance, the first HID event could be a touch event, a keyboard event, a mouse event, or a remote control event, etc.

[0009] Its beneficial effect is that by sending HID-enabled signals from a first electronic device to a second electronic device, the second electronic device can respond to commands from the peripherals of the first electronic device, thus realizing the function of controlling one electronic device through its peripherals. Furthermore, the first electronic device uses the HID standard protocol to receive the first signal and send the second signal to the second electronic device. Compared to related technologies, this eliminates the need to establish two channels, ensuring broad compatibility with various peripherals of the first electronic device (including ordinary remote controls, new pointing remote controls, mice, keyboards, touchscreens, and game controllers), as well as with various second electronic devices, including mobile phones and tablets. It also eliminates the need to develop a dedicated communication protocol stack for the second electronic device, reducing its development costs.

[0010] The specifications or peripheral characteristics of the first electronic device and the second electronic device may be the same or different. Correspondingly, the HID formats of the first electronic device and the second electronic device may be the same or different.

[0011] In the case where the HID formats of the first electronic device and the second electronic device are different, in one possible implementation, the process of sending a second signal with HID functionality to the second electronic device in response to a first signal includes: converting the first signal into a second signal conforming to the HID format of the second electronic device in response to the first signal.

[0012] In one possible implementation, a first electronic device establishes a connection with a second electronic device via a HID device. The HID device has a resolution attribute (i.e., the attributes of the HID device include its resolution). If the resolutions of the first and second electronic devices are different, the positions corresponding to the first signal on the display interfaces of the first and second electronic devices will be different. The process of converting the first signal into a second signal conforming to the HID format of the second electronic device includes: calculating a first coordinate based on the target position corresponding to the first signal on the display interface of the first display device, the first coordinate indicating the absolute coordinates in the screen coordinate system of the second electronic device; generating a second signal based on the first coordinate, so that the second electronic device responds to the second signal and performs the operation corresponding to the absolute coordinates in the screen coordinate system of the second electronic device on the target interface.

[0013] The target position corresponding to the first signal on the display interface of the first display device refers to the position where the user operates within the display interface of the first display device. For example, if the first signal is a mouse event, the mouse event carries a coordinate offset, and the coordinates obtained by adding the initial coordinates to the coordinate offset are the target positions. If the first signal is a touch event, the touch event carries absolute coordinates in the screen coordinate system of the first display device, and these absolute coordinates are the target positions.

[0014] As described above, the target position is set with reference to the screen coordinate system of the first electronic device. However, since the screen coordinate systems of the first and second electronic devices are different, the target position cannot be adapted to both screen coordinate systems. This embodiment calculates a first coordinate based on the target position, enabling the second electronic device to determine its absolute coordinates in its screen coordinate system, thereby ensuring the accuracy of control over the second electronic device.

[0015] In one possible implementation, the method further includes: configuring the attributes of the HID device, the attributes of the HID device including the HID device resolution; sending the attributes of the HID device to a second electronic device to cause the second electronic device to respond to the HID device resolution, determining a scaling mapping relationship between the screen resolution of the second electronic device and the HID device resolution; and calculating the first coordinates based on the target position corresponding to the display interface of the first display device according to the first signal, the process of determining the coordinates of the target position relative to the HID device resolution as the first coordinates to cause the second electronic device to respond to the second signal, and determining the absolute coordinates in the screen coordinate system of the second electronic device according to the scaling mapping relationship and the first coordinates.

[0016] The scaling mapping relationship refers to the ratio of the width of the screen resolution of the second electronic device to the width of the HID device resolution, and the ratio of the height of the screen resolution of the second electronic device to the height of the HID device resolution.

[0017] In one possible implementation, the coordinates of the target location relative to the display interface area of ​​the second electronic device can be determined as the first coordinates.

[0018] In one possible implementation, the first signal and the second signal are HID events, and the process of converting the first signal into a second signal conforming to the HID format of the second electronic device includes: converting the event type of the first signal to obtain the second signal.

[0019] For example, assuming the event type of the first signal is a touch event, and the second electronic device does not support touch events but supports mouse events, the first signal can be converted to obtain the second signal with the event type of mouse event.

[0020] In one possible implementation, the method further includes: in response to a first signal, displaying a cursor at the target position corresponding to the first signal in the current display interface of the first electronic device.

[0021] In related technologies, the first electronic device displays cursors drawn by both the first and second electronic devices. This results in two cursors on the first device, requiring the user to switch between them via buttons, leading to low human-computer interaction efficiency. Furthermore, the second electronic device needs to draw the cursor before transmitting the video stream to the first device, resulting in high cursor display latency. Compared to related technologies, the embodiments of this application avoid requiring the user to switch their gaze to operate the peripherals of both electronic devices, improving human-computer interaction efficiency. Additionally, eliminating the video stream projection layer reduces cursor display latency, allowing users to quickly locate the cursor and improving control performance.

[0022] In one possible implementation, the method further includes: obtaining the coordinate range of the display interface area of ​​the second electronic device in the screen coordinate system of the first electronic device; determining that the target of the first signal is the second electronic device when the target position is located in the display interface area of ​​the second electronic device; the target position is the position of the first signal corresponding to the display interface of the first display device.

[0023] In one possible implementation, the signals with HID functionality (including the first signal and the second signal) are transmitted using a target protocol; the target protocol is: Universal Serial Bus (USB) protocol, Inter Integrated Circuit (I2C) protocol, Serial Peripheral Interface (SPI) protocol, Bluetooth protocol, Wireless Fidelity (WiFi) protocol, or High Definition Multimedia Interface (HDMI) protocol.

[0024] In one possible implementation, the method further includes: receiving a second image sent by a second electronic device, the second image being an image displayed on the target interface after the second electronic device performs a corresponding operation on the target interface in response to a second signal; and displaying the second image in the display interface area of ​​the second electronic device.

[0025] Secondly, this application provides a control method applied to a second electronic device, wherein the second electronic device is connected to a first electronic device, and the display interface of the first electronic device includes a display interface area of ​​the second electronic device. The method includes: sending a first image to the first electronic device so that the first electronic device displays the first image in the display interface area of ​​the second electronic device; receiving a second signal sent by the first electronic device in response to a first signal, wherein the first signal and the second signal are signals with HID function; and performing a corresponding operation on a target interface displaying the first image in response to the second signal.

[0026] For example, in response to the second signal, the second electronic device opens an app, or performs an up swipe or a down swipe, etc.

[0027] In one possible implementation, the second electronic device establishes a connection with the HID device in the first electronic device, the HID device having a resolution attribute; the second signal includes a first coordinate, the first coordinate indicating the absolute coordinate in the screen coordinate system of the second electronic device; the process of performing a corresponding operation on the target interface displaying the first image in response to the second signal includes: in response to the first coordinate of the second signal, performing an operation corresponding to the absolute coordinate in the screen coordinate system of the second electronic device on the target interface.

[0028] In one possible implementation, the method further includes: receiving attributes of an HID device, the attributes of which include the HID device resolution; determining a scaling mapping relationship between the screen resolution of the second electronic device and the HID device resolution in response to the HID device resolution; and performing an operation corresponding to the absolute coordinates in the screen coordinate system of the second electronic device on the target interface in response to the first coordinates of the second signal, the process of which includes: determining the absolute coordinates in the screen coordinate system of the second electronic device based on the scaling mapping relationship and the first coordinates in response to the first coordinates of the second signal.

[0029] In one example, the first coordinate is a relative coordinate under the resolution of the HID device, and the second electronic device calculates the product of the scaling mapping relationship and the first coordinate. The absolute coordinate under the screen coordinate system of the second electronic device is the product.

[0030] In another example, the first coordinate is a relative coordinate within the display interface area of ​​the second electronic device. The second electronic device first calculates the relative coordinate of the first coordinate at the HID device resolution. Then, it calculates the product of the scaling mapping relationship and the relative coordinate. The absolute coordinate in the screen coordinate system of the second electronic device is this product.

[0031] In one possible implementation, the process of receiving a second signal sent by the first electronic device in response to the first signal includes: receiving the second signal using a target protocol, such as USB protocol, I2C protocol, SPI protocol, Bluetooth protocol, WiFi protocol, or HDMI protocol.

[0032] In one possible implementation, after performing a corresponding operation on the target interface displaying the first image in response to the second signal, the method further includes: sending a second image to the first electronic device, the second image being the image currently displayed on the target interface.

[0033] Thirdly, this application provides a control device applied to a first electronic device, wherein the first electronic device establishes a connection with a second electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device; the device includes: a transceiver module for receiving a first image sent by the second electronic device; a display module for displaying the first image in the display interface area of ​​the second electronic device; the transceiver module is further configured to receive a first signal with HID function sent by a peripheral device of the first electronic device; the transceiver module is further configured to, in response to the first signal, send a second signal with HID function to the second electronic device when the target of the first signal is the second electronic device, so that the second electronic device responds to the second signal and performs a corresponding operation on the target interface displaying the first image.

[0034] In one possible implementation, the processing module is further configured to, in response to the first signal, convert the first signal into a second signal conforming to the HID format of the second electronic device.

[0035] In one possible implementation, the first electronic device establishes a connection with the second electronic device through an HID device, the HID device having a resolution attribute; the processing module is specifically used to: calculate a first coordinate based on the target position corresponding to the display interface of the first display device according to the first signal, the first coordinate indicating the absolute coordinate in the screen coordinate system of the second electronic device; generate a second signal based on the first coordinate, so that the second electronic device responds to the second signal and performs the operation corresponding to the absolute coordinate in the screen coordinate system of the second electronic device on the target interface.

[0036] In one possible implementation, the processing module is further configured to configure the attributes of the HID device, including the HID device resolution; the transceiver module is further configured to send the attributes of the HID device to the second electronic device so that the second electronic device, in response to the HID device resolution, determines the scaling mapping relationship between the screen resolution of the second electronic device and the HID device resolution; the processing module is specifically configured to determine the coordinates of the target position relative to the HID device resolution as a first coordinate so that the second electronic device, in response to a second signal, determines the absolute coordinates in the screen coordinate system of the second electronic device based on the scaling mapping relationship and the first coordinate.

[0037] In one possible implementation, the first signal and the second signal are HID events, and the processing module is specifically used to convert the event type of the first signal to obtain the second signal.

[0038] In one possible implementation, the display module is further configured to display a cursor at the target position corresponding to the first signal in the current display interface of the first electronic device in response to the first signal.

[0039] In one possible implementation, the processing module is further configured to: obtain the coordinate range of the display interface area of ​​the second electronic device in the screen coordinate system of the first electronic device; determine that the target of the first signal is the second electronic device when the target position is located in the display interface area of ​​the second electronic device; the target position is the position of the first signal corresponding to the display interface of the first display device.

[0040] In one possible implementation, the signals with HID functionality (including the first signal and the second signal) are transmitted using a target protocol; the target protocol is: USB protocol, I2C protocol, SPI protocol, Bluetooth protocol, WiFi protocol, or HDMI protocol.

[0041] In one possible implementation, the transceiver module is further configured to receive a second image sent by the second electronic device, the second image being an image displayed on the target interface after the second electronic device performs a corresponding operation on the target interface in response to the second signal; the display module is further configured to display the second image in the display interface area of ​​the second electronic device.

[0042] Fourthly, this application provides a control device applied to a second electronic device, the second electronic device being connected to a first electronic device, the display interface of the first electronic device including a display interface area of ​​the second electronic device; the device includes: a transceiver module, used to send a first image to the first electronic device so that the first electronic device displays the first image in the display interface area of ​​the second electronic device; the transceiver module is further used to receive a second signal sent by the first electronic device in response to a first signal, the first signal and the second signal being signals with HID function; and a processing module, used to perform a corresponding operation on a target interface displaying the first image in response to the second signal.

[0043] In one possible implementation, the second electronic device establishes a connection with the HID device in the first electronic device, the HID device having a resolution attribute; the second signal includes a first coordinate, the first coordinate indicating the absolute coordinate in the screen coordinate system of the second electronic device; the processing module is specifically used to respond to the first coordinate of the second signal and perform the operation corresponding to the absolute coordinate in the screen coordinate system of the second electronic device on the target interface.

[0044] In one possible implementation, the transceiver module is further configured to receive attributes of the HID device, including the HID device resolution; the processing module is further configured to determine a scaling mapping relationship between the screen resolution of the second electronic device and the HID device resolution in response to the HID device resolution; specifically, the processing module is configured to determine the absolute coordinates of the second electronic device in the screen coordinate system in response to the first coordinate of the second signal, based on the scaling mapping relationship and the first coordinate.

[0045] In one possible implementation, the transceiver module is specifically used to receive a second signal using a target protocol, such as USB, I2C, SPI, Bluetooth, WiFi, or HDMI.

[0046] In one possible implementation, after the target interface displaying the first image performs a corresponding operation in response to the second signal, the transceiver module is further configured to send a second image to the first electronic device, the second image being the image currently displayed on the target interface.

[0047] Fifthly, this application provides an electronic device, corresponding to a first electronic device, wherein the first electronic device is connected to a second electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device; the first electronic device includes: an HID device; one or more processors; and a memory for storing one or more computer programs or instructions; when one or more computer programs or instructions are executed by the first electronic device, the first electronic device performs the following steps: receiving a first image sent by the second electronic device; displaying the first image in the display interface area of ​​the second electronic device; receiving a first signal with HID function sent by a peripheral device of the first electronic device; and, if the target of the first signal is the second electronic device, sending a second signal with HID function to the second electronic device in response to the first signal, so that the second electronic device performs a corresponding operation on the target interface displaying the first image in response to the second signal.

[0048] In one possible implementation, the HID device uses the USB protocol, I2C protocol, SPI protocol, Bluetooth protocol, WiFi protocol, or HDMI protocol.

[0049] In one possible implementation, when the one or more computer programs or instructions are executed by the first electronic device, the first electronic device performs the method as described in any one of the first aspects.

[0050] In a sixth aspect, this application provides an electronic device, corresponding to a second electronic device, the device comprising: one or more processors; a memory for storing one or more computer programs or instructions; and, when the one or more computer programs or instructions are executed by the one or more processors, causing the one or more processors to implement the method as described in any one of the second aspects.

[0051] In a seventh aspect, this application provides a control system, which includes a first electronic device, a second electronic device, and a peripheral device of the first electronic device. The first electronic device is connected to the second electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device. The peripheral device of the first electronic device is used to send a first HID event to the first electronic device. The first electronic device is used to perform the method as described in any one of the first aspects, and the second electronic device is used to perform the method as described in any one of the second aspects.

[0052] Eighthly, this application provides a chip comprising: a processing circuit and an interface circuit; wherein the interface circuit is used to couple with an external memory and provide a communication interface for the processing circuit to access the memory; the processing circuit is used to execute program instructions in the memory to implement the method as described in any one of the first to second aspects.

[0053] Ninthly, this application provides a computer-readable storage medium storing program code, which, when executed by a processor, implements the method as described in any one of the first to second aspects.

[0054] In a tenth aspect, this application provides a computer program product comprising instructions, characterized in that, when the instructions are executed by a computing device, the computing device performs the method as described in any one of the first to second aspects. Attached Figure Description

[0055] Figure 1 is a block diagram of a control system provided in an embodiment of this application;

[0056] Figure 2 is a schematic diagram of an application scenario of the control method provided in an embodiment of this application;

[0057] Figure 3 is a schematic diagram of another application scenario of the control method provided in the embodiments of this application;

[0058] Figure 4 is a block diagram of another control system provided in an embodiment of this application;

[0059] Figure 5 is a flowchart illustrating a control method provided in an embodiment of this application;

[0060] Figure 6 is a schematic diagram of the display interface of a first electronic device provided in an embodiment of this application;

[0061] Figure 7 is a flowchart illustrating another control method provided in an embodiment of this application;

[0062] Figure 8 is a flowchart illustrating another control method provided in an embodiment of this application;

[0063] Figure 9 is a schematic diagram of a second image provided in an embodiment of this application;

[0064] Figure 10 is a schematic diagram of another second image provided in an embodiment of this application;

[0065] Figure 11 is a flowchart of a control method provided in an embodiment of this application;

[0066] Figure 12 is a flowchart of another control method provided in an embodiment of this application;

[0067] Figure 13 is a block diagram of a control device provided in an embodiment of this application;

[0068] Figure 14 is a block diagram of another control device provided in an embodiment of this application;

[0069] Figure 15 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0070] Figure 16 is a schematic diagram of the structure of a control device provided in an embodiment of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0073] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0074] This application provides a control method in which the display interface of a second electronic device is projected onto a first electronic device. The interaction between the peripheral device of the first electronic device and the first electronic device, as well as between the first electronic device and the second electronic device, utilizes the HID protocol. The first electronic device can control the operation of the second electronic device by forwarding HID-enabled signals from its peripheral device to the second electronic device. In this application embodiment, the HID-enabled signal can be an HID event.

[0075] Peripherals can include remote controls (such as regular and smart remote controls), mice (such as air mice), keyboards, gamepads, touchscreens, or styluses. Smart remote controls, for example, can be pointing remote controls, which can use the Service Location Protocol (SLP) to accurately point at and hit a target by reporting remote control touch events. Users can perform actions such as skipping, swiping, and dragging using pointing remote controls. Here, a remote control touch event refers to the remote control sending operation commands to the target in the semantics of touch events.

[0076] Electronic devices can include mobile phones, cellular phones, smartphones, tablets, styluses, routers, cameras, screens, smart screens, wireless data cards, personal digital assistants (PDAs), personal computers (PCs), wireless modems, handsets, laptop computers, smartwatches, smart bracelets, electronic whiteboards, machine-type communication (MTC) terminals, computers with wireless transceiver capabilities, virtual reality (VR) terminals, augmented reality (AR) terminals, workshop equipment, wireless terminals in autonomous driving, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, and wireless terminals in smart homes. Wireless terminals, vehicle-mounted terminals, vehicle-mounted screens, projectors, set-top boxes, roadside units (RSUs) with terminal functions, and flying equipment (e.g., intelligent robots, hot air balloons, drones, airplanes) are examples of such devices. This application does not limit the specific type of electronic equipment described in its embodiments.

[0077] This control method can be applied in a control system, which includes a first electronic device, a second electronic device, and peripherals of the first electronic device. The first electronic device and the second electronic device are connected, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device. The peripherals of the first electronic device can interact with the second electronic device.

[0078] For example, please refer to Figure 1, which is a block diagram of a control system provided in an embodiment of this application. The first electronic device 110 in the control system 100 includes a display screen 111, a media terminal 112, and a controller 113. The display screen 111 displays the display interface area of ​​the second electronic device 120. The controller 113 is connected to a peripheral device 130 of the first electronic device, and the peripheral device 130 controls the display of the television screen by interacting with the controller 113. The media terminal 112 is connected to the second electronic device 120, and the second electronic device 120 transmits a video stream to the first electronic device 110 through the media terminal 112, thereby realizing screen projection on the display screen 111.

[0079] The following describes the application scenarios of the embodiments of this application. Please refer to Figure 2, which is a schematic diagram of an application scenario of the control method provided in the embodiments of this application. Figure 2 uses a computer as the first electronic device, a mobile phone as the second electronic device, and a remote control as the peripheral device of the computer for illustration. The mobile phone's current display interface is the main interface of the phone, which includes icons for multiple applications (APPs). The main interface includes a camera APP icon controlled by the remote control. Based on the user's click operation, the remote control sends a signal with HID function for the camera APP to the TV, and the TV forwards the signal to the mobile phone. The mobile phone, in response to the received signal for the camera APP, opens the camera.

[0080] Please refer to Figure 3, which is a schematic diagram of another application scenario of the control method provided in this application embodiment. Figure 3 illustrates the method using a personal computer as the first electronic device and a mobile phone as the second electronic device. The mobile phone's current display interface is the main screen including multiple app icons. The keyboard sends a signal with HID functionality to the personal computer based on the user's pull-down operation, and the personal computer forwards the signal to the mobile phone. Upon receiving the signal, the mobile phone displays a pull-down notification bar.

[0081] Figures 2 and 3 illustrate scenarios of controlling a screen-casting mobile phone through a TV or a personal computer peripheral. It is understood that the embodiments of this application are not limited to these two scenarios. For example, they can also be applied to various screen-casting scenarios such as controlling a screen-casting mobile phone through a projector peripheral and controlling a screen-casting tablet through a personal computer peripheral.

[0082] In this embodiment, the process of sending a signal with HID functionality to a second electronic device can be performed by an HID device. HID is a standard input protocol, and HID devices use the HID standard for communication. HID devices may include: pens (e.g., styluses), touchscreens, touchpads, mice, keyboards, remote controls, and gamepads, etc.

[0083] For example, please refer to Figure 4, which is a block diagram of another control system provided in an embodiment of this application. Based on Figure 1, the first electronic device 110 further includes an HID path module 114, which is connected to the second electronic device 120 and serves as an input device for the second electronic device 120, capable of supporting multiple physical channels and multiple logical channels. The second electronic device 120 recognizes the HID path module 114 as a standard HID device. Furthermore, Figure 4 does not show a display screen 111, but also shows the processor 115 of the first electronic device.

[0084] In one possible implementation, the HID path module 114 can employ USB, I2C, SPI, Bluetooth, WiFi, or HDMI protocols. Correspondingly, the second electronic device 120 uses a corresponding interface as its peripheral interface for connection to the HID path module 114. For example, when the HID path module 114 uses the USB protocol, the second electronic device 120 uses a USB OTG (USB on the go) interface as its peripheral interface. The second electronic device 120 treats the HID path module 114 as a standard USB HID device, and the two communicate via the USB bus, enabling a high transmission rate.

[0085] Since the HID path module 114 adopts the HID standard, it is compatible with various types of second electronic devices. Therefore, there is no need to adapt and develop a separate communication driver on the second electronic device, that is, no additional software stack is required.

[0086] The first electronic device 110 and its peripheral device 130 also communicate using protocols such as USB, I2C, SPI, Bluetooth, WiFi, or HDMI.

[0087] In one possible implementation, as shown in Figure 4, the first electronic device 110 may further include an HID event forwarding module 116. The functions that the HID event forwarding module 116 can perform include: parsing the received signal with HID function (e.g., HID event), signal conversion (e.g., HID event conversion), and forwarding the received signal to the HID path module 114, etc.

[0088] In this embodiment, the attributes of the HID path module 114 can be set according to user settings or hardware default configuration, and this process can be executed by the HID event forwarding module 116. For example, the attributes of the HID path module 114 may include: HID specification, HID device type, etc.

[0089] It should be noted that the division of the HID event forwarding module 116 and HID path module 114 shown in Figure 4 represents only a logical component division and does not constrain the physical implementation location. In specific implementations, these two components can be implemented in different chips or in the same chip; they can be implemented in hardware, software, or a combination of both; they can be implemented in the kernel mode or the user mode of the software. The HID path module 114 can be built into the first electronic device or externally placed in the first electronic device. For example, the HID event forwarding module 116 and the HID path module can be combined into a single circuit module at the chip level to achieve software-hardware integration, while avoiding interference from the operating system (OS) scheduling. This can improve the performance of operations based on HID events, and the application and ecosystem are unaware of this, effectively ensuring a high return on investment for HID. This application does not limit the specific implementation method of the HID event forwarding module 116 and the HID path module 114.

[0090] The modules shown in Figure 4 are merely illustrative examples. In a specific implementation, there may also be corresponding supporting circuitry devices and self-integrated standard software, or fewer modules than those shown in Figure 4. This application does not limit the scope of the implementation.

[0091] Please refer to Figure 5, which is a flowchart illustrating a control method provided in an embodiment of this application. This method can be applied, for example, to the aforementioned first electronic device. The first electronic device establishes a connection with a second electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device. The method may include the following processes:

[0092] 201. Receive the first image sent by the second electronic device.

[0093] The first electronic device can receive the first image via the media terminal shown in Figure 1 or Figure 4. For example, if the first electronic device is a television, the first image can be received via a video interface, such as a USB displayport alt interface.

[0094] The first image can be an image displayed on the virtual interface of the second electronic device, or it can be an image displayed on the current display interface of the second electronic device. This application embodiment does not limit this. That is, the first image does not have to be an image of the current display interface of the second electronic device. For example, if the image of the current display interface of the second electronic device is a camera page, and the image of the virtual interface of the second electronic device is a playback page of a video application, then the first image can be either the camera page or the playback page of that video application.

[0095] 202. Display the first image in the display interface area of ​​the second electronic device.

[0096] For example, please refer to Figure 6, which is a schematic diagram of the display interface of a first electronic device provided in an embodiment of this application. The white area in Figure 6 represents the display interface area of ​​the second electronic device, and the black area represents the black border of the display interface area of ​​the second electronic device. The first image will be displayed in the white area of ​​Figure 6.

[0097] 203. Receive a first signal with HID function sent by the peripheral device of the first electronic device.

[0098] The first signal is a signal transmitted by the peripheral device of the first electronic device using the HID standard. For example, the first signal can be a first HID event. For instance, the first HID event can be a touch event, a keyboard event, a mouse event, or a remote control event, etc.

[0099] In this embodiment, the number of first signals can be one or more. For example, for a user's click operation, the peripheral device of the first electronic device will send one first signal. For a user's swipe operation (e.g., swipe up, swipe down, swipe left, or swipe right), the peripheral device of the first electronic device will send multiple first signals. Specifically, a touch start event is triggered when the user begins to touch, a touch move event is triggered when the user swipes, and a touch end event is triggered when the user stops touching. The touch start event can carry a coordinate corresponding to the touch start operation, the touch move event can carry multiple coordinates corresponding to the swipe operation, and the touch end event can carry a coordinate corresponding to the touch end operation.

[0100] 204. If the target of the first signal is the second electronic device, in response to the first signal, a second signal with HID function is sent to the second electronic device so that the second electronic device responds to the second signal and performs a corresponding operation on the target interface displaying the first image.

[0101] When there are multiple first signals, there are also multiple second signals.

[0102] The first electronic device analyzes the first signal and determines whether the target of the first signal is the second electronic device. For example, the first electronic device can obtain the coordinate range of the display interface area of ​​the second electronic device in its screen coordinate system. Then, if the target location is within the display interface area of ​​the second electronic device, the target of the first signal is determined to be the second electronic device. If the target location is not within the display interface area of ​​the second electronic device, the target of the first signal is determined to be the first electronic device.

[0103] The target position refers to the location corresponding to the first signal on the display interface of the first display device, specifically the position where the user operates within the display interface. For example, if the first signal is a mouse event, the mouse event carries a coordinate offset; adding the initial coordinates to the coordinate offset yields the target position. If the first signal is a touch event, the touch event carries absolute coordinates in the screen coordinate system of the first display device; these absolute coordinates are the target position.

[0104] For example, the peripheral device of the first electronic device is a smart remote control. The smart remote control can use HID touch screen semantics, so the first signal it sends is a touch event semantic. The touch event semantic includes the absolute coordinates in the screen coordinate system of the first display device. The first electronic device can obtain the absolute coordinates in the screen coordinate system of the first display device by capturing the ABS_X and ABS_Y events in the touch event semantic.

[0105] To obtain the coordinate range of the display interface area of ​​the second electronic device, the first electronic device can perform window recognition and black border recognition of the second electronic device to obtain the starting point and ending point of the display interface area of ​​the second electronic device. The starting point is marked as the origin (0, 0), and the offset (Δpx, Δpy) of the ending point relative to the starting point is calculated. The coordinates of the starting point and ending point in the screen coordinate system of the first television device are also calculated. As shown in Figure 6 above, the starting point of the display interface area of ​​the second electronic device identified by the first electronic device is P(0, 0), and the ending point is ΔP(Δpx, Δpy).

[0106] In one example, an object detection algorithm can be used for window recognition to identify areas on the display interface of the first electronic device that are not used to display the image of the first electronic device (including the display interface area of ​​the second electronic device and the surrounding black borders). The object detection algorithm can be a Faster Region Based Convolutional Neural Network (Faster R-CNN), YOLO (You Only Look Once), or a Single Shot MultiBox Detector (SSD). The object detection algorithm identifies different windows on the display interface through multiple training iterations. The training process includes: acquiring a large number of screenshots, marking the positions of windows in the acquired screenshots, and then training the model using screenshots after the standard window. The trained model can then recognize different windows on the display interface.

[0107] In another example, a computer vision-based target recognition method can be used to obtain the layout tree information of the display interface by accessing a specific software development kit (SDK), thereby obtaining the position and type of each control on the display interface and thus achieving window recognition.

[0108] For black border recognition, for example, an image edge detection algorithm can be used to identify the edges of the display interface area of ​​the second electronic device from the area determined by window recognition, thereby determining the display interface area of ​​the second electronic device. Image edge detection algorithms can acquire locations where pixel values ​​change rapidly within the area determined by window recognition; these locations represent the boundaries between different areas or the contours of objects within the area determined by window recognition. For example, the Sobel operator can be used for image edge detection. The Sobel operator uses gradient information of pixels within the area determined by window recognition to detect edges.

[0109] As shown in Figure 6, if the target position is within P(0,0)-ΔP(Δpx,Δpy), then the target of the first signal is the second electronic device; if the target position is not within P(0,0)-ΔP(Δpx,Δpy), then the target of the first signal is the first electronic device.

[0110] In one possible implementation, the first electronic device can send a second signal to the second electronic device using a target protocol. The target protocol can be: USB, I2C, SPI, Bluetooth, WiFi, or HDMI.

[0111] In one possible implementation, regardless of whether the target of the first signal is a first electronic device or a second electronic device, the first electronic device can respond to the first signal by drawing and displaying a cursor at the target location on its current display screen. For example, the cursor can be seen in Figure 6. Thus, when the first electronic device receives the first signal, the user can immediately see the cursor's position. Although the cursor belongs to the display screen area of ​​the second electronic device, since the second electronic device's display screen area is located within the first electronic device's display screen, the user cannot distinguish which electronic device drew the cursor. This allows the peripheral to automatically switch between the display screen areas of the first and second electronic devices based on the cursor position without the user noticing.

[0112] In related technologies, the first electronic device displays cursors drawn by both the first and second electronic devices. This results in two cursors on the first device, requiring the user to switch between them via buttons, leading to low human-computer interaction efficiency. Furthermore, the second electronic device needs to draw the cursor before transmitting the video stream to the first device, resulting in high cursor display latency. Compared to related technologies, this application's embodiment avoids the need for the user to switch their gaze to operate the peripherals of both electronic devices, improving human-computer interaction efficiency. It also eliminates the video stream projection layer, reducing cursor display latency and allowing the user to quickly locate the cursor, thus improving display performance.

[0113] Please refer to Figure 7, which is a flowchart illustrating another control method provided in an embodiment of this application. This method can be applied, for example, to the aforementioned second electronic device. The second electronic device establishes a connection with the first electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device. The method may include the following processes:

[0114] 301. Send a first image to the first electronic device so that the first electronic device displays the first image in the display interface area of ​​the second electronic device.

[0115] The second electronic device transmits the first image through the media terminal of the first electronic device, thereby enabling the second electronic device to project its image onto the first electronic device. For example, the first image can be transmitted via a USB DisplayPort Alt interface.

[0116] 302. Receive a second signal sent by the first electronic device in response to the first signal, wherein the first signal and the second signal are signals with HID function.

[0117] The first signal can be the first HID event, and the second signal can be the second HID event.

[0118] In process 204, the first electronic device can send a second signal to the second electronic device using the target protocol, and correspondingly, the second electronic device can also receive the second signal using the target protocol. The target protocol can be: USB protocol, I2C protocol, SPI protocol, Bluetooth protocol, WiFi protocol, or HDMI protocol.

[0119] 303. In response to the second signal, perform the corresponding operation on the target interface displaying the first image.

[0120] Referring to process 203 above, if there is one first signal and one second signal, the second electronic device directly responds to this single second signal to perform the corresponding operation. For example, if the second signal is a click event, the second electronic device responds to this second signal by opening a specific app.

[0121] If there are multiple first signals and multiple second signals, the second electronic device can perform a final operation based on the multiple second signals. For example, the multiple second signals include a touch start event, a touch move event, and a touch end event. The second electronic device responds to these three second signals and performs the corresponding swipe operation.

[0122] In this embodiment, the specifications or peripheral characteristics of the first electronic device and the second electronic device may be the same or different. Correspondingly, the HID formats of the first electronic device and the second electronic device may be the same or different. The HID format may include, for example, supported HID modes (mouse, keyboard, touchscreen, touch tablet, stylus, etc.) and HID parameters (including the reporting rate of HID events, the resolution of touch events, and upper limits for various parameters, etc.). When the HID formats of the first electronic device and the second electronic device are the same, the second signal is the same as the first signal; that is, the first electronic device directly forwards the signal sent by the peripheral device to the second electronic device.

[0123] When the HID formats of the first electronic device and the second electronic device are different, the first signal needs to be converted into a second signal that conforms to the HID format of the second electronic device. This process can be performed by the HID event forwarding module 116 in the control system. For example, if the HID device has a resolution attribute (i.e., the attribute of the HID device includes the HID device resolution), and if the resolutions of the first electronic device and the second electronic device are different, the corresponding positions of the first signal on the display interfaces of the first electronic device and the second electronic device will be different. Therefore, if the target of the first signal is the second electronic device, a first coordinate can be calculated based on the target position of the first signal on the display interface of the first display device. The first coordinate indicates the absolute coordinates in the screen coordinate system of the second electronic device. Then, the second signal is generated based on the first coordinate. HID devices with resolution attributes may include, for example, touch screen devices, stylus devices, and touch tablet devices.

[0124] As another example, if the first electronic device and the second electronic device support different event types, the event type of the first signal needs to be converted to obtain the second signal. For example, assuming the first signal is a touch event, and the second electronic device does not support touch events but supports mouse events, the first signal can be converted into a second signal with the event type of mouse event.

[0125] As another example, when the first signal is a keyboard event, if the key value definitions of the first electronic device and the second electronic device are different, then the key value in the first signal needs to be converted to the key value defined by the second electronic device. For example, if the key value of the return key defined by the first electronic device is K1, but the key value of the return key defined by the second electronic device is not K1, then K1 can be converted to the key value of the return key defined by the second electronic device.

[0126] As another example, if the operating systems of the first electronic device and the second electronic device are different, then the first signal needs to be converted into a second signal that conforms to the operating system of the second electronic device.

[0127] The following describes a control method requiring coordinate transformation. For example, please refer to Figure 8, which is a flowchart illustrating another control method provided in an embodiment of this application. This method can be applied, for example, to a control system. A first electronic device establishes a connection with a second electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device. The method is illustrated using an HID device with resolution attributes as an example. The method may include the following steps:

[0128] 401. The second electronic device sends the first image to the first electronic device.

[0129] This process can refer to the aforementioned processes 201 and 301, and will not be repeated here in the embodiments of this application.

[0130] 402. The first electronic device displays the first image in the display interface area of ​​the second electronic device.

[0131] This process can be referred to as the aforementioned process 202, and will not be repeated here in the embodiments of this application.

[0132] 403. The first electronic device receives a first signal with HID function sent by a peripheral device.

[0133] This process can be referred to as the aforementioned process 203, and will not be repeated here in the embodiments of this application.

[0134] 404. When the target of the first signal is the second electronic device, the first electronic device responds to the first signal and calculates the first coordinates according to the target position corresponding to the first signal on the display interface of the first display device. The first coordinates indicate the absolute coordinates in the screen coordinate system of the second electronic device.

[0135] In one example, the first signal includes the coordinates of the target location (i.e., the absolute coordinates of the target location in the screen coordinate system of the first electronic device), and the first electronic device can directly obtain the coordinates of the target location from the first signal. For example, if the first signal is a touch event, the first electronic device can obtain the coordinates of the target location by capturing the ABS_X and ABS_Y events in the touch event semantics.

[0136] In another example, the first signal does not include the coordinates of the target position. The first electronic device can calculate the coordinates of the target position based on other information carried by the first signal. For example, the first signal is a mouse event, which carries a mouse offset. The first electronic device calculates the coordinates of the target position based on the mouse offset and the initial position.

[0137] To enable HID standard communication with the second electronic device, it is necessary to configure the attributes of the module in the first electronic device that communicates with the second electronic device, and send these attributes to the second electronic device so that the first electronic device recognizes the module as an HID device. Referring to the aforementioned description of the control system, the HID path module 114 is used to communicate with the second electronic device, therefore, the attributes of the HID path module 114 need to be configured.

[0138] HID devices have resolution attributes (i.e., the attributes of an HID device can include its resolution). A second electronic device can, in response to the HID device resolution, determine a scaling mapping relationship between its screen resolution and the HID device resolution. This scaling mapping relationship refers to the ratio of the width of the second electronic device's screen resolution to the width of the HID device's screen resolution, and the ratio of the height of the second electronic device's screen resolution to the height of the HID device's screen resolution.

[0139] In one possible implementation, when performing coordinate transformation, the first electronic device determines the coordinates of the target position relative to the HID device resolution as the first coordinates. For example, the coordinates of the target position relative to the HID device resolution are determined as the first coordinates.

[0140] The resolution of the HID device can be the same as the size of the display interface area of ​​the second electronic device within the first electronic device. Referring to the display interface area of ​​the second electronic device shown in Figure 6, the resolution of the HID device can be ΔP(Δpx*Δpy). This simplifies the coordinate transformation process of the first electronic device, and also simplifies the process of the second electronic device determining its absolute coordinates in its screen coordinate system based on the target position. For example, if the first electronic device calculates the offset of the target position relative to point P0 in its screen coordinate system as ΔT(Δtx*Δty), then the first coordinate can be ΔT(Δtx, Δty).

[0141] Of course, the first electronic device can also arbitrarily set the HID device resolution, for example, it can be a fixed value such as M*(Δpx, Δpy) or (32556, 32556). This provides high flexibility in setting the HID device resolution and good compatibility with various control systems and APPs. For example, the HID device resolution is set to other values ​​(rx, ry). The first electronic device first calculates the offset ΔT(Δtx*Δty) of the target position relative to point P0 in the screen coordinate system of the first electronic device. Then it calculates the coordinates of ΔT(Δtx*Δty) relative to (rx, ry) as ((Δtx*rx) / Δpx, (Δty*ry) / Δpy)), and the first coordinate is ((Δtx*rx) / Δpx, (Δty*ry) / Δpy)).

[0142] In another possible implementation, when performing coordinate transformation, the first electronic device can determine the coordinates of the target position relative to the display interface area of ​​the second electronic device as the first coordinate. For example, the first coordinate is determined as the coordinates of the target position relative to the display interface area of ​​the second electronic device, meaning the first coordinate is independent of the HID device resolution. For example, referring to the display interface area ΔP(Δpx*Δpy) of the second electronic device shown in Figure 6, the first electronic device calculates the offset ΔT(Δtx*Δty) of the target position relative to point P0 in the screen coordinate system of the first electronic device, and the first coordinate is ΔT(Δtx, Δty).

[0143] HID attributes can also include an HID device type report descriptor.

[0144] It should be noted that if there are multiple first signals, a first coordinate needs to be calculated according to process 404 based on the target position corresponding to each first signal in the display interface of the first electronic device. If a first signal corresponds to multiple target positions in the display interface of the first electronic device (for example, a touch move event carries multiple coordinates), then multiple first coordinates need to be calculated according to process 404.

[0145] 405. The first electronic device generates a second signal based on the first coordinate.

[0146] 406. The first electronic device sends a second signal to the second electronic device.

[0147] 407. The second electronic device responds to the first coordinate in the second signal and performs an operation corresponding to the absolute coordinates in the screen coordinate system of the second electronic device on the target interface displaying the first image.

[0148] The second electronic device responds to the second signal and determines its absolute coordinates in the screen coordinate system based on the scaling mapping relationship and the first coordinates. Then, it executes the operation corresponding to the absolute coordinates in the screen coordinate system of the second electronic device on the target interface.

[0149] Referring to the aforementioned process 404, in one possible implementation, the first coordinate is a relative coordinate under the HID device resolution, the second electronic device calculates the product of the scaling mapping relationship and the first coordinate, and the absolute coordinate under the screen coordinate system of the second electronic device is the product.

[0150] In one example, the HID device has a resolution of ΔP and a first coordinate of ΔT(Δtx, Δty). The second electronic device calculates its absolute coordinates in the screen coordinate system as (xscale*Δtx, yscale*Δty). xscale represents the ratio of the width of the second electronic device's resolution to the width of the HID device's resolution, and yscale represents the ratio of the height of the second electronic device's resolution to the height of the HID device's resolution.

[0151] In another example, the HID device has a resolution of other settings (rx, ry), and its first coordinates are ((Δtx*rx) / Δpx, (Δty*ry) / Δpy). The second electronic device calculates its absolute coordinates in the screen coordinate system as ((xscale*Δtx*rx) / Δpx, (yscale*Δty*ry) / Δpy).

[0152] In another possible implementation, the first coordinate is a relative coordinate within the display interface area of ​​the second electronic device. The second electronic device first calculates the relative coordinate of the first coordinate under the HID device resolution (as described above, the first electronic device sends the HID attribute to the second electronic device, so the second electronic device can obtain the HID device resolution). Then, it calculates the product of the scaling mapping relationship and the relative coordinate, and the absolute coordinate in the screen coordinate system of the second electronic device is the product.

[0153] In one example, the HID device resolution is ΔP. ​​Since the HID device resolution is the same as ΔP, the relative coordinates of the first coordinate in the HID device resolution calculated by the second electronic device are still ΔT(Δtx, Δty). The absolute coordinates of the second electronic device in the screen coordinate system are calculated as (xscale*Δtx, yscale*Δty).

[0154] In another example, if the HID device resolution is set to other values ​​(rx, ry), the second electronic device calculates the relative coordinates of the first coordinate under the HID device resolution as ((Δtx*rx) / Δpx, (Δty*ry) / Δpy)). Then, the second electronic device calculates its absolute coordinates in its screen coordinate system as ((xscale*Δtx*rx) / Δpx, (yscale*Δty*ry) / Δpy).

[0155] 408. The second electronic device sends a second image to the first electronic device. The second image is the image currently displayed on the target interface.

[0156] The second electronic device can send a second image through the media terminal of the first electronic device, enabling screen mirroring from the second electronic device to the first electronic device. For example, the second image can be sent via a USB DisplayPort Alt interface.

[0157] 409. The first electronic device displays the second image in the display interface area of ​​the second electronic device.

[0158] For example, please refer to Figure 9, which is a schematic diagram of a second image provided in an embodiment of this application. Figure 9 illustrates the application scenario shown in Figure 2. As shown in Figure 9, the mobile phone opens the camera in response to the received HID event and displays the photo-taking interface (i.e., the second image), and the display interface area of ​​the second electronic device also displays the photo-taking interface.

[0159] As another example, please refer to Figure 10, which is a schematic diagram of another second image provided in an embodiment of this application. Figure 10 is illustrated using the application scenario shown in Figure 3 as an example. As shown in Figure 10, the mobile phone responds to the received HID event by swiping down the notification bar and displays the swiping notification bar interface (i.e., the second image). The display interface area of ​​the second electronic device also displays the swiping notification bar interface.

[0160] The following description further illustrates the entire method in conjunction with the structure of the control system, using the example of a first signal representing a first HID event, a second signal representing a second HID event, and a television as the first electronic device and a mobile phone as the second electronic device. Please refer to Figure 11, which is a flowchart of a control method provided in an embodiment of this application. Figure 11 shows the television's peripherals, the television itself, and the mobile phone. The television includes an HID event forwarding module and an HID path module. The HID event forwarding module can perform the judgment, conversion, and forwarding of peripheral HID events.

[0161] As shown in Figure 11, the HID event forwarding module first executes the configuration process: 1. Setting the attributes of the HID path module, such as HID specifications and HID device type descriptors; 2. The TV uses software / hardware to create the corresponding HID device, and the attributes of the HID device are reported to the mobile phone; 3. When the mobile phone screen changes, the TV recalculates the coordinate range of the mobile phone's display interface area, and recreates and declares the HID device based on this coordinate range, triggering the mobile phone's OS to automatically refresh the HID device instance. The HID path module is configured as / dev / hidgX and supports multiple physical and logical channels.

[0162] The open-source standard HID driver on the mobile phone side obtains the / dev / hidgX declared by the TV during the enumeration process, creates the / dev / hidrawX device on the mobile phone side, and learns that this device is an HID device by reading the configuration information reported by the HID path module, and creates the corresponding input device. This allows it to communicate with the HID path module using the HID standard.

[0163] After configuration, the mobile phone will project its display interface onto the TV's display interface via the video interface. Upon receiving the first HID event sent by the peripheral device, the TV first draws and displays the cursor. Then, it determines whether the target of the first HID event is the mobile phone. As shown in Figure 11, this determination process can be executed by the HID event forwarding module.

[0164] If the target of the first HID event is a mobile phone, the HID event forwarding module converts the first HID event into a second HID event conforming to the mobile phone's HID format, and then sends the second HID event to the HID path module. Specifically, the HID event forwarding module sends the second HID event to the HID path module through the corresponding physical or logical channel, based on the nature of the second HID event or user settings. The HID path module then sends the received second HID event to the mobile phone.

[0165] The mobile phone receives the second HID event and performs the corresponding operation in response. It then sends the resulting image to the television via the video interface. The television processes and displays the resulting image.

[0166] Please refer to Figure 12, which is a flowchart of another control method provided in an embodiment of this application. Figure 12 shows a television peripheral, a television, and a mobile phone. The television includes an HID event forwarding module and an HID path module. The HID event forwarding module can perform the judgment, conversion, and forwarding of peripheral HID events. Figure 12 illustrates the method using a first HID event as a touch event and the HID path module configured as a USB HID touchscreen device as an example.

[0167] The TV's processor's USB port supports being set to USB OTG device mode, allowing the HID path module to be configured as a USB HID touchscreen device. The HID path module supports converting mobile phone video streams to DP input for the TV's processor.

[0168] As shown in Figure 12, the HID event forwarding module first executes the configuration process: 1. Set the attributes of the USB HID touchscreen device, such as HID device resolution and HID device type; 2. The TV uses software / hardware to create the corresponding USB HID touchscreen device, and the attributes of the USB HID touchscreen device are reported to the mobile phone; 3. When the mobile phone screen changes, the TV recalculates the coordinate range of the mobile phone's display interface area, and recreates and declares the USB HID touchscreen device based on this coordinate range. The OS on the mobile phone side automatically refreshes the USB HID touchscreen device instance. The HID path module is configured as USB HID touchscreen device / dev / hidg0, supporting multiple physical and logical channels.

[0169] The open-source USB HID driver on the mobile phone's USB OTG interface obtains the / dev / hidg0 declared by the TV during the enumeration process, creates the / dev / hidraw0 device on the mobile phone side, and learns that this device is a USB HID touchscreen device by reading the configuration information reported by the HID path module, and creates the corresponding input device. This allows communication with the HID path module via a USB cable.

[0170] The mobile OS reads the HID device resolution from the HID attributes according to the HID standard and calculates the scaling mapping relationship (xscale, yscale) between the resolution of the USB HID touchscreen device (i.e., the HID device resolution) and the mobile phone screen resolution. The scaling mapping relationship is the ratio of the mobile phone screen resolution to the resolution of the USB HID touchscreen device.

[0171] After configuration, the mobile phone will project its display onto the TV's screen via the USB DisplayPort Alt interface. The peripheral device can report a first HID event using HID touchscreen device semantics, which includes the first coordinate T(x, y). After receiving the first HID event from the peripheral device, the TV can obtain the first coordinate by capturing the ABS_X and ABS_Y events within the first HID event.

[0172] The TV first draws and displays the cursor at the first coordinate T(x, y), allowing the user to quickly locate the cursor. Next, the TV determines whether the first coordinate T(x, y) falls within the display area of ​​the phone (e.g., the area formed by P and ΔP in Figure 6), thus determining whether the target of the first HID event is the TV or the phone. In other words, it determines whether the user is operating the TV or the phone using a peripheral device. As shown in Figure 12, this determination process can be executed by the HID event forwarding module.

[0173] If the target of the first HID event is a mobile phone, the HID event forwarding module calculates the relative coordinates (i.e., the first coordinates) of the second coordinate T(x, y) under the HID device resolution. This calculation process can be referred to the aforementioned process 404, and will not be repeated here in this embodiment. The HID event forwarding module can also convert the first HID event into a second HID event that conforms to the mobile phone, and the second HID event includes the first coordinates.

[0174] The HID event forwarding module sends a second HID event to the HID path module. Specifically, based on the nature of the second HID event or user settings, the HID event forwarding module sends the second HID event to the HID path module through the corresponding physical or logical channel. The HID path module then sends the received second HID event to the mobile phone.

[0175] The mobile phone receives the second HID event, parses it to obtain the first coordinate, calculates the product of the scaling mapping relationship (xscale, yscale) and the first coordinate, and obtains the absolute coordinate in the mobile phone screen coordinate system. This absolute coordinate is the coordinate point where the user wants to operate. The mobile phone performs the corresponding operation at this absolute coordinate on the current display screen. Then, it sends the resulting image to the TV via the USB DisplayPort Alt interface. The TV processes and displays the resulting image.

[0176] In summary, the control method provided in this application involves a second electronic device sending a first image to a first electronic device, the first electronic device displaying the first image on the display interface area of ​​the second electronic device, the first electronic device receiving a first signal with HID functionality sent by a peripheral device, and, in response to the first signal being the second electronic device, sending a second signal with HID functionality to the second electronic device. The second electronic device, in response to the second signal, performs a corresponding operation on the target interface displaying the first image. By sending the HID functionality signal from the first electronic device to the second electronic device, the second electronic device can respond to the instructions of the peripheral device of the first electronic device, thus realizing the function of controlling one electronic device through the peripheral device of another. For example, when the first electronic device is a television and the second electronic device is a mobile phone, and the mobile phone's display interface is projected onto the television's display interface, when the television's peripheral device sends a move, click, or selection event on the television screen, it can move, click, or select the icon of the mobile app displayed on the television screen, thereby achieving good interactive operation. Since mobile phones have a richer application ecosystem than televisions, the method in this application can achieve interactive operation between large and small screens, connecting the ecosystems of televisions and mobile phones, thereby effectively improving the user experience and simultaneously enhancing the application value of televisions.

[0177] Furthermore, the first electronic device uses the HID standard protocol to receive the first signal and send the second signal to the second electronic device. Compared with related technologies, it does not require the establishment of two channels and can be widely compatible with various peripherals of the first electronic device (including ordinary remote control, new pointing remote control, mouse, keyboard, touch screen and game controller, etc.) and various second electronic devices, including mobile phones and tablets. The second electronic device does not need to develop a dedicated communication protocol stack, which reduces the development cost of the second electronic device.

[0178] Furthermore, the first electronic device responds to a first signal and calculates first coordinates based on the target position corresponding to the first signal on its display interface. These first coordinates indicate the absolute coordinates in the screen coordinate system of the second electronic device. The target position is set with reference to the screen coordinate system of the first electronic device; however, since the screen coordinate systems of the first and second electronic devices are different, the target position cannot be adapted to both screen coordinate systems. This embodiment of the application calculates the first coordinates based on the target position, enabling the second electronic device to determine its absolute coordinates in its screen coordinate system, thereby ensuring the accuracy of control over the second electronic device.

[0179] The order of the methods provided in the embodiments of this application can be appropriately adjusted, and the processes can be added, subtracted, and / or combined, or partially combined, as appropriate. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and the embodiments of this application do not limit them in this regard.

[0180] The foregoing primarily describes the control method provided in the embodiments of this application from the perspective of the device. It is understood that, in order to achieve the above functions, the device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into a management node or terminal node. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0182] Figure 13 is a block diagram of a control device provided in an embodiment of this application. When each functional module is divided according to its corresponding function, the control device 500 may include a transceiver module 501, a display module 502, and a processing module 503. Exemplarily, the control device may be a first electronic device, or a chip within the first electronic device, or other combined devices or components having the aforementioned control device functions. When the control device 500 is a first electronic device, the transceiver module 501 may be a transceiver, which may include an antenna and radio frequency circuits, etc.; the processing module 503 may be a processor (or processing circuit), such as a baseband processor, which may include one or more CPUs. When the control device 500 is a device or component having the aforementioned functions, the transceiver module 501 may be a radio frequency unit; the processing module 503 may be a processor (or processing circuit), such as a baseband processor. When the control device 500 is a chip system, the transceiver module 501 may be an input / output interface of a chip (e.g., a baseband chip); the processing module 503 may be a processor (or processing circuit) of the chip system, and may include one or more central processing units. It should be understood that the transceiver module 501 in the embodiments of this application can be implemented by a transceiver or transceiver-related circuit components; the processing module 503 can be implemented by a processor or processor-related circuit components (or, referred to as processing circuit).

[0183] The first electronic device establishes a connection with the second electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device. A transceiver module 501 is used to receive a first image sent by the second electronic device; a display module 502 is used to display the first image in the display interface area of ​​the second electronic device; the transceiver module 501 is also used to receive a first signal with HID functionality sent by a peripheral device of the first electronic device; and when the target of the first signal is the second electronic device, the transceiver module 501, in response to the first signal, sends a second signal with HID functionality to the second electronic device, so that the second electronic device responds to the second signal and performs a corresponding operation on the target interface displaying the first image.

[0184] In conjunction with the above scheme, the processing module 503 is also used to respond to the first signal by converting the first signal into a second signal that conforms to the HID format of the second electronic device.

[0185] In conjunction with the above scheme, the first electronic device establishes a connection with the second electronic device through an HID device, the HID device having a resolution attribute; the processing module 503 is specifically used to: calculate a first coordinate based on the target position corresponding to the first signal on the display interface of the first display device, the first coordinate indicating the absolute coordinate of the second electronic device in the screen coordinate system; generate a second signal based on the first coordinate, so that the second electronic device responds to the second signal and performs the operation corresponding to the absolute coordinate of the second electronic device in the screen coordinate system on the target interface.

[0186] In conjunction with the above scheme, the processing module 503 is also used to configure the attributes of the HID device, including the HID device resolution; the transceiver module 501 is also used to send the attributes of the HID device to the second electronic device so that the second electronic device responds to the HID device resolution and determines the scaling mapping relationship between the screen resolution of the second electronic device and the HID device resolution; the processing module 503 is specifically used to determine the coordinates of the target position relative to the HID device resolution as the first coordinates so that the second electronic device responds to the second signal and determines the absolute coordinates in the screen coordinate system of the second electronic device according to the scaling mapping relationship and the first coordinates.

[0187] Combining the above scheme, the first signal and the second signal are HID events. The processing module 503 is specifically used to convert the event type of the first signal to obtain the second signal.

[0188] In conjunction with the above scheme, the display module 502 is also used to respond to the first signal and display a cursor at the target position corresponding to the first signal in the current display interface of the first electronic device.

[0189] In conjunction with the above scheme, the processing module 503 is further configured to: obtain the coordinate range of the display interface area of ​​the second electronic device in the screen coordinate system of the first electronic device; determine that the target of the first signal is the second electronic device when the target position is located in the display interface area of ​​the second electronic device; the target position is the position of the first signal corresponding to the display interface of the first display device.

[0190] In combination with the above scheme, the signals with HID function (including the first signal and the second signal) are transmitted using the target protocol; the target protocol is: USB protocol, I2C protocol, SPI protocol, Bluetooth protocol, WiFi protocol or HDMI protocol.

[0191] In conjunction with the above scheme, the transceiver module 501 is also used to receive a second image sent by the second electronic device. The second image is an image displayed on the target interface after the second electronic device performs a corresponding operation on the target interface in response to the second signal. The display module is also used to display the second image in the display interface area of ​​the second electronic device.

[0192] Figure 14 is a block diagram of another control device provided in an embodiment of this application. When each functional module is divided according to its corresponding function, the control device 600 may include a transceiver module 601 and a processing module 602. Exemplarily, the control device may be a second electronic device, or a chip within the second electronic device, or other combined devices or components having the functions described above. When the control device 600 is a second electronic device, the transceiver module 601 may be a transceiver, which may include an antenna and radio frequency circuits, etc.; the processing module 602 may be a processor (or processing circuit), such as a baseband processor, which may include one or more CPUs. When the control device 600 is a device or component having the above functions, the transceiver module 601 may be a radio frequency unit; the processing module 602 may be a processor (or processing circuit), such as a baseband processor. When the control device 600 is a chip system, the transceiver module 601 may be an input / output interface of a chip (e.g., a baseband chip); the processing module 602 may be a processor (or processing circuit) of the chip system, and may include one or more central processing units. It should be understood that the transceiver module 601 in the embodiments of this application can be implemented by a transceiver or transceiver-related circuit components; the processing module 602 can be implemented by a processor or processor-related circuit components (or, referred to as processing circuit).

[0193] The second electronic device establishes a connection with the first electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device. A transceiver module 601 is used to send a first image to the first electronic device, causing the first electronic device to display the first image in the display interface area of ​​the second electronic device; the transceiver module 601 is also used to receive a second signal sent by the first electronic device in response to a first signal, wherein the first and second signals are signals with HID functionality; a processing module 602 is used to perform corresponding operations on the target interface displaying the first image in response to the second signal.

[0194] In conjunction with the above scheme, the second electronic device establishes a connection with the HID device in the first electronic device, and the HID device has a resolution attribute; the second signal includes a first coordinate, which indicates the absolute coordinates in the screen coordinate system of the second electronic device; the processing module 602 is specifically used to respond to the first coordinate of the second signal and perform the operation corresponding to the absolute coordinates in the screen coordinate system of the second electronic device on the target interface.

[0195] In conjunction with the above scheme, the transceiver module 601 is also used to receive the attributes of the HID device, including the HID device resolution; the processing module 602 is also used to determine the scaling mapping relationship between the screen resolution of the second electronic device and the HID device resolution in response to the HID device resolution; specifically, the processing module 602 is used to determine the absolute coordinates of the second electronic device in the screen coordinate system in response to the first coordinate of the second signal, based on the scaling mapping relationship and the first coordinate.

[0196] In conjunction with the above scheme, the transceiver module 601 is specifically used to receive the second signal using a target protocol, which may be: USB protocol, I2C protocol, SPI protocol, Bluetooth protocol, WiFi protocol, or HDMI protocol.

[0197] In conjunction with the above scheme, after responding to the second signal and performing the corresponding operation on the target interface displaying the first image, the transceiver module 601 is also used to send the second image to the first electronic device, the second image being the image currently displayed on the target interface.

[0198] Figure 15 is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 700 can be a management node or a terminal node, or it can be a chip or functional module in a management node or a terminal node. As shown in Figure 15, the electronic device 700 includes a processor 701, a transceiver 702, and a communication line 703.

[0199] The processor 701 is used to execute any step in the method embodiments shown in Figures 5, 7 and 8, and when performing processes such as sending physical layer-specific configuration information, it can selectively call the transceiver 702 and the communication line 703 to complete the corresponding operations.

[0200] Furthermore, the electronic device 700 may also include a memory 704. The processor 701, memory 704, and transceiver 702 can be connected via a communication line 703.

[0201] Transceiver 702 is used to communicate with other devices or other communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Transceiver 702 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0202] The transceiver 702 is mainly used for sending and receiving messages, and may include a transmitter and a receiver to send and receive messages, respectively; operations other than sending and receiving messages are implemented by the processor, such as generating transmission frames.

[0203] Communication line 703 is used to transmit information between the various components included in electronic device 700.

[0204] In one design, the processor can be viewed as a logic circuit, and the transceiver as an interface circuit.

[0205] Memory 704 is used to store instructions. These instructions can be computer programs.

[0206] It should be noted that the memory 704 can exist independently of the processor 701, or it can be integrated with the processor 701. The memory 704 can be used to store instructions, program code, or some data, etc. The memory 704 can be located inside or outside the electronic device 700, without limitation. The processor 701 is used to execute the instructions stored in the memory 704 to implement the method provided in the above embodiments of this application.

[0207] In one example, processor 701 may include one or more processors, such as processor 0 and processor 1 in Figure 15.

[0208] As an alternative implementation, the electronic device 700 may include multiple processors, for example, in addition to the processor 701 shown in FIG15, it may also include a processor 707.

[0209] As an optional implementation, the electronic device 700 also includes an output device 705 and an input device 706. For example, the input device 706 is a device such as a keyboard, mouse, microphone, or joystick, and the output device 705 is a device such as a display screen or speaker.

[0210] It should be noted that the electronic device 700 can be a chip system or a device with a similar structure to that shown in Figure 15. The chip system can be composed of chips or include chips and other discrete devices. Actions, terms, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages used for interaction between devices in the embodiments of this application are merely examples; other names can be used in specific implementations without limitation. Furthermore, the composition structure shown in Figure 15 does not constitute a limitation on the electronic device 700. In addition to the components shown in Figure 15, the electronic device 700 may include more or fewer components than shown in Figure 15, or combine certain components, or have different component arrangements.

[0211] The processor and transceiver described in this application can be implemented on integrated circuits (ICs), analog ICs, radio frequency integrated circuits, mixed-signal ICs, application-specific integrated circuits (ASICs), printed circuit boards (PCBs), electronic devices, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductors (CMOS), n-metal-oxide-semiconductor (NMOS), p-type metal oxide semiconductors (PMOS), bipolar junction transistors (BJTs), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0212] Figure 16 is a schematic diagram of a communication device provided in an embodiment of this application. This communication device is applicable to the scenarios shown in the above-described method embodiments. For ease of explanation, Figure 16 only shows the main components of the communication device, including a processor, memory, control circuit, and input / output devices. The processor is mainly used to process communication protocols and communication data, execute software programs, and process the data of the software programs. The memory is mainly used to store software programs and data. The control circuit is mainly used for power supply and the transmission of various electrical signals. The input / output devices are mainly used to receive user input data and output data to the user.

[0213] When the communication device is a management node or a terminal node, the control circuit can be a motherboard, the memory includes storage media such as hard disks, RAM, and ROM, and the processor can include a baseband processor and a central processing unit (CPU). The baseband processor is mainly used to process communication protocols and communication data, while the CPU is mainly used to control the entire communication device, execute software programs, and process data from the software programs. Input / output devices include displays, keyboards, and mice. The control circuit can further include or be connected to transceiver circuits or transceivers, such as network cable interfaces, for sending or receiving data or signals, such as for data transmission and communication with other devices. Furthermore, it can also include an antenna for sending and receiving messages, for data / request transmission with other devices.

[0214] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to perform any of the methods described in the embodiments of this application.

[0215] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by a computer or a communication-capable device using computer programs or instructions to control related hardware. The computer program or set of instructions can be stored in the computer-readable storage medium. When executed, the computer program or set of instructions can include the processes described in the above method embodiments. The computer-readable storage medium can be an internal storage unit of the first or second electronic device in any of the foregoing embodiments, such as a hard disk, memory, or chip of the first or second electronic device. The computer-readable storage medium can also be an external storage device of the first or second electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the first or second electronic device. Further, the computer-readable storage medium can include both internal storage units and external storage devices of the first or second electronic device. The computer-readable storage medium is used to store the computer program or instructions and other programs and data required by the first or second electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0216] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0217] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0218] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0219] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0220] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0221] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or management node, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0222] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control method, characterized in that, The method is applied to a first electronic device, which establishes a connection with a second electronic device, wherein the display interface of the first electronic device includes the display interface area of ​​the second electronic device; the method includes: Receive the first image sent by the second electronic device; The first image is displayed in the display interface area of ​​the second electronic device; Receive a first signal with HID (Human Interface Device) function sent by the peripheral device of the first electronic device; When the target of the first signal is the second electronic device, in response to the first signal, a second signal with HID function is sent to the second electronic device, so that the second electronic device responds to the second signal and performs a corresponding operation on the target interface displaying the first image.

2. The method according to claim 1, characterized in that, The step of sending a second signal with HID functionality to the second electronic device in response to the first HID event includes: In response to the first signal, the first signal is converted into a second signal conforming to the HID format of the second electronic device.

3. The method according to claim 2, characterized in that, The first electronic device establishes a connection with the second electronic device via an HID device, wherein the HID device has a resolution attribute; the step of converting the first signal into a second signal conforming to the HID format of the second electronic device includes: Based on the target position corresponding to the first signal on the display interface of the first display device, the first coordinates are calculated, and the first coordinates indicate the absolute coordinates in the screen coordinate system of the second electronic device; The second signal is generated based on the first coordinates, so that the second electronic device responds to the second signal and performs an operation corresponding to the absolute coordinates in the screen coordinate system of the second electronic device on the target interface.

4. The method according to claim 3, characterized in that, The method further includes: Configure the properties of the HID device, including the HID device resolution; Send the attributes of the HID device to the second electronic device so that the second electronic device, in response to the HID device resolution, determines the scaling mapping relationship between the screen resolution of the second electronic device and the HID device resolution; The step of calculating the first coordinates based on the target position corresponding to the first signal on the display interface of the first display device includes: The coordinates of the target position relative to the resolution of the HID device are determined as the first coordinates, so that the second electronic device responds to the second signal and determines the absolute coordinates of the second electronic device in the screen coordinate system according to the scaling mapping relationship and the first coordinates.

5. The method according to any one of claims 2 to 4, characterized in that, The first signal and the second signal are HID events, and the step of converting the first signal into the second signal conforming to the HID format of the second electronic device includes: The event type of the first signal is converted to obtain the second signal.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: In response to the first signal, a cursor is displayed at the target position corresponding to the first signal in the current display interface of the first electronic device.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: In the screen coordinate system of the first electronic device, obtain the coordinate range of the display interface area of ​​the second electronic device; When the target location is within the display interface area of ​​the second electronic device, the target of the first signal is determined to be the second electronic device, and the target location is the position of the first signal on the display interface of the first display device.

8. The method according to any one of claims 1 to 7, characterized in that, Signals with HID functionality are transmitted using a target protocol; the target protocol is: Universal Serial Bus (USB) protocol, Integrated Circuit Bus (I2C) protocol, Serial Peripheral Interface (SPI) protocol, Bluetooth protocol, Wireless Fidelity (WiFi) protocol, or High Definition Multimedia Interface (HDMI) protocol.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Receive a second image sent by the second electronic device, wherein the second image is an image displayed on the target interface after the second electronic device performs a corresponding operation on the target interface in response to the second signal; The second image is displayed in the display interface area of ​​the second electronic device.

10. The method according to any one of claims 1 to 9, characterized in that, The first signal is a first HID event, and the second signal is a second HID event.

11. A control method, characterized in that, The method is applied to a second electronic device, which establishes a connection with a first electronic device, wherein the display interface of the first electronic device includes the display interface area of ​​the second electronic device; the method includes: Send a first image to the first electronic device so that the first electronic device displays the first image in the display interface area of ​​the second electronic device; Receive a second signal sent by the first electronic device in response to a first signal, wherein the first signal and the second signal are signals with HID (Human Interface Device) functionality; In response to the second signal, a corresponding operation is performed on the target interface displaying the first image.

12. The method according to claim 11, characterized in that, The second electronic device establishes a connection with the HID device in the first electronic device, the HID device having a resolution attribute; the second signal includes a first coordinate, the first coordinate indicating the absolute coordinates in the screen coordinate system of the second electronic device; the step of performing a corresponding operation on the target interface displaying the first image in response to the second signal includes: In response to the first coordinate of the second signal, an operation corresponding to the absolute coordinates in the screen coordinate system of the second electronic device is performed on the target interface.

13. The method according to claim 12, characterized in that, The method further includes: Receive attributes of the HID device, including the HID device resolution; In response to the HID device resolution, a scaling mapping relationship between the screen resolution of the second electronic device and the HID device resolution is determined; The first coordinate, responding to the second signal, performs an operation on the target interface corresponding to the absolute coordinates in the screen coordinate system of the second electronic device, including: In response to the first coordinate of the second signal, the absolute coordinates in the screen coordinate system of the second electronic device are determined according to the scaling mapping relationship and the first coordinate.

14. The method according to any one of claims 11 to 13, characterized in that, Receiving the second signal sent by the first electronic device in response to the first signal includes: The second signal is received using a target protocol, which is: Universal Serial Bus (USB) protocol, Integrated Circuit Bus (I2C) protocol, Serial Peripheral Interface (SPI) protocol, Bluetooth protocol, WiFi protocol, or High Definition Multimedia Interface (HDMI) protocol.

15. The method according to any one of claims 11 to 14, characterized in that, After the target interface displaying the first image performs a corresponding operation in response to the second signal, the method further includes: Send a second image to the first electronic device, the second image being the image currently displayed on the target interface.

16. The method according to any one of claims 11 to 15, characterized in that, The first signal is a first HID event, and the second signal is a second HID event.

17. A control device, characterized in that, The device is applied to a first electronic device, which establishes a connection with a second electronic device, wherein the display interface of the first electronic device includes the display interface area of ​​the second electronic device; the device includes: A transceiver module is used to receive the first image sent by the second electronic device; The display module is used to display the first image in the display interface area of ​​the second electronic device; The transceiver module is also used to receive a first signal with human-machine interface device (HID) function sent by the peripheral device of the first electronic device; The transceiver module is further configured to, in response to the first signal, send a second signal with HID function to the second electronic device when the target of the first signal is the second electronic device, so that the second electronic device responds to the second signal and performs a corresponding operation on the target interface displaying the first image.

18. A control device, characterized in that, An apparatus for use in a second electronic device, the second electronic device being connected to a first electronic device, wherein the display interface of the first electronic device includes the display interface area of ​​the second electronic device; the apparatus includes: A transceiver module is used to send a first image to the first electronic device so that the first electronic device displays the first image in the display interface area of ​​the second electronic device; The transceiver module is further configured to receive a second signal sent by the first electronic device in response to the first signal, wherein the first signal and the second signal are signals with HID (Human Interface Device) function. The processing module is configured to perform a corresponding operation on the target interface displaying the first image in response to the second signal.

19. An electronic device, corresponding to a first electronic device, characterized in that, The first electronic device establishes a connection with the second electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device; The first electronic device includes: Human-machine interface (HID) devices; One or more processors; Memory, used to store one or more computer programs or instructions; When the first electronic device executes the one or more computer programs or instructions, the first electronic device performs the following steps: Receive the first image sent by the second electronic device; The first image is displayed in the display interface area of ​​the second electronic device; Receive a first signal with HID (Human Interface Device) function sent by the peripheral device of the first electronic device; When the target of the first signal is the second electronic device, in response to the first signal, a second signal with HID function is sent to the second electronic device, so that the second electronic device responds to the second signal and performs a corresponding operation on the target interface displaying the first image.

20. The electronic device according to claim 19, characterized in that, The HID device adopts the Universal Serial Bus (USB) protocol, the Integrated Circuit Bus (I2C) protocol, the Serial Peripheral Interface (SPI) protocol, the Bluetooth protocol, the Wireless Fidelity (WiFi) protocol, or the High Definition Multimedia Interface (HDMI) protocol.

21. A control system, characterized in that, The control system includes a first electronic device, a second electronic device, and peripherals of the first electronic device. The first electronic device is connected to the second electronic device, and the display interface of the first electronic device includes the display interface area of ​​the second electronic device. The peripheral device of the first electronic device is used to send a first signal with the human-machine interface device (HID) function to the first electronic device; The first electronic device is used to perform the method as described in any one of claims 1 to 10, and the second electronic device is used to perform the method as described in any one of claims 11 to 16.

22. A chip, characterized in that, The chip includes: Processing circuits and interface circuits; The interface circuit is used to couple with the memory outside the chip and to provide a communication interface for the processing circuit to access the memory. The processing circuit is used to execute program instructions in the memory to implement the method as described in any one of claims 1 to 16.

23. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 16.

24. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 16.

Images