Control method and apparatus

Abstract

The present application provides a control method and apparatus. The method comprises: acquiring a target mode, the target mode being used for indicating that a first electronic device serves as a target-type input device of a second electronic device; establishing a first communication connection with the second electronic device on the basis of the target mode; on the basis of a first image collected by a camera, acquiring a first reverse control parameter indicated by a user in the first image; sending the first reverse control parameter to the second electronic device by means of the first communication connection; and displaying a second image sent by the second electronic device by means of a second communication connection, the second image being generated by the second electronic device on the basis of the first reverse control parameter. In the present application, an electronic device is used as an input device of a second electronic device, and a reverse control parameter is acquired on the basis of an acquired image so as to achieve reverse control on the second electronic device.

Description

Control method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on March 21, 2024, with application number 202410330283.X and application name “Control Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of terminal technology, and in particular to a control method and device. Background Art

[0003] With the development of terminal device technology, the types of terminal devices are increasing, and their application scenarios are becoming more and more extensive. For example, a user can use a mobile phone to connect to a projector, transmit the screen on the mobile phone side to the projector side, and then project the screen. However, the current method of interaction between the terminal and the user is all carried out on the content generation side (such as the mobile phone). The interaction method is relatively simple, which affects the user experience. Summary of the Invention

[0004] This application provides a control method and device. In this method, an electronic device serving as a display terminal can also serve as an input device for another electronic device, sending reverse control parameters to the other electronic device to achieve reverse control of the other electronic device. The user can control the content terminal from the display terminal, improving the quality of interaction and enhancing the user experience.

[0005] In a first aspect, the present application provides a control method. The method includes: obtaining a target mode, the target mode being used to indicate that a first electronic device serves as a target type input device for a second electronic device. Based on the target mode, a first communication connection is established with the second electronic device. Based on a first image captured by a camera, a first reverse control parameter indicated by a user in the first image is obtained. The first reverse control parameter is transmitted to the second electronic device via the first communication connection. A second image, sent by the second electronic device via a second communication connection, is displayed, the second image being generated by the second electronic device based on the first reverse control parameter. Thus, in the present application, after obtaining the target mode, the electronic device can determine which type of input device to connect to another electronic device (i.e., the second electronic device). A first communication connection is established with the second electronic device for transmitting the reverse control parameter. After the connection is established, the electronic device can serve as an input device of the specified type for the second electronic device to control the second electronic device. In other words, the electronic device serving as the display terminal can also serve as a virtual input device for the second electronic device, thereby achieving reverse control of the second electronic device. For the second electronic device, the electronic device acts as an input device to connect to the second electronic device and exchange data with the second electronic device. The software and hardware of the second electronic device do not need to be updated to implement the reverse control scheme in the embodiments of the present application. That is, the scheme in the present application is compatible with existing products, achieving "connect and use" functionality. Furthermore, the electronic device can obtain the reverse control parameters indicated by the user in the image through the image captured by the camera, so as to control the second electronic device through the reverse control parameters, so that the user can control the image generated by the second electronic device by indicating on the image, and further control the image displayed by the electronic device, so that the user can achieve reverse control of the content side (i.e., the second electronic device) on the display end. Furthermore, the way in which the user indicates on the image can make what the user sees, feels, and controls consistent, while improving the quality of interaction and effectively improving the user experience. Furthermore, the electronic device in the present application can control the second electronic device by sending the reverse control parameters to the second electronic device. The amount of data transmitted is small, and while reducing the occupancy of the communication transmission bandwidth, it effectively reduces the data transmission delay, so that the second electronic device can respond to the user's instructions in a timely manner, further improving the user experience.

[0006] Exemplarily, the electronic device may optionally be a projector, and the electronic device may display an image in a projection manner, that is, projecting the image sent by the second electronic device onto a screen or a wall.

[0007] Exemplarily, the acquisition range of the electronic device's camera is greater than or equal to the size of the image displayed by the electronic device.

[0008] In one possible implementation, the first image includes a first display image and an image of a target object, wherein the image of the target object is within the first display image; wherein the first display image is an image generated by the first electronic device, or the first display image is an image sent by the second electronic device via the second communication connection, and the image of the target object includes at least one of an image corresponding to a user's gesture, an image of a laser point mapped by the user on the first display image using a laser device, and an image corresponding to a pointing tool used by the user. In this way, the user can control the direction of the content side (i.e., the second electronic device) in different ways to adapt to different application scenarios and improve the user experience.

[0009] In one possible implementation, based on a first image captured by a camera, obtaining a first reverse control parameter indicated by a user in the first image includes: obtaining the absolute pointing position coordinates of the target object's image in the first displayed image, wherein the first reverse control parameter is used to indicate the absolute pointing position coordinates. Thus, the present application utilizes absolute pointing technology to obtain the absolute pointing position coordinates of the target object's image in the first displayed image to improve the accuracy of reverse control, thereby ensuring that what the user sees, feels, and controls is consistent, thereby enhancing the user experience.

[0010] In one possible implementation, the first reverse control parameter includes a position parameter and a first auxiliary parameter. Based on a first image captured by a camera, obtaining the first reverse control parameter indicated by the user in the first image includes: obtaining the absolute pointing position coordinates of the target object's image in the first displayed image, where the position parameter indicates the absolute pointing position coordinates. Image recognition is performed on the target object's image to obtain the first auxiliary parameter. This allows the user to trigger a specified event through gestures or other means, and the electronic device can obtain the corresponding auxiliary parameter based on the recognized image to reversely control the second electronic device.

[0011] In one possible implementation, the second image includes a second display image and a cursor. The second display image is generated by the second electronic device based on the displayed content, and the cursor is generated by the second electronic device based on the first reverse control parameter. In this way, the electronic device in this application can be virtualized as a mouse-type input device, that is, the first electronic device is equivalent to a mouse for the second electronic device. The user can control the movement of the cursor in the screen through gestures and other methods to achieve reverse control of the content side.

[0012] In one possible implementation, after displaying the second image sent by the second electronic device via the second communication connection, the method further includes: obtaining a second reverse control parameter indicated by the user in the third image based on the third image captured by the camera; sending the second reverse control parameter to the second electronic device via the first communication connection; displaying a fourth image sent by the second electronic device via the second communication connection, where the fourth image is generated by the second electronic device based on the second reverse control parameter; the second reverse control parameter is used to indicate moving the cursor from a first position in the second displayed image to a second position in the fourth displayed image. In this way, the user can control the cursor position in the image generated by the content side by moving a finger or other means on the image displayed on the electronic device side of the display, thereby achieving reverse control of the content side.

[0013] In one possible implementation, the third image includes the second image and an image of the target object; obtaining a second reverse control parameter indicated by the user in the third image based on the third image captured by the camera includes obtaining the second reverse control parameter based on the absolute pointing position coordinates of the target object's image in the second displayed image. Thus, the present application utilizes absolute pointing technology to obtain the absolute pointing position coordinates of the target object's image in the displayed image to improve the accuracy of reverse control, ensuring that what the user sees, feels, and controls is consistent, thereby enhancing the user experience.

[0014] In one possible implementation, the method further includes: obtaining a first reverse control parameter indicated by the user in the first image based on a first image captured by the camera, and obtaining a second auxiliary parameter based on a second operation triggered by the user using a control device; sending the first reverse control parameter to the second electronic device via the first communication connection, and further includes: sending the second auxiliary parameter to the second electronic device via the first communication connection; wherein the second image is generated by the second electronic device based on the first reverse control parameter and the second auxiliary parameter. In this way, while the user indicates the position coordinates in the reverse control information through gestures, laser devices, etc., he can also indicate the auxiliary information through a control device (such as a remote control, etc.), so that the content side (i.e., the second electronic device) can generate a corresponding image based on the position coordinates and the auxiliary information.

[0015] In one possible implementation, establishing a first communication connection with a second electronic device based on a target mode includes: sending a target descriptor to the second electronic device, where the target descriptor is used to indicate that the first electronic device is a target-type input device. In this way, the electronic device in the present application, by carrying the target descriptor, can enable the opposite device to identify the type of input device virtualized by the electronic device, thereby achieving seamless compatibility on the content side (i.e., the second electronic device).

[0016] In a possible implementation, the target type input device includes: a mouse type, a keyboard type, a stylus type, or a handle type.

[0017] In a second aspect, the present application provides a control device, which includes: a first acquisition module for acquiring a target mode, wherein the target mode is used to indicate that the first electronic device serves as a target type input device of the second electronic device; a communication module for establishing a first communication connection with the second electronic device based on the target mode; a second acquisition module for acquiring a first reverse control parameter indicated by the user in the first image based on the first image captured by the camera; the communication module for sending the first reverse control parameter to the second electronic device through the first communication connection; and a display module for displaying a second image sent by the second electronic device through the second communication connection, wherein the second image is generated by the second electronic device based on the first reverse control parameter.

[0018] In one possible implementation, the first image includes a first display image and an image of a target object, and the image of the target object is within the first display image; wherein the first display image is an image generated by the first electronic device, or the first display image is an image sent by the second electronic device through the second communication connection, and the image of the target object includes at least one of an image corresponding to a user's gesture, an image of a laser point mapped by the user on the first display image using a laser device, and an image corresponding to an indicating tool used by the user.

[0019] In a possible implementation, the second acquisition module is specifically configured to acquire the absolute pointing position coordinates of the image of the target object in the first display image, and the first reverse control parameter is used to indicate the absolute pointing position coordinates.

[0020] In one possible implementation, the first reverse control parameter includes a position parameter and a first auxiliary parameter, and the second acquisition module is specifically used to: obtain the absolute pointing position coordinates of the image of the target object in the first display image, the position parameter is used to indicate the absolute pointing position coordinates; perform image recognition on the image of the target object to obtain the first auxiliary parameter.

[0021] In a possible implementation, the second image includes a second display image and a cursor, the second display image is generated by the second electronic device based on display content, and the cursor is generated by the second electronic device based on the first reverse control parameter.

[0022] In one possible implementation, the second acquisition module is further used to obtain the second reverse control parameter indicated by the user in the third image based on the third image captured by the camera; the communication module is further used to send the second reverse control parameter to the second electronic device through the first communication connection; the display module is further used to display the fourth image sent by the second electronic device through the second communication connection, and the fourth image is generated by the second electronic device based on the second reverse control parameter; the second reverse control parameter is used to indicate moving the cursor from a first position in the second display image to a second position in the fourth display image.

[0023] In a possible implementation, the third image includes the second image and the image of the target object; the second acquisition module is specifically configured to acquire the second reverse control parameter based on the absolute pointing position coordinates of the image of the target object in the second display image.

[0024] In one possible implementation, the second acquisition module is used to obtain the first reverse control parameter indicated by the user in the first image based on the first image captured by the camera, and is also used to obtain the second auxiliary parameter based on the second operation triggered by the user using the control device; the communication module is also used to send the second auxiliary parameter to the second electronic device through the first communication connection; wherein, the second image is generated by the second electronic device based on the first reverse control parameter and the second auxiliary parameter.

[0025] In a possible implementation, the communication module is specifically configured to send a target descriptor to the second electronic device, where the target descriptor is used to indicate that the first electronic device is a target-type input device.

[0026] In a possible implementation, the target type input device includes: a mouse type, a keyboard type, a stylus type, or a handle type.

[0027] In a third aspect, the present application provides an electronic device comprising one or more processors and a memory; and one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the method in the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0029] In a fifth aspect, the present application provides a computer program comprising instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0030] In a sixth aspect, the present application provides a chip comprising a processing circuit and a transceiver pin. The transceiver pin and the processing circuit communicate with each other via an internal connection path, and the processing circuit executes the method of the first aspect or any possible implementation of the first aspect to control the receive pin to receive a signal and to control the transmit pin to send a signal.

[0031] In a seventh aspect, an embodiment of the present application provides a control system, which includes the first electronic device and the second electronic device involved in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is an exemplary diagram of a communication system architecture;

[0033] FIG2 is a schematic diagram illustrating a hardware structure of an electronic device;

[0034] FIG3 is a schematic diagram illustrating a software structure of an electronic device;

[0035] FIG4 is a schematic diagram illustrating a hardware structure of an electronic device;

[0036] FIG5 is a schematic diagram illustrating a software structure of an electronic device;

[0037] FIG6 exemplarily shows a schematic diagram of a connection between an electronic device and a camera;

[0038] FIG7 is a schematic structural diagram of an exemplary camera;

[0039] FIG8 is a schematic diagram illustrating an exemplary application scenario;

[0040] FIG9 is a schematic diagram showing the principle of an exemplary control method;

[0041] FIG10 is a schematic diagram illustrating exemplary communication connections between a first electronic device and a second electronic device;

[0042] 11A to 11D are schematic diagrams illustrating exemplary connection methods;

[0043] FIG12 is a flow chart showing an exemplary control method;

[0044] FIG13 is a schematic diagram illustrating an exemplary application scenario;

[0045] FIG14 is a schematic diagram illustrating an exemplary application scenario;

[0046] FIG15 is a schematic diagram illustrating an exemplary process of establishing a Bluetooth connection;

[0047] 16A and 16B are schematic diagrams of exemplary user interfaces;

[0048] FIG17 is a schematic diagram illustrating an exemplary application scenario;

[0049] 18A and 18B are schematic diagrams illustrating exemplary application scenarios;

[0050] 19A and 19B are schematic diagrams illustrating exemplary module interactions;

[0051] FIG20 is a schematic diagram showing an exemplary coordinate transformation;

[0052] 21A and 21B are schematic diagrams illustrating exemplary application scenarios;

[0053] FIG22 is a schematic diagram illustrating an exemplary application scenario;

[0054] FIG23 is a schematic structural diagram of an exemplary device;

[0055] FIG24 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0057] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0058] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0059] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0060] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0061] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on a mobile device or electronic device, and finally presented as content that the user can recognize. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of a mobile device or electronic device.

[0062] First, the communication system 1000 provided in an embodiment of the present application is introduced. Figure 1 is an illustrative diagram of the communication system architecture. Please refer to Figure 1. The system includes but is not limited to a first electronic device and a second electronic device. In an embodiment of the present application, the first electronic device is a projector. In other embodiments, the first electronic device may also be other electronic devices with camera and display functions. For example, it may be a television, etc. The second electronic device is an electronic device such as a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not limit the specific type of the terminal device.

[0063] FIG2 shows a schematic diagram of the structure of a second electronic device 100. It should be understood that the electronic device 100 shown in FIG2 is merely an example of an electronic device, and that the electronic device 100 may have more or fewer components than shown, may combine two or more components, or may have a different component configuration. The various components shown in FIG2 may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0064] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0065] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0066] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0067] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0068] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0069] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the electronic device 100.

[0070] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0071] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0072] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0073] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0074] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0075] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0076] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0077] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0078] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0079] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0080] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0081] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0082] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0083] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0084] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0085] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0086] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0087] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0088] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0089] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0090] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0091] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0092] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0093] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0094] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0095] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0096] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0097] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0098] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0099] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0100] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0101] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0102] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0103] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0104] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0105] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0106] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0107] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0108] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to prevent the electronic device 100 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.

[0109] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0110] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0111] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0112] FIG3 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0113] The layered architecture of electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0114] The application layer can include a series of application packages.

[0115] As shown in Figure 3, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, and screen projection.

[0116] In an embodiment of the present application, a user can project video data from an electronic device through the screen projection function in a video application, that is, project the video screen to a projector for display. Alternatively, a user can also project the screen currently displayed on the electronic device to a projector for display through third-party screen projection software or screen projection software developed by the original manufacturer in the electronic device. The specific screen projection method can be set according to actual needs and is not limited by this application.

[0117] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0118] As shown in FIG3 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0119] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0120] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0121] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0122] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0123] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0124] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0125] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0126] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0127] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0128] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0129] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0130] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0131] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0132] A 2D graphics engine is a drawing engine for 2D drawings.

[0133] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, Bluetooth driver, etc.

[0134] It is understood that the components included in the framework layer, system library, and runtime layer shown in FIG3 do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine or split some components, or arrange the components differently.

[0135] Figure 4 illustrates an exemplary hardware structure of a first electronic device 200. As shown in Figure 4 , the first electronic device 200 may include a processor 201, a memory 202, a wireless communication processing module 203, a power switch 204, a high-definition multimedia interface (HDMI) communication processing module 205, a USB communication processing module 206, an image projection module 207, an audio module 208, and a camera 209 (also referred to as an image acquisition module). Each module may be connected via a bus.

[0136] Processor 201 can be used to read and execute computer-readable instructions. In a specific implementation, processor 202 may mainly include a controller, an arithmetic unit, and registers. The controller is primarily responsible for decoding instructions and issuing control signals for operations corresponding to the instructions. The arithmetic unit is primarily responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations, and may also perform address operations and conversions. Registers are primarily responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. In a specific implementation, the hardware architecture of processor 201 may be an application-specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, or an NP architecture.

[0137] In some embodiments, the processor 201 may be configured to parse signals received by the wireless communication processing module 203, such as a projection request or projection instruction sent by the second electronic device. The processor 201 may be configured to perform corresponding processing operations based on the parsing results, such as driving the image projection module 207 to perform a projection operation based on the projection request or projection instruction.

[0138] In some examples, processor 201 includes a video codec for compressing or decompressing digital video. In embodiments of the present application, the video codec can decompress multimedia content from the second electronic device 100. Electronic device 200 may support one or more video codecs and can play videos encoded in one or more formats, such as MPEG1, MPEG2, MPEG3, and MPEG4. Processor 201 can be configured to drive the image projection module to display the content based on the decompression results of the video codec.

[0139] The wireless communication processing module 203 may include a Bluetooth (BT) communication processing module 203A, a Wi-Fi communication processing module 203B, an infrared communication module 203C, and other communication modules.

[0140] In some embodiments, the wireless communication processing module 203 can be used to establish a communication connection with the second electronic device 100, and receive encoded data sent by the second electronic device 100 based on the communication connection. For example, the Wi-Fi communication processing module 203B can be used to establish a Wi-Fi direct communication connection with the second electronic device 100, and the Bluetooth (BT) communication processing module 203A can be used to establish a Bluetooth communication connection with the second electronic device 100, that is, the wireless communication processing module 203 can support the first electronic device 200 and the second electronic device 100 to share multimedia content through mirroring (such as miracast). The infrared communication module 203C can be used to receive infrared signals sent by auxiliary devices (such as remote controls, game controllers, etc.). That is, the wireless communication module 203 also supports the first electronic device 200 to receive control signals (which may also be referred to as auxiliary information, auxiliary parameters or control parameters, control information, etc., which are not limited in this application) sent by other auxiliary devices.

[0141] In one embodiment, the wireless communication processing module 203 can monitor signals transmitted by the second electronic device 100, such as detection requests and scanning signals, discover the second electronic device 100, and establish a communication connection with the second electronic device 100. In another embodiment, the wireless communication processing module 203 can also transmit signals, such as detection requests and scanning signals, so that the second electronic device 100 can discover the first electronic device 200 and establish a communication connection (such as a Wi-Fi P2P connection) with the second electronic device 100.

[0142] The memory 202 is coupled to the processor 201 and is used to store various software programs and / or multiple sets of instructions. In a specific implementation, the memory 202 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 202 can store an operating system, such as an embedded operating system such as uCOS, VxWorks, RTLinux, Harmony, Android, etc. The memory 202 can also store a communication program that can be used to communicate with the second electronic device 100, one or more servers, or an additional device.

[0143] The power switch 204 may be used to control the supply of power to the first electronic device 100 .

[0144] The HDMI communication processing module 207 can be used to communicate with other devices via an HDMI interface (not shown). In some embodiments, the first electronic device 200 may further include a serial interface such as an RS-232 interface. The serial interface can be connected to the first electronic device, such as an audio external speaker device such as a speaker, so that the display and the audio external speaker device collaborate to play audio and video. It will be understood that the structure illustrated in Figure 4 does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0145] The USB communication processing module 206 may be configured to communicate with other devices via a USB interface (not shown).

[0146] The image projection module 207 may have a light source (not shown), and may modulate the light emitted from the light source according to the image data and project the image on a screen (or an object such as a wall). The image data may be an image generated by the first electronic device 200 based on the data content of the local end, or may be an image sent by the second electronic device 100 (which may be referred to as a projection image, projection data, etc., which is not limited in this application). Optionally, in the embodiment of the present application, only the second electronic device is described as a projector as an example. As described above, in other embodiments, other electronic devices such as televisions may also implement the reverse control scheme in this application. In these examples, the second electronic device may include a display screen.

[0147] The audio module 208 can be used to output audio signals through the audio output interface, so that the first electronic device 200 can support audio playback. The audio module 230 can also be used to receive audio data through the audio input interface. The audio module 208 may include but is not limited to: a microphone, a speaker, a receiver, etc.

[0148] The camera 209, which may also be referred to as an image acquisition module, is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0149] The software system of the first electronic device 200 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Specifically, the software system of the first electronic device 200 may be Windows, Linux or other operating systems. Taking the system as an example, the software structure of the first electronic device 200 is exemplarily described.

[0150] FIG5 is a software structure block diagram of the first electronic device 200 according to an embodiment of the present application.

[0151] The layered architecture of the first electronic device 200 divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: application layer, framework layer, system library and runtime layer, and kernel layer, from top to bottom.

[0152] The application layer may include applications such as camera, gallery, calendar, gesture control, WLAN, Bluetooth, music, video, reverse control module, etc. It should be noted that the applications included in the application layer shown in Figure 5 are only exemplary and are not limited in this application. It is understandable that the applications included in the application layer do not constitute a specific limitation on the first electronic device 200. In other embodiments of the present application, the first electronic device 200 may include more or fewer applications than the applications included in the application layer shown in Figure 5, and the first electronic device 200 may also include completely different applications.

[0153] In an embodiment of the present application, the reverse control module is used to provide a mode selection interface. The reverse control module can determine the target mode in response to the received user operation. Moreover, after the reverse control module determines the target mode, it can drive the corresponding module (such as a Bluetooth driver, etc.) to establish a communication connection with the second electronic device (which may also be referred to as a control data transmission channel, etc., which is not limited in this application). In addition, the reverse control module can transmit the reverse control information (which may also be referred to as reverse control parameters, reverse operation parameters, etc., which are not limited in this application) input by the control information generation module to the second electronic device through the control data transmission channel.

[0154] The framework layer provides an application programming interface (API) and programming framework for applications in the application layer, including various components and services to support developers' Android development. The framework layer includes some predefined functions. As shown in Figure 3, the framework layer may include a view system, a window manager, a resource manager, a content provider, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. The display interface can be composed of one or more views. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc. The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc. The content provider is used to store and obtain data and make this data accessible to applications. The data may include video, images, audio, etc.

[0155] The system library and runtime layer include the system library and the Android runtime. The system library can include multiple functional modules, such as the browser kernel, 3D graphics library (for example, OpenGL ES), and font library. The browser kernel is responsible for interpreting web page syntax (such as HTML and JavaScript in Standard Generalized Markup Language) and rendering (displaying) the web page. The 3D graphics library is used to implement three-dimensional graphics drawing, image rendering, compositing, and layer processing. The font library is used to implement input of different fonts. The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one part contains function calls required by the Java language, and the other part is the Android core library. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0156] It is understood that the components included in the framework layer, system library, and runtime layer shown in Figure 5 do not constitute a specific limitation on the first electronic device 200. In other embodiments of the present application, the first electronic device 200 may include more or fewer components than shown, or combine or split some components, or arrange the components differently.

[0157] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, control information generation module, etc.

[0158] In one possible implementation, the control information generation module may obtain an image captured by a camera. The control information generation module may recognize the image and obtain control information. Optionally, the control information generation module may also obtain auxiliary information based on an auxiliary signal received from an auxiliary device. The control information generation module may output reverse control information to the reverse control module. The reverse control information may include control information, or may include control information and auxiliary information.

[0159] In another possible implementation, the control information generation module may obtain control information sent by the camera via the camera driver. That is, the camera may obtain control information based on the captured image and send the control information to the control information generation module via the camera driver. Optionally, the control information generation module may also obtain auxiliary information based on an auxiliary signal received from an auxiliary device. The control information generation module may output reverse control information to the reverse control module. The reverse control information may include control information, or may include both control information and auxiliary information.

[0160] That is to say, in the embodiment of the present application, the process of identifying the image to obtain control information can be executed by the control information generation module or by the camera, and this application does not limit it.

[0161] Exemplarily, the camera driver is used to abstract the camera to hide the specific channel of the camera so that the application can access (or call) the camera. The camera driver can communicate with the camera based on the universal serial bus video class (UVC) protocol. The UVC protocol can also be understood as a protocol based on the UVC channel, that is, the camera and the camera driver establish a UVC connection, and transmit messages that comply with the UVC protocol based on the UVC connection. The camera driver can also communicate with the camera based on the remote network driver interface specification (RNDIS) protocol. It should be noted that the RNDIS protocol can also be understood as a protocol based on the Socket channel, that is, the camera and the camera driver establish a Socket connection through the Socket channel, and transmit messages that comply with the RNDIS protocol based on the Socket connection.

[0162] Optionally, the UVC channel can be used to transmit control instructions and video streams; the Socket channel can be used to transmit control information and other information in the embodiments of the present application.

[0163] The camera of the first electronic device 200 may be an external camera and / or a built-in camera. The external camera may be connected to the USB interface of the first electronic device 200 via a USB cable. The built-in camera may be embedded in the first electronic device 200, and the built-in camera may be connected to the USB interface of the first electronic device 200 via a USB cable.

[0164] FIG6 exemplarily shows a schematic diagram of the connection between an electronic device 600 and a camera. As shown in FIG6 , the hardware in the electronic device 600 includes, but is not limited to, a USB interface and a camera. The camera can be connected to the USB interface via a USB cable and data can be exchanged via the USB connection. The USB interface can exchange data with the upper layer. For example, the USB interface outputs data or instructions (such as control information) input by the camera to the control information generation module, and can also output data or instructions input by other applications or modules to the camera.

[0165] Optionally, in an embodiment of the present application, the image acquisition range of the camera of the second electronic device is greater than or equal to the projection range of the image projection module. Optionally, the image acquisition range of the camera of the second electronic device can change with the projection range of the image projection module. Exemplarily, the projection range can be understood as the size of the picture projected by the image projection module on the screen (or object such as a wall). The projection range can be adjusted automatically or manually by the user (for example, using an adjustment button on the projector body or a remote control), and this application does not limit it.

[0166] Figure 7 is a schematic diagram of the software and hardware structure of the camera provided in an embodiment of the present application. As shown in Figure 7, the embodiment of the present application uses a Linux system with a layered camera architecture as an example to illustrate the structure of the camera. The layered architecture of the camera divides the software into several layers, each with clear roles and divisions of labor. The layers communicate with each other through software interfaces. From top to bottom, the camera architecture includes: an application layer and a kernel layer.

[0167] The application layer may include, but is not limited to, applications such as image processing applications. Optionally, image processing applications may include, but are not limited to, sub-functions (or sub-applications) such as video input (VI), video processing sub-system (VPSS), video encoder (VENC), and video graphic system (VGS). Image processing applications are used to process images; for example, one or more sub-functions within an image processing application may be used to perform image processing processes such as noise reduction and color correction on an image; the processed image is then output to an electronic device. The applications included in the application layer shown in FIG7 are merely illustrative and are not limited in this application. It should be understood that the applications included in the application layer do not constitute a specific limitation on the camera. In other embodiments of the present application, a camera may include more or fewer applications than those included in the application layer shown in FIG7. Optionally, each application in the application layer may be pre-installed on the camera or the electronic device including the camera before leaving the factory, or may be installed during an upgrade of the camera or the electronic device including the camera.

[0168] The kernel layer is the middle layer between the hardware and the software (i.e., the application layer). It passes application requests to the hardware and acts as a low-level driver, addressing various devices and components in the system. The kernel layer manages hardware devices and provides access to applications. The kernel layer includes one or more components. For example, it may include the system call interface, process management, memory management, virtual file system, network stack, vision processing module, ISP driver, sensor driver, etc.

[0169] Exemplarily, as described above, the process of obtaining control information can be executed by a camera. In this example, the visual processing module in the camera can obtain the image processed by the ISP, and the visual processing module can obtain control information based on the obtained image.

[0170] Exemplarily, the camera hardware includes a CPU, an ISP, and sensors. Optionally, the hardware may also include devices such as memory. The ISP is used to process images and video streams and output the processed video streams and images in two ways. The CPU is merely an illustrative example; various microcontrollers such as a microcontroller unit (MCU) or devices that function as processors or microcontrollers may be alternatives to the CPU.

[0171] The ISP chip houses processing circuitry and transceiver pins. The processing circuitry controls the pins to send and receive data or signals, enabling communication with the camera's CPU and electronic devices (such as projectors). The processing circuitry also supports applications in the application layer, such as image processing applications.

[0172] The CPU chip is equipped with processing circuitry and transceiver pins. Similarly, the processing circuitry controls the transceiver pins to send or receive data or signals to communicate with electronic devices (such as projectors) and camera ISPs, respectively. Furthermore, the processing circuitry supports applications or functions within applications, such as gesture recognition applications or gesture recognition functions within video applications.

[0173] It should be noted that, in the embodiments of the present application, each application (or module) is described as the main body for implementing each function. In fact, the function of each application is implemented by the processing circuit in the ISP or CPU, and will not be repeated below.

[0174] The sensor is the photosensitive element of the camera, which is used to collect light signals and convert the collected light signals into electrical signals. The electrical signals are then passed to the ISP for processing and converted into images or video streams.

[0175] Optionally, the CPU and ISP can be integrated on the chip or on different chips and connected via a bus. The CPU can respond to the request of the electronic device and output a control signal to the ISP through the control channel between the CPU and the ISP to trigger the corresponding processing circuit (also understood as a module) in the ISP. The CPU can also output data to the ISP through the data channel between the CPU and the ISP, such as the palm coordinates described in the embodiments of the present application. The ISP can output data to the CPU through the data channel between the ISP and the CPU. It should be noted that the control channel and data channel described above can refer to the same physical circuit or different physical circuits, and this application does not limit this.

[0176] Figure 8 is a schematic diagram of an exemplary application scenario. Please refer to Figure 8. A communication connection is established between a first electronic device (for example, a projector) and a second electronic device (for example, a tablet). The second electronic device triggers the screen projection function in response to the received user operation, such as starting the screen projection application. The second electronic device sends the screen projection data to the first electronic device. The screen projection data can also be called a screen projection image, etc., which is not limited in this application. The first electronic device receives the screen projection data, encodes and decodes the screen projection data, and the first electronic device projects the screen projection data, that is, projects the screen projection data onto the screen (or other objects such as a wall). The screen displays the projection picture, wherein the projection picture includes the projection data and other images (optional). Optionally, other images can be generated by the first electronic device, for example, including time information, projection frame, etc., which is not limited in this application. In some instances, the projection picture can also be understood as projection data. In the embodiment of the present application, the projection picture, projection image, and projection data can be arbitrarily replaced, which is not limited in this application.

[0177] In the scenario shown in Figure 8, the interaction between the user and the projection screen is usually completed by the second electronic device, which can also be called the content side or content generation side. For example, if the user wants to adjust the progress of the video displayed in the projection screen, the user needs to adjust it on the second electronic device side. The second electronic device adjusts the video playback progress in response to the received user operation, and generates corresponding projection data based on the adjusted progress, and sends the projection data to the first electronic device. The first electronic device projects the projection data onto the screen, and the projected image on the screen shows the video image after the progress adjustment.

[0178] That is to say, the user interaction method in the screen projection scenario in the existing technology is usually completed on the second electronic device side, and the interaction method is limited to simple operations, such as progress adjustment, volume adjustment, etc.

[0179] The present application provides a control method, electronic device, and system. FIG9 is an exemplary schematic diagram of the principle. Please refer to FIG9. In this method, a first electronic device establishes a communication connection with a second electronic device as a virtual input device. The communication connection can also be referred to as a control channel. The user can point to any position in the projection screen through gestures, pointing tools, laser devices, etc. The first electronic device obtains control information (which can also be referred to as control parameters, operating parameters, etc., which are not limited in this application) based on the image captured by the camera. The control information can be used to indicate the position pointed by the user. The first electronic device transmits the control information to the second electronic device through the control channel. The second electronic device can generate projection data (which can also be referred to as projection image, content data, etc., which are not limited in this application) based on the control information, and transmit the projection data to the first electronic device through the data transmission channel between the first electronic device and the first electronic device. The first electronic device displays the received image. For example, the first electronic device projects the projection data onto the screen, and the projection screen is displayed on the screen. The present application enables the user to reversely control the second electronic device through the first electronic device by virtualizing the first electronic device as an input device. In this way, the amount of control information data sent by the first electronic device to the second electronic device is small, which can effectively reduce the transmission delay and occupy less bandwidth. The second electronic device can instantly generate corresponding projection data based on the control information. In addition, the user can perform reverse control on the first electronic device, achieving an immersive interactive experience and enhancing the user experience.

[0180] Figure 10 is a schematic diagram of a communication connection between a first electronic device and a second electronic device. Referring to Figure 10, a first communication connection and a second communication connection are established between the first electronic device and the second electronic device, wherein the first communication connection can be referred to as a data transmission channel or a data communication connection, which is used by the second electronic device to transmit screen projection data to the first electronic device. The second communication connection can be referred to as a reverse control channel or a control information transmission channel, which is used by the first electronic device to transmit control information to the second electronic device. The control information can also be referred to as reverse control information, which is generated by the first electronic device and transmitted to the second electronic device to reversely control the second electronic device. In the embodiment of the present application, reverse control can be understood as the first electronic device acting as a virtual input terminal of the second electronic device to control the second electronic device, that is, the user can reversely control the second electronic device through the first electronic device. Optionally, the first communication connection can also be used to transmit control information sent by the second electronic device to the first electronic device. The control information can also be understood as forward control information, that is, in the implementation of the present application, the control information is divided into forward and reverse, and the control information sent by the second electronic device to the first electronic device is forward control information. The control information sent by the first electronic device to the second electronic device is reverse control information.

[0181] In the embodiment of the present application, the protocols supported by the first communication connection and the second communication connection may be the same or different. Optionally, the first communication connection may be wired or wireless. The second communication connection may be wired or wireless. For example, the wired connection method of the first communication connection may include but is not limited to: Type C, HDMI, USB, etc. The wired connection method of the second communication connection may include but is not limited to: Type C, USB, etc. The wireless connection method of the first communication connection may include but is not limited to: Wi-Fi, Bluetooth, Star Flash, etc. The wireless connection method of the second communication connection may include but is not limited to: Wi-Fi, Bluetooth, Star Flash, etc.

[0182] Figures 11A to 11D are schematic diagrams of exemplary connection methods. It should be noted that the connection (including physical links and network connections) methods shown in the embodiments of the present application are only illustrative examples and are not limited in this application. Referring to Figure 11A, the HDMI port of the first electronic device is connected to the HDMI port of the second electronic device via a first communication link (a physical link, such as an HDMI cable). The first communication link can be used to support the establishment of a first communication connection, that is, the data or instructions transmitted through the first communication link described in the embodiment of the present application, such as screen projection data, are all transmitted in the first communication link. The USB port of the first electronic device is connected to the USB port of the second electronic device via a second communication link (a physical link, such as a USB cable). The second communication link can be used to support the establishment of a second communication connection (i.e., a reverse control information transmission channel), that is, the data or instructions transmitted through the second communication connection described in the embodiment of the present application, such as reverse control information, are all transmitted in the second communication link. It can be understood that the "links" involved in the embodiments of the present application refer to physical lines, and the "connection" can be understood as a network connection established at the upper layer of the link.

[0183] Referring to Figure 11B, the HDMI port of the first electronic device is connected to the HDMI port of the second electronic device through a first communication link (a physical link, such as an HDMI cable). The first communication link can be used to support the establishment of a first communication connection, that is, the data or instructions transmitted through the first communication link described in the embodiment of the present application, such as screen projection data, are all transmitted in the first communication link. A Bluetooth communication link, that is, a second communication link, can be established between the first electronic device and the second electronic device, which can also be understood as a second communication connection for transmitting reverse control information.

[0184] Please refer to Figure 11C. The USB port of the first electronic device is connected to the USB port of the second electronic device through a communication link (a physical link, such as a USB cable). The communication link supports the first electronic device to establish a first communication connection and a second communication connection with the second electronic device, that is, the data or instructions transmitted through the first communication link described in the embodiment of the present application, such as screen projection data, are all transmitted in the communication link. The data or instructions transmitted through the second communication connection described in the embodiment of the present application, such as reverse control information, are also transmitted in the communication link. In other words, the same physical link can support the upper layer to establish two or more communication connections.

[0185] 11D , the first electronic device establishes a first communication connection and a second communication connection with the second electronic device, wherein both the first communication connection and the second communication connection are Bluetooth connections. Optionally, the second communication connection may be a low-power Bluetooth connection.

[0186] The connection methods and connection types involved in the embodiments of the present application are only illustrative examples. In actual applications, corresponding connections can be established according to the protocol types supported by the first electronic device and the second electronic device, and this application does not limit them. For example, in other embodiments, the first communication connection can be a Bluetooth connection, and the second communication connection can be a Wi-Fi connection. Alternatively, the first communication connection is a Wi-Fi connection, the second communication connection is a Bluetooth connection, etc., and can be any combination of wired and wired, wired and wireless, or wireless and wireless methods, and this application does not limit them.

[0187] In one possible implementation, if the first communication connection and / or the second communication connection are established based on a physical link (such as a USB connection), before establishing the first communication connection and the second communication connection, the user needs to complete the configuration of the physical link, and then the first electronic device and the second electronic device establish a corresponding network connection.

[0188] In the embodiment of the present application, the order in which the second communication connection and the first communication connection are established is not limited. In one example, the first electronic device establishes a first communication connection with the second electronic device, and during the screen projection process, the user can operate the first electronic device to establish a second communication connection with the second electronic device. In another example, the first electronic device can first establish a second communication connection with the second electronic device, and then establish a first communication connection with the first electronic device. In another example, the first electronic device can establish a first communication connection with the second electronic device, but does not perform the screen projection service. The first electronic device establishes a second communication connection with the second electronic device, and then the second electronic device performs the screen projection service with the first electronic device. It can be understood that the first communication connection and the second communication connection in the embodiment of the present application are independent of each other and do not affect each other. The screen projection service and reverse control are also independent of each other and do not affect each other.

[0189] FIG12 is a flow chart showing an exemplary control method according to an embodiment of the present application. Referring to FIG12 , the control method includes but is not limited to the following steps:

[0190] S1201: The first electronic device determines a target mode based on a user operation.

[0191] Exemplarily, after the first electronic device is started, the first electronic device displays a preset screen, and the first electronic device initializes a camera (also referred to as an image sensor module).

[0192] Optionally, the preset screen can be a settings interface of the first electronic device or a desktop image of the first electronic device, which is not limited in this application. In the embodiment of this application, the first electronic device is a projector as an example. After the projector is started, the preset screen can be projected onto the wall facing forward. The preset screen can be the desktop of the projector. The desktop includes, but is not limited to, at least one application icon.

[0193] For example, the user can select the target mode by triggering a function button on the first electronic device, performing gesture control, or using a control device (such as a remote control). In an embodiment of the present application, the target mode is used to indicate that the first electronic device acts as a virtual target-type input device. Target-type input devices include, but are not limited to: a mouse (also referred to as a virtual mouse), a keyboard (also referred to as a virtual keyboard), a stylus (also referred to as a virtual stylus), a game controller (also referred to as a virtual game controller), etc., and are not limited in this application.

[0194] FIG13 is a schematic diagram of an exemplary application scenario. Referring to FIG13 , a user may use a remote control to select a setting application, and the first electronic device may display a control screen in response to the received control signal. The control screen includes at least one mode selection option, including but not limited to: mouse, stylus, keyboard, and more options. The user may select a mouse as the target mode through the remote control. Accordingly, in this scenario, the first electronic device may be connected to other devices as a virtual mouse, or may be understood as being connected to other devices as a virtual mouse-type input device, such as the second electronic device in the embodiment of the present application.

[0195] Exemplarily, the first electronic device determines the target mode in response to a received user operation (which can also be understood as a user instruction). It can be understood that the first electronic device is virtualized into an input device and connected to another electronic device (such as a second electronic device).

[0196] Optionally, after the camera is initialized, it can start collecting images. In one possible implementation, the user can control the first electronic device through gestures or other methods. For example, the user's finger points to the "mouse" mode, the camera collects the image, performs image recognition on the image, determines that the user's gesture points to the "mouse" mode, and starts the mouse mode. The specific implementation method is similar to the method described below and will not be repeated here. It can be understood that before the first electronic device establishes a first communication connection with the second electronic device, that is, before the first electronic device is connected to the second electronic device as an input device (or after the control information transmission channel between the first electronic device and the second electronic device is disconnected), the first electronic device can obtain control information based on the collected image on this end, and perform corresponding operations based on the control information.

[0197] In one possible implementation, the settings application may include a virtual input device startup option. After the user selects to trigger the option, the first electronic device starts the target mode selection function, that is, displays the selection screen shown in Figure 13. In one example, the user can select the corresponding target mode based on the method shown in Figure 13. In another example, after the virtual input device startup option is started, the second electronic device can automatically select the target mode. For example, the second electronic device selects a preset mode as the target mode, where the preset mode can be pre-set. The second electronic device can also use the last selected mode (that is, the most recently selected mode) as the current target mode. Optionally, the first electronic device can also select the corresponding mode based on conditions such as the display mode or device type of the second electronic device. For example, if the second electronic device is a mobile phone, the touch mode can be selected. If the second electronic device is a computer, the mouse or keyboard mode can be recommended.

[0198] In an embodiment of the present application, the user can modify the current mode at any time through the selection screen shown in Figure 13. Optionally, after the target mode is modified, the second electronic device disconnects the second communication connection with the first electronic device, and the first electronic device re-establishes the second communication connection with the second electronic device based on the new target mode, so that the second electronic device recognizes the input device of the updated target type.

[0199] S1202: The first electronic device serves as a virtual input device and establishes a second communication connection (ie, a control information transmission channel) with the second electronic device.

[0200] Exemplarily, after the first electronic device determines the target mode, it virtualizes itself into an input device of the target type (e.g., a virtual mouse type) and establishes a first communication connection with the second electronic device, which can also be understood as establishing a control information transmission channel. Figure 14 is a schematic diagram of an exemplary application scenario. As shown in Figure 14, the first electronic device (e.g., a projector) determines to be virtualized as a mouse in response to a received user operation and establishes a second communication connection with the second electronic device (e.g., a tablet). That is, for the second electronic device, the first electronic device is equivalent to a mouse.

[0201] Optionally, taking the second communication connection as a Bluetooth connection as an example, FIG15 is a schematic diagram of an exemplary Bluetooth connection establishment process, referring to FIG15 , specifically but not limited to:

[0202] S1501: The first electronic device sends a Discovery Request message to the second electronic device.

[0203] Exemplarily, the first electronic device pre-stores descriptors corresponding to different modes (which may also be called target type descriptors, which are not limited in this application). After the first electronic device determines the target mode, it obtains the pre-stored corresponding descriptor. For example, after the first electronic device determines that the target mode is the "virtual mouse" mode, it obtains the descriptor corresponding to the "virtual mouse" mode, such as "function=HIDfunction(gadget,'mouse')", which is used to indicate that the current device is a mouse type input device. The content of the descriptor is only an illustrative example, and different descriptors can be defined according to different operating systems, which is not limited in this application.

[0204] Exemplarily, the first electronic device sends the Discovery Request message in a broadcast manner through a Bluetooth interface.

[0205] Optionally, the Discovery Request message includes but is not limited to: identification information of the first electronic device, Bluetooth address of the first electronic device, descriptor, etc. Exemplarily, the identification information of the first electronic device may be the device name of the first electronic device, etc., which is not limited in this application.

[0206] S1502: The second electronic device sends a Discovery Response message to the first electronic device.

[0207] Exemplarily, after receiving the Discovery Request message sent by the first electronic device, the second electronic device determines that a target type input device, such as a mouse (ie, a mouse-type input device), has been discovered.

[0208] Optionally, a prompt message such as “Mouse discovered” or “Pairing with mouse” may be displayed on the display screen of the second electronic device to indicate that the second electronic device has successfully discovered the first electronic device serving as a virtual mouse device.

[0209] Exemplarily, the second electronic device sends a Discovery Response message to the first electronic device to instruct the first electronic device to establish a Bluetooth connection. Exemplarily, the Discovery Response includes, but is not limited to, identification information of the second electronic device, a Bluetooth address of the second electronic device, and other information.

[0210] S1503: The first electronic device establishes a Bluetooth connection (ie, a second communication connection, which can also be understood as a control information transmission channel) with the second electronic device.

[0211] Illustratively, the first electronic device and the second electronic device transmit information required to establish a Bluetooth connection through multiple Bluetooth signaling interactions. For example, during the multiple interactions in S1503, a Bluetooth encryption key may be negotiated, which can be used to encrypt Bluetooth data during data transmission. The specific interaction process and content are described in the Bluetooth protocol and will not be further elaborated in this application.

[0212] S1504: The first electronic device transmits data with the second electronic device.

[0213] Exemplarily, the first electronic device and the second electronic device may exchange data based on the established Bluetooth connection. For example, the first electronic device may send a Bluetooth data packet to the second electronic device, wherein the Bluetooth data packet includes but is not limited to reverse control information.

[0214] It should be noted that, in the embodiments of the present application, the Bluetooth connection between the second electronic device and the first electronic device may be established with S1501 as the start time of the Bluetooth connection process and S1503 as the completion time of the Bluetooth connection establishment. In other embodiments, the Bluetooth connection between the second electronic device and the first electronic device may also be established with S1502 or S1503 as the start time, which is not limited in the present application.

[0215] Optionally, the Bluetooth connection between the second electronic device and the first electronic device can be maintained through the BLE (Bluetooth Low Energy) protocol. Optionally, the Bluetooth connection between the second electronic device and the first electronic device can be maintained through the classic Bluetooth protocol, including BR (basic rate) and EDR (enhanced data rate). It should be noted that the Bluetooth connection maintained by the BLE protocol supports transmission rates of 1Mbps, 2Mbps, 500Kbps and 125Kbps, and the supported bandwidth is 1MHz or 2MHz. The Bluetooth connection maintained by the BR / EDR protocol supports a maximum transmission rate of 3Mbps and a supported bandwidth of 1MHz.

[0216] In the embodiment of this application, only the process of establishing a Bluetooth connection is used as an example to illustrate. As mentioned above, the second communication connection in the embodiment of this application can support any protocol. The specific establishment process can refer to the corresponding protocol content, and this application does not limit it.

[0217] FIG16A is a diagram illustrating an exemplary user interface. Referring to FIG16A , a Bluetooth settings interface 1501 of a second electronic device includes, but is not limited to, a paired device list 1502. Paired device list 1502 includes "Meet." The second electronic device has been successfully paired with at least one Bluetooth device. A successfully paired Bluetooth device can also be understood as a device that has established a Bluetooth connection with the second electronic device.

[0218] The paired devices in this example include a projector and a mouse. Among them, the projector and the mouse are both first electronic devices. It can be understood that when the first electronic device establishes a first communication connection with the second electronic device, the descriptor it sends is used to indicate that the first electronic device is an output device of the projector type, the second electronic device establishes a first communication connection with the first electronic device, and identifies the object connected to the first communication connection (i.e., the first electronic device) as a projector. When the first electronic device establishes a second communication connection with the second electronic device, the descriptor it sends is used to indicate that the first electronic device is an input device of the mouse type, the second electronic device establishes a second communication connection with the first electronic device, and identifies the object connected to the second communication connection (i.e., the first electronic device) as a mouse. In this scenario, the first communication connection and the second communication connection are illustrated as an example of Bluetooth connection. As mentioned above, the first communication connection and the second communication connection can support any protocol, and this application does not limit it.

[0219] It should be noted that the establishment time of the first communication connection in the embodiment of the present application can be before the establishment of the second communication connection, or after the establishment of the second communication connection, which is not limited in this application. Optionally, after the first communication connection is established, the second electronic device can cast the screen of the first electronic device through the first communication connection, or it can temporarily not cast the screen, and can perform corresponding services according to the actual scenario, which is not limited in this application.

[0220] Figure 16B is an exemplary user interface diagram. Please refer to Figure 16B. After the first electronic device establishes a second communication connection with the second electronic device as a virtual mouse input device. The second electronic device identifies the first electronic device as a virtual mouse. A cursor can be displayed in the display interface of the second electronic device (the cursor position can be a default position, which is not limited in this application). Insurance is only illustrated by virtualizing the first electronic device as a mouse as an example. In an embodiment of the present application, the first electronic device can also be virtualized into other types of input devices such as a stylus and a game controller. The second electronic device can display a corresponding icon or image on the display screen. For example, if the first electronic device is virtualized as a stylus. The image corresponding to the stylus can be displayed on the display screen of the corresponding second electronic device, for example, a paintbrush can be displayed.

[0221] For example, FIG17 is a schematic diagram of an application scenario shown as an example. Please refer to FIG17 and assume that the first electronic device and the second electronic device have established a first communication connection and a second communication connection, and the second electronic device is sending projection data to the first electronic device through a communication connection. In combination with the user interface schematic diagram shown in FIG16B, the image generated by the second electronic device includes a display image and a cursor. The second electronic device sends projection data to the first electronic device, and the projection data corresponds to the image currently displayed by the second electronic device, that is, including a display image and a cursor. Among them, the display image is an image generated based on the video content. The first electronic device projects the projection data onto the screen, and the projection picture is displayed on the screen, and the picture includes a display image and a cursor. It can be understood that the image displayed on the projection picture is consistent with the image displayed by the first electronic device. Of course, in the embodiment of the present application, only the synchronous display of the image by the first electronic device is used as an example for explanation. In other embodiments, during the process of the first electronic device projecting to the second electronic device, the first electronic device may also be in a black screen state. That is, the corresponding projection data is generated in the background.

[0222] S1203: The first electronic device obtains control information based on the image captured by the camera, and sends the control information to the second electronic device through the second communication connection.

[0223] In an embodiment of the present application, the first electronic device obtains an image captured in real time by a camera. Optionally, the shooting range of the camera is greater than or equal to the display range of the projection screen.

[0224] Exemplarily, the user can achieve reverse control through gestures, pointing tools, etc. in front of the camera, that is, at any position between the camera and the projected screen. The user can also use a laser device (such as infrared) to achieve reverse control. Accordingly, the image captured by the camera includes a display image and a target object image. Among them, the display image refers to the projection screen, that is, the image projected by the projection data sent by the first electronic device. The target object image refers to the image corresponding to the user gesture, pointing tool and laser mapping point.

[0225] Take a user gesture as an example. As shown in Figure 9, the user's finger is in front of the camera of the first electronic device, that is, between the camera of the first electronic device and the projected image. Accordingly, the image captured by the camera of the first electronic device includes the projected image and the image corresponding to the user gesture.

[0226] Taking a laser device as an example, Figure 18A is a schematic diagram illustrating an exemplary application scenario. Referring to Figure 18A , a user uses a laser device to emit a laser, and the emitted laser light is mapped to the projected image on the screen, i.e., the laser landing point in the projected image. Optionally, the projected image also includes a cursor.

[0227] Exemplarily, the first electronic device can perform image recognition on the acquired image, identify the target object image, and further use absolute pointing (AP) positioning technology to obtain the absolute pointing position coordinates of the target object image in the projection screen, which can also be called absolute pointing position information, position information, position parameters, etc., which are not limited in this application. It can be understood that the position information acquired by the first electronic device is the position (or position coordinates) indicated by the user in the coordinate system where the displayed image is located. Compared with the relative pointing technology corresponding to traditional mouse control (that is, the electronic device acquires the movement distance and direction of the mouse in space, and then maps it to the coordinate system of the image display), this application uses absolute pointing positioning technology to achieve high-precision positioning, which can greatly improve the user experience and achieve consistency in what is seen, felt, and controlled. For example, the cursor displayed in the image can move with the position of the user's finger.

[0228] FIG19A is an exemplary diagram of module interaction. Please refer to FIG19A. For example, the camera uploads the captured image to the control information generation module through the USB interface. The control information generation module recognizes the image. The position coordinates of the laser landing point in the coordinate system corresponding to the projection screen (i.e., the absolute directional position coordinates) are obtained. It can also be understood as obtaining the corresponding position coordinates based on the relative position of the image of the laser landing point in the projection screen and the projection screen.

[0229] For example, as shown in Figure 20, the control information generation module performs coordinate transformation on the absolute directivity coordinates to obtain the system coordinates. For example, the control information generation module calculates the center method by brightness differentiation:

[0230] 1. Assume that the maximum brightness value of the projection light machine when projecting the display content (i.e., projection data) onto the projection area (i.e., projection screen) is L0, and the laser pen projects a laser spot into the projection area, and the brightness value L1 is higher than L. The control information generation module pre-sets the threshold value L2 (L_0+t_0<=L2<=L1-t_1, t_0 and t_1 are preset division coefficients for noise resistance and robustness improvement), records the pixel coordinate set M whose grayscale value of the image pixel is greater than L2, and calculates the average value of the M coordinates to obtain the center value of the spot (x0, y0). This value is the absolute pointing position coordinate of the laser mapping point. Among them, the coordinates of each pixel are a coordinate system established based on the projection screen. The specific coordinate system and establishment method can be set according to actual needs, and this application does not limit it.

[0231] 2. Homography matrix conversion,

[0232] Homography matrix and calibration: Since the pixel coordinates in the projected image and the system coordinates of the projector projected content belong to two different coordinate systems. In computer vision, the perspective projection from plane to plane conforms to the constraints of the homography matrix A. The specific description of the homography matrix can refer to the existing technical embodiments, and this application will not go into details. Exemplarily, the control information generation module can perform a coordinate transformation on the absolute pointing position coordinates obtained in the previous step to obtain the coordinate position of the laser mapping point in the system. For example, the control module can obtain the coordinates (mx, my) of the converted system cursor based on the following formula: [mx, my] = A*[x0, y0]

[0233] Exemplarily, the information obtained by the control information generation module based on the collected image can be called position information, position parameters, etc., which is not limited in this application.

[0234] In an embodiment of the present application, the control information generation module can also obtain auxiliary parameters based on the auxiliary signal sent by the user using an auxiliary device or by recognizing the user's gestures, which can also be called auxiliary information. For example, the user can click the button on the game controller, click the button on the remote control, use the auxiliary touchpad, etc. The first electronic device can receive the auxiliary signal sent by the auxiliary device to obtain the corresponding auxiliary parameters. For another example, the control information generation module can further perform gesture recognition on the gesture image to obtain the corresponding gesture features. The gesture recognition method can refer to the existing technical embodiments, and this application does not limit it. The control information generation module can determine the corresponding auxiliary information based on the gesture features. For example, the control information generation module pre-stores the auxiliary information corresponding to different gesture features. For example, the auxiliary information corresponding to the pinch gesture is "click the mouse", etc., which can be set according to actual needs and is not limited by this application.

[0235] It can be understood that in mouse mode, users can use gestures or auxiliary devices to trigger different events, such as single-click, double-click, etc. Similarly, in stylus mode, users can use gestures or auxiliary devices to trigger events such as press, move, lift, and multiple selection. In other words, users can control the movement of objects such as fingers and laser mapping points to achieve the movement of the mouse or stylus on the projection screen. Users can further control gestures and auxiliary devices to trigger different events.

[0236] Still referring to FIG. 19A , the control information generation module sends control information, which may also be referred to as reverse control information, reverse control parameters, etc., to the reverse control module, although this application does not limit this. The control information includes, but is not limited to, position information and / or auxiliary information. Specifically, it may include position information, auxiliary information, or both.

[0237] Exemplarily, as described above, the acquisition of position information can also be performed by the visual processing module in the camera. As shown in FIG19B , the visual processing module can acquire the position information, and the camera transmits the position information to the control information generation module via the USB interface. The control information generation module optionally obtains auxiliary information. The control information generation module transmits control information to the reverse control module. The control information includes, but is not limited to, position information and auxiliary information.

[0238] Exemplarily, the reverse control module may generate a corresponding data packet, such as a Bluetooth data packet, based on the connection type of the second communication connection between the first electronic device and the second electronic device, which is not limited in this application. The reverse control module may call a corresponding module (such as a Bluetooth driver) to send a data packet to the first electronic device via the second communication connection, wherein the data packet includes but is not limited to control information (i.e., reverse control information).

[0239] S1204: The first electronic device receives the projection image sent by the second electronic device through the first communication connection, and displays the projection image.

[0240] Exemplarily, the first electronic device sends control information to the second electronic device through the second communication connection, and the second electronic device receives the control information. The second electronic device can generate corresponding projection data based on the position information and / or auxiliary information in the control information, so that the cursor moves to the specified position. Specifically, after the application in the second electronic device generates a display image based on the video content, the view system (refer to Figure 3, or other modules, this application is not limited) can generate a cursor based on the position information. Of course, in the embodiment of the present application, only the movement of the cursor is used as an example for illustration. In other embodiments, the events triggered by the position information and the auxiliary information are different, and the way the second electronic device generates projection data is also different. It can be implemented according to the specific scenario, and this application is not limited.

[0241] The second electronic device sends the projection data to the first electronic device through the first communication connection. The first electronic device receives the projection data and projects the projection data onto the screen, which displays the projection image. Figure 18B is a schematic diagram of an exemplary application scenario. Please refer to Figure 18B and Figure 18A. In Figure 18A, the cursor is at a first position in the projection image, and the laser mapping point is at a second position in the projection image. In the image shown in Figure 18B, the second electronic device moves the cursor based on the position information and moves the cursor to the specified position. Correspondingly, in the projection image, the specified position to which the cursor moves is the second position where the laser mapping point is located.

[0242] In one possible implementation, the user's gesture and pointing tool (such as a pointer, etc.) can be at any position between the projector's camera and the projected image, as shown in FIG21A. In this scenario, the first electronic device can identify the mapping point of the user's index finger (i.e., the tracking point) on the projected image based on the acquired image, and further obtain the position information corresponding to the mapping point. It can be understood that in this example, there may be a difference between the user's gesture and the actual mapping point (which can also be understood as the display point of the cursor). Optionally, the user's finger and the pointing tool can also be attached to the screen, as shown in FIG21B, and there is almost no difference between the corresponding mapping point and the user's finger.

[0243] In an embodiment of the present application, a user can control the movement of a cursor in a projected image by moving a gesture. For example, as shown in FIG22 , the user's finger moves horizontally by a distance of D1. The first electronic device periodically (the period can be the camera sampling interval, for example, 13.3ms, which can be set according to actual needs and is not limited in this application) obtains images captured by the camera and obtains the position information corresponding to the finger image (for example, the tip of the index finger, which is the tracking point) in each image frame. The first electronic device transmits the position information obtained each time to the second electronic device, and the second electronic device can generate corresponding projection data based on each position information and send it to the first electronic device. The first electronic device continuously projects multiple projection data. Accordingly, from the user's perspective, the mouse in the projected image observed by the user will move continuously, for example, from mapping point 1 to mapping point 2, and the movement distance is d1. Optionally, in mouse mode (other modes may also be the same), after the first electronic device sends the initial position information for the first time, in the subsequent control process, the position information sent by the first electronic device to the second electronic device can also be relative displacement information, used to indicate the length and direction of the movement. That is, the first electronic device may further obtain relative displacement information based on the identified position information, and send the relative displacement information to the second electronic device, and the second electronic device may draw a cursor based on the relative displacement information.

[0244] In one possible implementation, in the projector usage scenario (that is, the second electronic device is a projector), the projector needs to have a vertical projection area to obtain a better projection effect. If it is not vertical (daily use scenarios are usually not completely vertical), the picture will have trapezoidal distortion. The projector can pre-correct the image through keystone correction (KC), which can correct the projected picture from a trapezoid to a rectangle. When the projection surface is a plane, the pre-correction is constrained by the homography matrix (Homograhy). That is, the homography matrix (or equivalent correction information) will be obtained during the trapezoidal correction process.

[0245] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the interaction between various network elements. It can be understood that in order to realize the above functions, the non-control device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0246] In the embodiment of the present application, the control device can be divided into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0247] In the case of dividing the functional modules according to their respective functions, FIG23 shows a possible structural diagram of a control device 2300 involved in the above embodiment. As shown in FIG23, the control device may include: a first acquisition module 2301, a second acquisition module 2303, a communication module 2302, and a display module 2304. The first acquisition module 2301 is used to acquire a target mode, which is used to indicate that the first electronic device serves as a target type input device for the second electronic device; the communication module 2302 is used to establish a first communication connection with the second electronic device based on the target mode; the second acquisition module 2303 is used to acquire a first reverse control parameter indicated by the user in the first image based on the first image captured by the camera; the communication module 2302 is used to send the first reverse control parameter to the second electronic device via the first communication connection; and the display module 2304 is used to display a second image sent by the second electronic device via the second communication connection, the second image being generated by the second electronic device based on the first reverse control parameter.

[0248] In one possible implementation, the first image includes a first display image and an image of a target object, and the image of the target object is within the first display image; wherein the first display image is an image generated by the first electronic device, or the first display image is an image sent by the second electronic device through the second communication connection, and the image of the target object includes at least one of an image corresponding to a user's gesture, an image of a laser point mapped by the user on the first display image using a laser device, and an image corresponding to an indicating tool used by the user.

[0249] In a possible implementation, the second acquisition module 2303 is specifically configured to acquire the absolute pointing position coordinates of the image of the target object in the first display image, where the first reverse control parameter is used to indicate the absolute pointing position coordinates.

[0250] In one possible implementation, the first reverse control parameter includes a position parameter and a first auxiliary parameter, and the second acquisition module 2303 is specifically used to: obtain the absolute pointing position coordinates of the image of the target object in the first display image, the position parameter is used to indicate the absolute pointing position coordinates; perform image recognition on the image of the target object to obtain the first auxiliary parameter.

[0251] In a possible implementation, the second image includes a second display image and a cursor, the second display image is generated by the second electronic device based on display content, and the cursor is generated by the second electronic device based on the first reverse control parameter.

[0252] In one possible implementation, the second acquisition module 2303 is also used to obtain the second reverse control parameter indicated by the user in the third image based on the third image captured by the camera; the communication module 2302 is also used to send the second reverse control parameter to the second electronic device through the first communication connection; the display module is also used to display the fourth image sent by the second electronic device through the second communication connection, and the fourth image is generated by the second electronic device based on the second reverse control parameter; the second reverse control parameter is used to indicate moving the cursor from the first position in the second display image to the second position in the fourth display image.

[0253] In a possible implementation, the third image includes the second image and the image of the target object; the second acquisition module 2303 is specifically configured to acquire the second reverse control parameter based on the absolute pointing position coordinates of the image of the target object in the second display image.

[0254] In one possible implementation, the second acquisition module is used to obtain the first reverse control parameter indicated by the user in the first image based on the first image captured by the camera, and is also used to obtain the second auxiliary parameter based on the second operation triggered by the user using the control device; the communication module is also used to send the second auxiliary parameter to the second electronic device through the first communication connection; wherein, the second image is generated by the second electronic device based on the first reverse control parameter and the second auxiliary parameter.

[0255] In a possible implementation, the communication module 2302 is specifically configured to send a target descriptor to the second electronic device, where the target descriptor is used to indicate that the first electronic device is a target-type input device.

[0256] In a possible implementation, the target type input device includes: a mouse type, a keyboard type, a stylus type, or a handle type.

[0257] In another example, Figure 24 shows a schematic block diagram of a control device 2400 according to an embodiment of the present application. The control device may include a processor 2401 and a transceiver / transceiver pin 2402, and optionally, a memory 2403. The processor 2401 may be configured to execute the steps performed by the first electronic device in each method of the aforementioned embodiments, and control the receive pin to receive signals and the transmit pin to send signals.

[0258] The various components of control device 2400 are coupled together via bus 2404. Bus system 2404 includes not only a data bus but also a power bus, a control bus, and a status signal bus. However, for clarity, the various buses are collectively referred to as bus system 2404 in the figure.

[0259] Optionally, the memory 2403 may be used to store instructions in the aforementioned method embodiment.

[0260] It should be understood that the control device 2400 according to the embodiment of the present application may correspond to the first device in each method of the aforementioned embodiment, and the above-mentioned and other management operations and / or functions of each element in the control device 2400 are respectively for implementing the corresponding steps of each of the aforementioned methods. For the sake of brevity, they will not be repeated here.

[0261] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0262] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program includes at least one section of code, and the at least one section of code can be executed by an electronic device to control the electronic device to implement the above method embodiment.

[0263] Based on the same technical concept, an embodiment of the present application also provides a computer program, which, when executed by an electronic device, is used to implement the above method embodiment.

[0264] The program may be stored in whole or in part on a storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor.

[0265] Based on the same technical concept, the embodiment of the present application further provides a processor, which is used to implement the above method embodiment. The above processor can be a chip.

[0266] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0267] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0268] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A control method, characterized in that: include: Acquire a target mode, where the target mode is used to indicate that the first electronic device serves as a target type input device for the second electronic device; establishing a first communication connection with the second electronic device based on the target mode; Based on a first image captured by a camera, obtaining a first reverse control parameter indicated by a user in the first image; sending the first reverse control parameter to the second electronic device through the first communication connection; A second image sent by the second electronic device through the second communication connection is displayed, where the second image is generated by the second electronic device based on the first reverse control parameter.

2. The method according to claim 1, characterized in that The first image includes a first display image and an image of a target object, and the image of the target object is within the first display image; wherein, the first display image is an image generated by the first electronic device, or the first display image is an image sent by the second electronic device through the second communication connection, and the image of the target object includes at least one of an image corresponding to a user's gesture, an image of a laser point mapped by the user on the first display image using a laser device, and an image corresponding to a pointing tool used by the user.

3. The method according to claim 2, characterized in that The acquiring, based on the first image captured by the camera, a first reverse control parameter indicated by the user in the first image, includes: The absolute pointing position coordinates of the image of the target object in the first display image are acquired, and the first reverse control parameter is used to indicate the absolute pointing position coordinates.

4. The method according to claim 2, characterized in that The first reverse control parameter includes a position parameter and a first auxiliary parameter. The acquiring, based on the first image captured by the camera, the first reverse control parameter indicated by the user in the first image includes: Acquire the absolute pointing position coordinates of the image of the target object in the first display image, where the position parameter is used to indicate the absolute pointing position coordinates; Perform image recognition on the image of the target object to obtain the first auxiliary parameter.

5. The method according to claim 2, characterized in that The second image includes a second display image and a cursor. The second display image is generated by the second electronic device based on display content, and the cursor is generated by the second electronic device based on the first reverse control parameter.

6. The method according to claim 5, characterized in that After displaying the second image sent by the second electronic device through the second communication connection, the method further includes: Based on a third image captured by the camera, obtaining a second reverse control parameter indicated by the user in the third image; sending the second reverse control parameter to the second electronic device through the first communication connection; Display a fourth image sent by the second electronic device through the second communication connection, where the fourth image is generated by the second electronic device based on the second reverse control parameter; the second reverse control parameter is used to instruct moving the cursor from a first position in the second display image to a second position in the fourth display image.

7. The method according to claim 6, characterized in that The third image includes the second image and the image of the target object; The acquiring, based on the third image captured by the camera, a second reverse control parameter indicated by the user in the third image, includes: The second reverse control parameter is acquired based on the absolute pointing position coordinates of the image of the target object in the second display image.

8. The method according to claim 1, characterized in that The method further comprises: In a process of acquiring, based on a first image captured by a camera, a first reverse control parameter indicated by a user in the first image, acquiring, based on a second operation triggered by the user using a control device, a second auxiliary parameter; The sending the first reverse control parameter to the second electronic device through the first communication connection further includes: The second auxiliary parameter is sent to the second electronic device through the first communication connection; wherein the second image is generated by the second electronic device based on the first reverse control parameter and the second auxiliary parameter.

9. The method according to any one of claims 1 to 8, characterized in that The establishing a first communication connection with the second electronic device based on the target mode includes: A target descriptor is sent to the second electronic device, where the target descriptor is used to indicate that the first electronic device is the target type input device.

10. The method according to any one of claims 1 to 9, characterized in that The target type input device includes: a mouse type, a keyboard type, a stylus type, or a handle type.

11. A control device, characterized in that: include: A first acquisition module is configured to acquire a target mode, where the target mode is used to indicate that the first electronic device serves as a target type input device for the second electronic device; a communication module, configured to establish a first communication connection with the second electronic device based on the target mode; A second acquisition module is configured to acquire, based on a first image captured by a camera, a first reverse control parameter indicated by a user in the first image; The communication module is configured to send the first reverse control parameter to the second electronic device via the first communication connection; The display module is configured to display a second image sent by the second electronic device via a second communication connection, where the second image is generated by the second electronic device based on the first reverse control parameter.

12. The device according to claim 11, characterized in that The first image includes a first display image and an image of a target object, and the image of the target object is within the first display image; wherein, the first display image is an image generated by the first electronic device, or the first display image is an image sent by the second electronic device through the second communication connection, and the image of the target object includes at least one of an image corresponding to a user's gesture, an image of a laser point mapped by the user on the first display image using a laser device, and an image corresponding to a pointing tool used by the user.

13. The device according to claim 12, characterized in that The second acquisition module is specifically configured to: The absolute pointing position coordinates of the image of the target object in the first display image are acquired, and the first reverse control parameter is used to indicate the absolute pointing position coordinates.

14. The device according to claim 12, characterized in that The first reverse control parameter includes a position parameter and a first auxiliary parameter, and the second acquisition module is specifically configured to: Acquire the absolute pointing position coordinates of the image of the target object in the first display image, where the position parameter is used to indicate the absolute pointing position coordinates; Perform image recognition on the image of the target object to obtain the first auxiliary parameter.

15. The device according to claim 12, characterized in that The second image includes a second display image and a cursor. The second display image is generated by the second electronic device based on display content, and the cursor is generated by the second electronic device based on the first reverse control parameter.

16. The device according to claim 15, characterized in that The second acquisition module is further configured to acquire, based on a third image captured by the camera, a second reverse control parameter indicated by the user in the third image; The communication module is further configured to send the second reverse control parameter to the second electronic device via the first communication connection; The display module is further configured to display a fourth image sent by the second electronic device via the second communication connection, where the fourth image is generated by the second electronic device based on the second reverse control parameter; The second reverse control parameter is used to instruct to move the cursor from a first position in the second display image to a second position in the fourth display image.

17. The device according to claim 16, characterized in that The third image includes the second image and the image of the target object; the second acquisition module is specifically configured to: The second reverse control parameter is acquired based on the absolute pointing position coordinates of the image of the target object in the second display image.

18. The device according to claim 11, characterized in that The second acquisition module is further configured to acquire a second auxiliary parameter based on a second operation triggered by the user using a control device during the process of acquiring the first reverse control parameter indicated by the user in the first image based on the first image captured by the camera; The communication module is further configured to send the second auxiliary parameter to the second electronic device through the first communication connection; wherein the second image is generated by the second electronic device based on the first reverse control parameter and the second auxiliary parameter.

19. The device according to any one of claims 11 to 18, characterized in that The communication module is specifically used for: A target descriptor is sent to the second electronic device, where the target descriptor is used to indicate that the first electronic device is the target type input device.

20. The device according to any one of claims 11 to 19, characterized in that The target type input device includes: a mouse type, a keyboard type, a stylus type, or a handle type.

21. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the method according to any one of claims 1 to 10.

22. A computer storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to perform the method according to any one of claims 1 to 10.

24. A chip, characterized in that: The electronic device comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method according to any one of claims 1 to 10.

Images