Screen projection control method, electronic equipment and storage medium

Abstract

The invention discloses a screen projection control method, electronic equipment and a storage medium, and belongs to the technical field of terminals. The method comprises the following steps: in response to a reverse control operation triggered by a user, establishing a reverse control channel with source end equipment; determining man-machine interaction data of the display end equipment under the condition that a reverse control channel is established between the display end equipment and the source end equipment; according to the data type corresponding to the man-machine interaction data and an input device, target man-machine interaction data is sent to the source end device, the target man-machine interaction data is used for reversely controlling a display interface of the source end device, and the input device is a device adopted by an input operation corresponding to the man-machine interaction data. In this way, the display end device can directly determine to output the target man-machine interaction data to the source end device based on the data type corresponding to the man-machine interaction data input by the user and the input device, recoding and repackaging processing on the man-machine interaction data is avoided, and therefore more convenient interaction experience is achieved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a screen projection control method, an electronic device and a storage medium. Background Art

[0002] At present, the development of the screen projection market is influenced by many factors, such as the popularization of smart terminals, wireless network coverage, and cloud computing support. It has gradually been applied to multiple daily scenarios such as audio and video, and games, from meeting niche needs of business offices.

[0003] As users' demand for screen projection services increases, reverse control technology (sending device A projects the screen to receiving device B, at which time sending device A can control receiving device B, and receiving device B conversely controls sending device A, i.e. reverse control) is also constantly innovating and optimizing to achieve multi-screen interaction, content sharing, remote collaboration and other functions, providing users with a richer audio-visual experience and a more convenient way of working.

[0004] However, the current reverse control process is redundant and can be optimized. Summary of the invention

[0005] The present application provides a screen projection control method, an electronic device and a storage medium. When a display device performs reverse control over a source device, the screen projection control method can directly determine the output of target human-computer interaction data to the source device based on the data type and input device corresponding to the human-computer interaction data input by the user, thereby avoiding re-encoding and re-packaging of the human-computer interaction data, thereby achieving a more convenient interactive experience.

[0006] The technical solution is as follows:

[0007] The first aspect of the present application provides a screen projection control method, which is applied to a display terminal device, comprising:

[0008] In response to a reverse control operation triggered by a user, a reverse control channel is established with the source device;

[0009] Under the condition that a reverse control channel is established between the source terminal device and the display terminal device, determining the human-computer interaction data of the display terminal device;

[0010] According to the data type and input device corresponding to the human-computer interaction data, the target human-computer interaction data is sent to the source device, the target human-computer interaction data is used to reversely control the display interface of the source device, and the input device is the device used for the input operation corresponding to the human-computer interaction data.

[0011] In the screen projection control method provided in the embodiment of the present application, when the user performs a reverse control operation on the display device, the display device will apply to the source device to establish a reverse control channel so as to transmit data between the display device and the source device. After the reverse control channel is established, the display device will obtain local human-computer interaction data, for example, including user clicks, slides, stylus input, keyboard input, etc. The display device will analyze the obtained human-computer interaction data to determine its corresponding data type (such as screen projection reverse control data, local control data, etc.) and input device (such as touch screen, mouse, keyboard, etc.).

[0012] Afterwards, the display device filters out the target human-computer interaction data that will be used to reverse control the source device based on the data type and input device. Compared with the prior art, the present application directly determines the output of the target human-computer interaction data to the source device based on the data type and input device of the local human-computer interaction data of the display device, thereby optimizing the redundancy in the reverse control processing process and eliminating the need to re-encode and re-package the human-computer interaction data, thereby achieving a more convenient reverse control experience.

[0013] According to the first aspect, or any implementation of the first aspect above, before establishing a reverse control channel with the source device in response to a reverse control operation triggered by a user, the method further includes:

[0014] In response to the screen projection request sent by the source device, establish a wireless screen projection connection with the source device;

[0015] When a wireless screen projection connection is established with the source device, a screen projection display interface is displayed, wherein the screen projection display interface at least includes the content requested for screen projection display by the source device.

[0016] According to the first aspect, or any implementation of the first aspect above, in the case of establishing a reverse control channel with the source device, determining the human-computer interaction data of the display device includes:

[0017] In a case where a reverse control channel is established with the source device, determining an input operation received by the display device;

[0018] Corresponding human-computer interaction data is determined based on the input operation, wherein the input operation includes: a local input operation and a peripheral input operation.

[0019] In this implementation, during the reverse control process of screen projection, the display device can accurately obtain the user's input operations (control operations), classify and distinguish the input operations, and convert them into corresponding human-computer interaction data.

[0020] According to the first aspect, or any implementation of the first aspect above, determining the corresponding human-computer interaction data based on the input operation includes:

[0021] In the case where the input operation is the peripheral input operation, determining original input data corresponding to the peripheral input operation and a peripheral driver corresponding to the peripheral input device adopted by the peripheral input operation;

[0022] The peripheral device driver is used to encapsulate the original input data corresponding to the peripheral device input operation to obtain the human-computer interaction data.

[0023] During the screen projection control process, the original input data of the peripheral input operation can be accurately obtained and converted into the data format required for human-computer interaction to achieve more precise screen projection control.

[0024] According to the first aspect, or any implementation of the first aspect above, determining the corresponding human-computer interaction data based on the input operation includes:

[0025] In a case where the input operation is the local input operation, determining original input data corresponding to the local input operation;

[0026] Performing input event conversion processing on the original input data corresponding to the local input operation to obtain an interaction event corresponding to the local input operation;

[0027] Through the screen projection display interface process of the display terminal device, the interaction event corresponding to the local input operation is encapsulated to obtain the human-computer interaction data.

[0028] According to the first aspect, or any implementation of the first aspect above, before sending the target human-computer interaction data to the source device according to the data type and input device corresponding to the human-computer interaction data, the method further includes:

[0029] In a case where the human-computer interaction data acts on the projection display interface of the display terminal device, determining that the data type corresponding to the human-computer interaction data is projection reverse control data;

[0030] In the case where the human-computer interaction data acts on the local service interface of the display terminal device, it is determined that the data type corresponding to the human-computer interaction data is local control data.

[0031] According to the first aspect, or any implementation of the first aspect above, before sending the target human-computer interaction data to the source device according to the data type and input device corresponding to the human-computer interaction data, the method further includes:

[0032] In the case where the input operation received by the display terminal device is an external input operation, determining that the input device corresponding to the human-computer interaction data is an external input device, the external input device includes at least one of the following: a stylus pen and a Bluetooth device;

[0033] In a case where the input operation received by the display terminal device is a local input operation, it is determined that the input device corresponding to the human-computer interaction data is a local input device, and the local input device includes a touch screen.

[0034] In this implementation, according to the data type corresponding to the human-computer interaction data, the display terminal device can identify and distinguish the user's reverse control operation or local operation, thereby improving the intelligence and adaptability of the reverse control.

[0035] According to the first aspect, or any implementation of the first aspect above, sending the target human-computer interaction data to the source device according to the data type and input device corresponding to the human-computer interaction data includes:

[0036] When the data type is screen projection reverse control data and the input device is an external input device, determining that the human-computer interaction data is the target human-computer interaction data;

[0037] Sending the target human-computer interaction data to the screen projection display interface process of the display terminal device through the peripheral driver corresponding to the peripheral input device;

[0038] The target human-computer interaction data is sent to the source device through the screen projection display interface process.

[0039] According to the first aspect, or any implementation of the first aspect above, when the data type is screen projection reverse control data and the input device is an external input device, determining that the human-computer interaction data is the target human-computer interaction data includes:

[0040] In the case where the peripheral input device is a Bluetooth device, detecting a key value to be responded to contained in the human-computer interaction data;

[0041] In a case where the key value to be responded to is a key value to be responded to by the source device, the human-computer interaction data is determined to be the target human-computer interaction data.

[0042] According to the first aspect, or any implementation of the first aspect above, when the peripheral input device is a Bluetooth device, after detecting the key value to be responded to contained in the human-computer interaction data, the method further includes:

[0043] In a case where the key value to be responded to is a key value to be responded to by the display terminal device, decapsulating the human-computer interaction data to obtain an interaction event corresponding to the human-computer interaction data;

[0044] Sending the interaction event to the local service interface process of the display terminal device;

[0045] The interaction event is applied to the corresponding local service through the local service interface process.

[0046] According to the first aspect, or any implementation of the first aspect above, sending the target human-computer interaction data to the source device according to the data type and input device corresponding to the human-computer interaction data includes:

[0047] When the data type is screen projection reverse control data and the input device is a local input device, determining that the human-computer interaction data is the target human-computer interaction data;

[0048] The target human-computer interaction data is sent to the source device through the screen projection display interface process of the display device.

[0049] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0050] When the data type is local control data and the input device is an external input device, decapsulating the human-computer interaction data to obtain a corresponding interaction event;

[0051] Sending the interaction event to the local service interface process of the display terminal device;

[0052] The interaction event is applied to the corresponding local service through the local service interface process.

[0053] In a second aspect, an electronic device is provided, comprising a module / unit for executing the first aspect or any one of the methods in the first aspect.

[0054] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0055] In a third aspect, an electronic device is provided, comprising one or more processors and a memory;

[0056] The memory is coupled to one or more processors, and the memory is used to store computer program code, wherein the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the first aspect or any one of the methods in the first aspect.

[0057] In a fourth aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the first aspect or any one of the methods in the first aspect.

[0058] In a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the first aspect or any one of the methods in the first aspect.

[0059] In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program code, when the computer program code is executed by an electronic device, the electronic device executes the first aspect or any one of the methods in the first aspect.

[0060] The present application provides a screen projection control method, an electronic device and a storage medium, which belongs to the field of terminal technology. The method includes: in response to a reverse control operation triggered by a user, establishing a reverse control channel with a source device; in the case of establishing a reverse control channel with the source device, determining the human-computer interaction data of the display device; according to the data type and input device corresponding to the human-computer interaction data, determining to output the target human-computer interaction data to the source device, the target human-computer interaction data is used to reversely control the display interface of the source device, and the input device is the device used for the input operation corresponding to the human-computer interaction data. The display device can directly determine to output the target human-computer interaction data to the source device based on the data type and input device corresponding to the human-computer interaction data input by the user, avoiding re-encoding and re-packaging of the human-computer interaction data, thereby achieving a more convenient interactive experience.

[0061] Through the above technical solution, since the electronic device can execute or implement the above screen projection control method, it at least has all the beneficial effects of the screen projection control method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application;

[0063] Figure 2 is a software structure block diagram of an exemplary electronic device 100;

[0064] Figure 3 A schematic diagram of an optional screen projection reverse control process provided for this application example;

[0065] Figure 4 It is a flowchart of triggering screen projection control provided in an embodiment of the present application;

[0066] Figure 5 is a schematic diagram of a reverse control process based on a stylus provided in an embodiment of the present application;

[0067] Figure 6 is a schematic diagram of a flow chart of a local control based on a stylus provided in an embodiment of the present application;

[0068] Figure 7 It is a flowchart of reverse control based on a Bluetooth keyboard provided in an embodiment of the present application;

[0069] Figure 8 It is a flowchart of a local control based on a Bluetooth keyboard provided in an embodiment of the present application;

[0070] Fig. 9 This is a schematic diagram of a flow chart of a reverse control based on a touch screen provided in an embodiment of the present application;

[0071] Fig.10 It is a flowchart of another local control based on a Bluetooth keyboard provided in an embodiment of the present application;

[0072] Fig.11 An optional logic diagram of screen projection reverse control provided for this application example. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0074] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is just a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0075] In addition, in order to facilitate the clear description of the technical solution of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. The terms "first" and "second", etc. in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe the specific order of objects. For example, the first target object and the second target object, etc. are used to distinguish different target objects, rather than to describe the specific order of target objects.

[0076] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0077] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0078] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0079] Screen projection refers to the technology of wirelessly transmitting the screen content of mobile phones, tablets, computers and other devices to large screens such as TVs and projectors. It is also called wireless screen projection, screen mirroring, screen sharing, etc.

[0080] User Input Back Channel (UIBC) is a function that sends input commands from the receiving end to the sending end for processing. There are two types of protocols supported by the channel: hardware-independent generic (for smartphones) and human interface device control HID (for PCs).

[0081] Wireless connection Wi-Fi Display (WFD) is a specification developed by the Wi-Fi Alliance that enables multimedia devices (such as mobile phones / mobile PCs and TVs / monitors) to establish and maintain a Wi-Fi-based connection and use this connection to promote the presentation and playback of video / audio on the target device.

[0082] Miracast, a wireless display standard, is a wireless display standard based on Wi-Fi Direct. Consumer electronics products (also known as 3C devices) that support this standard can share video images wirelessly. For example, a mobile phone can use Miracast to play videos or photos directly on a TV or other device without any connection cables or wireless hotspots (AP, Access Point).

[0083] Miracast data devices are divided into the sending end (Source end) and the receiving end (Sink end). The Source end is the sending end of Miracast audio and video data, which is generally a device that generates images such as a GPU, and is responsible for the collection, encoding and sending of audio and video data. The Sink end is the receiving end of Miracast data, which is generally a device such as a display screen, responsible for receiving the audio and video code stream from the Source end and decoding and displaying it, and the connection is made through Wi-Fi Direct technology.

[0084] Human Interface Devices (HID), also known as human-machine interface, is a medium for interaction and information exchange. It realizes the conversion between the internal form of information and the form acceptable to humans. Human-machine interface exists in all fields involving human-machine information exchange.

[0085] A touch screen (TP), also known as a "touch screen" or "touch panel", is an inductive liquid crystal display device that can receive input signals such as contacts. When a graphic button on the screen is touched, the tactile feedback system on the screen can drive various connection devices according to a pre-programmed program. It can be used to replace mechanical button panels and create vivid audio and video effects through the LCD display.

[0086] The touch screen TP driver is a driver that calls the touch screen function of the screen device.

[0087] Wi-Fi peer-to-peer (Wi-Fi P2P for short, also known as Wi-Fi-Direct) is a technology launched by the Wi-Fi Alliance based on the original Wi-Fi technology that allows direct connection between devices, allowing users to conduct one-to-one or one-to-many communications without the help of a local area network or AP (Access Point).

[0088] The above is a brief introduction to the nouns involved in the embodiments of the present application, which will not be repeated below.

[0089] See also Figure 1 , Figure 1A schematic diagram of the structure of an electronic device 100 provided for an embodiment of the present application. Optionally, the electronic device 100 may be referred to as a terminal, or may be referred to as a terminal device, and the specific product form of the electronic device 100 may be a smart terminal, such as a mobile phone, a tablet computer, a wearable device, an augmented reality / virtual reality device, a laptop computer, a vehicle-mounted device, a personal digital assistant (PDA), and other electronic devices with a reverse screen projection control function. Specifically, the functional modules involved in the present application may be deployed on the DSP chip of the relevant device, and may specifically be an application program or software therein. A reverse screen projection control function may be implemented through software installation or upgrade, and through hardware call coordination.

[0090] It should be understood that Figure 1 The electronic device 100 shown is only one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0091] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0092] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0093] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0094] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0095] The USB interface 130 is an interface that complies with USB standard specifications, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.

[0096] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 is charging the battery 142, it can also power the electronic device through the power management module 141. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and powers the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0097] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, the baseband processor, and the like.

[0098] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas.

[0099] The mobile communication module 150 may provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0100] The wireless communication module 160 can provide wireless communication solutions for application in the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0101] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0102] The electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor, etc. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0103] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0104] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

[0105] The ISP is used to process data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye.

[0106] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0107] Among them, the camera 193 can be located in the edge area of ​​the electronic device, can be an under-screen camera, or can be a liftable camera. The camera 193 can include a rear camera, and can also include a rear camera. The embodiment of the present application does not limit the specific position and form of the camera 193. The electronic device 100 can include cameras of one or more focal lengths. For example, cameras of different focal lengths can include a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, or a panoramic camera.

[0108] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement a data storage function.

[0109] The internal memory 121 can be used to store computer executable program codes, and the above-mentioned executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, such as enabling the electronic device 100 to implement the task processing method in the embodiment of the present application. The internal memory 121 may include a storage program area and a storage data area. Among them, the storage program area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The storage data area may store data (such as audio data, a phone book, etc.) created during the use of the electronic device 100, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0110] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170 and the application processor.

[0111] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0112] The touch sensor is also called a "touch panel". The touch sensor can be set on the display screen 194. The touch sensor and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor is used to detect a touch operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194.

[0113] The pressure sensor is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor can be disposed on the display screen 194. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor.

[0114] The gyro sensor may be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor.

[0115] The accelerometer can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the accelerometer can detect the magnitude and direction of gravity. The accelerometer can also be used to identify the posture of the electronic device, and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0116] The key 190 includes a power key (or power key), a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0117] Next, the software system of the electronic device 100 is described. The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0118] like Figure 2The software structure diagram of the electronic device 100 is shown as an example. The layered architecture of the electronic device 100 divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, which are the application layer, the application framework layer, the Android runtime (Android runtime), the system layer, and the kernel layer from top to bottom.

[0119] The kernel layer is the layer between the hardware and the above software layers. The kernel layer contains at least display drivers, camera drivers, and sensor drivers. The hardware may include devices such as cameras, displays, microphones, processors, and memory.

[0120] The application layer can include a series of application packages, such as Figure 2 As shown, the application package may include applications such as gallery, creation assistant, quick application engine, light editing service, etc. The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer, including various components and services to support developers' Android development. The application framework layer includes some predefined functions. The application framework layer may include a window manager, content provider, notification manager, resource manager, search engine, learning memory library, and visual image module, etc.

[0121] Understandably, Figure 2 The components included in the system framework layer, system library and runtime layer shown do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine some components, or split some components, or arrange the components differently.

[0122] An embodiment of the present application provides a screen projection control method that can be applied to a display terminal device. The screen projection control method includes: in response to a reverse control operation triggered by a user, establishing a reverse control channel with a source terminal device. When a reverse control channel is established with the source terminal device, determining the human-computer interaction data of the display terminal device. According to the data type and input device of the human-computer interaction data, determining to output target human-computer interaction data to the source terminal device, the target human-computer interaction data is used to reversely control the display interface of the above-mentioned source terminal device, and the input device is a device used for the input operation corresponding to the human-computer interaction data.

[0123] Exemplarily, the display end device and the source end device may include: mobile phones, tablet computers, smart watches, laptop computers, virtual and real fusion devices, super mobile personal computers, smart TVs, smart screens, high-definition TVs, smart speakers, smart projectors, etc. The embodiments of the present application can be applied to scenarios where various display end devices perform reverse control of user input on source end devices.

[0124] It is understandable that screen projection is also called the same screen, screen sharing, and multi-screen interaction. Specifically, it is to display the screen of the source device (such as a mobile phone or computer) in "real time" on another display device (such as a projector, conference tablet, splicing screen, electronic whiteboard, Pad, TV) through a certain technical method. The output content includes various multimedia information such as albums, videos, audio, etc., and also includes the real-time operation screen on the source device.

[0125] In one example, the source device is a screen projection sending end (also called a sending end device, Source end), which can use a Windows operating system. The screen projection protocol used by the source device can be, but is not limited to, a Miracast screen projection protocol, or a screen mirroring method.

[0126] In one implementation, the display device is the screen projection receiving device (also called the receiving device, Sink end), and the human-computer interaction data of the display device is HID data input by the human-computer interface device HID of the display device (hereinafter referred to as HID device or input device).

[0127] It is understandable that a human interface device HID refers to any input device used to interact with a computer or other electronic device. Optionally, some common HID devices include: touch screen, mouse, keyboard, game controller, digital pen, scanner, etc.

[0128] Generally speaking, the interaction between HID devices and computers or other electronic devices depends on drivers, which provide instructions and information for HID devices. HID devices usually use standard interface protocols to communicate with computers or other electronic devices, which makes it easy for developers to write drivers to support different devices.

[0129] Figure 3 A schematic diagram of the screen projection reverse control process provided for this application example, refer to Figure 3 As shown, step S101, in the screen projection reverse control scenario, when the display device (for example, a smart phone or a tablet computer) receives a reverse control operation input by the user, for example, when the user clicks a shortcut button, slides a gesture, pulls down the screen, or touches the screen on the display device, the display device will sense this operation, execute step S103, and apply to the source device (for example, a computer PC) to establish a reverse control channel.

[0130] The reverse control channel can be understood as a bidirectional data transmission connection, for example, it can be implemented based on the TCP / IP protocol or other network protocols, and is used for the display end device to reversely control the source end device, rather than just receiving the output of the source end device.

[0131] In step S105, if the user of the source device allows the source device to establish a reverse control channel with the display device, step S107 is executed, and the source device sends a response message of permission or approval to the display device, indicating approval and establishment of the reverse control channel. After that, in step S109, the display device determines the human-computer interaction data (i.e., HID data) generated by the local user operation. These human-computer interaction data correspond to different data types and input devices, and contain the interaction information between the user and the display device, which is used to indicate the user's operation intention.

[0132] In step S111, the display device determines the corresponding data type and input device according to the content of the human-computer interaction data. The data type is used to indicate that the data is projection screen reverse control data or local control data. The input device refers to the device used for the input operation corresponding to the human-computer interaction data, such as a touch screen, mouse, keyboard, etc.

[0133] In step S113, the display device determines the target human-computer interaction data to be output to the source device according to the data type and the input device. The target data is directly obtained by the display device through screening of the human-computer interaction data according to the data type and the input device. The display device does not need redundant decapsulation and recapsulation processing locally. After receiving the target data, the source device executes step S115, that is, the key event can be obtained by parsing or converting it locally on the source device for reverse control of the display interface of the source device.

[0134] By adopting the screen projection control method provided in the example of this application, in response to the reverse control operation triggered by the user, a reverse control channel is established with the source device, which can realize two-way interaction between the display device and the source device, and improve the user's operating convenience and experience. In the case of establishing a reverse control channel with the source device, the human-computer interaction data of the display device is determined, and the user's reverse control operation is identified, which can improve the accuracy of reverse control. By determining the data type and input device corresponding to the human-computer interaction data, and determining to output the target human-computer interaction data to the source device according to the data type and input device, the display device can classify and distinguish the user's control operations, and reduce redundant decapsulation and repackaging in the data processing process, so as to realize reverse control of the source device and improve the compatibility and efficiency of reverse control.

[0135] Before describing the technical solution of the embodiment of the present application, first, in the embodiment of the present application, before establishing a reverse control channel with the source device in response to a reverse control operation triggered by a user, the triggering method of the source device on the screen projection control of the display device is described.

[0136] In one example, taking the source device as a laptop (also called PC) and the display device as a Pad, the PC can use Wi-Fi P2P to search for nearby display devices (including: laptops, smart phones, Pads) that support Wi-Fi P2P. After discovering the display device and pairing it to establish a P2P connection, the user chooses to send a screen projection request to one or more of the display devices.

[0137] Figure 4 A schematic diagram of a process for triggering screen projection control is provided for this application, for example Figure 4 As shown, S401, the user selects to request to project the display interface of the PC to the Pad, S402, the Pad will respond to this request and determine whether to establish a wireless screen projection connection. When a wireless screen projection connection is successfully established (for example, based on wireless technologies such as Wi-Fi or Bluetooth), the PC can transmit data and images to the Pad, and the Pad will display the projection display interface on its display screen, which at least contains the content requested by the PC to be projected (such as desktop, applications, games, etc.). For example, if the PC requests to display a document, video, or application, the Pad will display the content on its screen.

[0138] In another example, taking the source device as a laptop and the display device as a Pad, the user wants to project the screen on the laptop to the Pad. When the Pad and the laptop are in the same local area network, the user turns on the screen projection function on the laptop's control panel and can share the laptop's video screen with the Pad wirelessly.

[0139] For example, a laptop computer supports the Miracast screen projection protocol (a wireless display standard based on Wi-Fi Direct, i.e., Wi-Fi direct connection). When using the Miracast screen projection protocol for screen projection, the user presses the screen projection shortcut key (such as Windows+K) on the laptop computer, and the display devices including the Pad are automatically searched for. The user simply clicks on the device logo of the Pad to play the video or photo on the laptop computer directly on the Pad without any connection cables or sharing through a wireless hotspot (AP, Access Point).

[0140] For example, users can also use the built-in screen projection function on their laptops to directly project the screen to the Pad. The operation method is to pull down the screen projection function notification bar and click the screen projection operation in the notification bar.

[0141] For another example, users can also use the screen projection software on their laptop to directly project the screen to the Pad. The operation method is to click the screen projection button of the video software and directly select the receiving device including the Pad. The user directly clicks the device logo of the Pad to play the video or photo on the laptop directly on the Pad.

[0142] It should be noted that in this application example, the user is allowed to project any content displayed on the source device, including applications, games, videos, etc. Of course, the user can choose the content of the screen projection as needed, making the screen projection operation more flexible.

[0143] This application example can transmit the screen projection display in real time, ensuring that the user can immediately see the operation results on the source device during the screen projection process, with low latency and smoother user experience. In addition, there is no need to install additional third-party applications, and the screen projection control can be achieved using the functions of the source device itself.

[0144] In one example, Figure 4 As shown, in S403, the Pad applies to the PC to establish a reverse control channel. In S404, after the PC agrees to establish the reverse control channel, S405 is executed to send a response message to the display device. When a reverse control channel is established between the source device and the display device, the display device can accurately obtain the user's input operation and convert it into corresponding human-computer interaction data. The display device classifies and distinguishes the user's control operation. For example, according to whether the input device is local to the display end, the input operation of the human-computer interaction data is divided into: local input operation and reverse control operation, and according to the data type, the human-computer interaction data is divided into projection screen reverse control data and local control data.

[0145] The display device (e.g., Pad) will sense various operations performed by the user using various input devices (e.g., keyboard, game controller, digital pen, touch screen, etc.) connected to the display device, such as clicking, selecting, sliding, writing, touching, etc. After determining the user's input operation, the display device will map these operations to the corresponding human-computer interaction data. The input operations of these human-computer interaction data include: local input operations and peripheral input operations. In this way, the display device can support multiple input methods, improving the user's operation convenience and selectivity.

[0146] It is understood that local input operations refer to operations directly processed by the display device itself, such as operations performed by a user directly on the touch screen of the display device, such as clicking, sliding, dragging, etc., and pressing a button, etc. Peripheral input operations refer to operations transmitted to the display device through an external device (such as a keyboard, mouse, stylus, etc.).

[0147] Generally speaking, the interaction between HID devices and computers or other electronic devices depends on drivers, which provide instructions and information for HID devices. HID devices usually use standard interface protocols to communicate with computers or other electronic devices, which makes it easy for developers to write drivers to support different devices.

[0148] In an optional example, when the input operation is a peripheral input operation, the original input data corresponding to the peripheral input operation and the peripheral driver corresponding to the peripheral input device used by the peripheral input operation are determined; for example, the original input data may involve input from external input devices such as a keyboard, a mouse, and a stylus, such as the position of a mouse, a key on a keyboard, the pressure of a stylus, etc. The original input data corresponding to the peripheral input operation is encapsulated using the peripheral driver to obtain the human-computer interaction data.

[0149] It is understandable that the peripheral driver is responsible for processing peripheral input operations and converting them into a usable data format. In this step, the original input data is encapsulated into data suitable for human-computer interaction, which is to convert the original input data into a format and protocol that conforms to the electronic device's recognition and reception, such as converting the mouse position into the distance and direction of the mouse movement, converting the keyboard keys into ASCII codes, etc., so that they can be processed and displayed during the screen projection process, ensuring that during the screen projection control process, the original input data of the peripheral input operation can be accurately obtained and converted into the data format required for human-computer interaction, so as to achieve more accurate screen projection control.

[0150] Schematically, determining the corresponding human-computer interaction data based on the user input operation means that the display device generates data for representing the user's operation intention, such as touch coordinates, sliding direction, dragging distance, etc., according to the type and content of the user input operation.

[0151] Still taking the display device as the Pad and the source device as the PC as an example, Figure 5 is a schematic diagram of a reverse control process based on a stylus provided in an embodiment of the present application, such as Figure 5As shown, taking the external input device as a stylus as an example, S501, the original input data of the user operating the stylus (for example, touch screen coordinates, pen inclination, pen direction and other data) is detected. S502, the original input data of the stylus is sent to the touch screen TP driver (also called TP driver). S503, the TP driver encapsulates the above-mentioned original input data of the stylus to obtain HID data (i.e., human-computer interaction data). S504, the TP driver determines whether the HID data acts on the projection display interface, so that it can be determined that the data type of the HID data is projection screen reverse control data or local control data. If the TP driver determines that the HID data acts on the projection display interface of the Pad, that is, it determines that the data type of the HID data is projection screen reverse control data, S505, the TP driver directly passes the HID data to the projection display interface process.

[0152] S506, the screen projection display interface process determines the input device and data type of the HID data. If it is determined that the input device of the HID data is a stylus and the data type is screen projection reverse control data, there is no need to re-encapsulate or decapsulate the HID data. Directly execute S507 to forward the HID data to the PC device. In S508, the PC can parse the HID data or convert it into a Key event.

[0153] As you can understand, the key event returns the key identifier when the key is pressed. The key identifier is a string that represents the keyboard button.

[0154] In another embodiment, when the data type is local control data and the input device is an external input device, the human-computer interaction data can be unpacked to obtain corresponding interaction events; the interaction events can be sent to the local business interface process of the display device, and the interaction events can be applied to corresponding local businesses, such as local settings, local applications, etc., through the local business interface process.

[0155] Still taking the display device as the Pad and the source device as the PC as an example, Figure 6 is a schematic diagram of a flow chart of a local control based on a stylus provided in an embodiment of the present application, such as Figure 6As shown, taking the external input device as a stylus as an example, S601, the original input data of the user operating the stylus (for example, touch screen coordinates, pen tilt, pen direction and other data) is detected. S602, the original input data of the stylus is sent to the touch screen TP driver (also called TP driver). S603, the TP driver encapsulates the above original input data of the stylus to obtain HID data (i.e., human-computer interaction data). S604, the TP driver determines whether the HID data acts on the projection display interface, so that it can be determined whether the data type of the HID data is projection reverse control data or local control data.

[0156] If the TP driver determines that the HID data acts on the local service interface (also called other service interface), S605 is executed to pass the HID data to the HID device. In S607, the HID device decapsulates the HID data to obtain the corresponding input event. In S607, the input event is passed to the local service interface process.

[0157] It is understandable that both the projection display interface and the local service interface refer to the interface displayed on the display device. The difference between the two is that the projection display interface at least includes the content requested by the source device for projection display, such as desktop, application, game, etc. The local service interface at least includes the content of the display device itself, such as settings, menus, icons, etc.

[0158] It still needs to be explained that the projection display interface process and the local business interface process are both instances of programs running in the computer. Each process has its own independent memory space and execution context, including program counters, registers, stacks, etc.

[0159] Still taking the display end device as the Pad end and the source end device as the PC end as an example, Figure 7 A flow chart of reverse control based on a Bluetooth keyboard is shown in FIG. Figure 7 As shown, taking the external input device as a Bluetooth keyboard as an example, S701 detects the original input data of the user operating the Bluetooth keyboard. S702, the Bluetooth keyboard encapsulates the original input data into HID data.

[0160] Then, S703 is executed, and the Bluetooth keyboard transmits the HID data to the HID device of the Pad. In S704, the HID device determines that the HID data acts on the projection display interface of the display device, so that it can be determined that the data type of the HID data is projection reverse control data. Then, S705 is executed, and the HID device sends the HID data to the projection display interface process of the Pad. In S706, the projection display interface process determines that the input device corresponding to the external input operation is a Bluetooth keyboard based on the HID data, and the data type is projection reverse control data.

[0161] Furthermore, if the key value to be responded to by the HID data is the key value to be responded to by the source device, the projection display interface process does not need to re-encapsulate or de-encapsulate the HID data, and directly executes S707 to forward the HID data to the PC device. In S708, the PC can parse the HID data or convert it into a Key event.

[0162] Still taking the display device as the Pad and the source device as the PC as an example, Figure 8 is a flowchart of a local control based on a Bluetooth keyboard provided in an embodiment of the present application, such as Figure 8 As shown, taking the external input device as a Bluetooth keyboard as an example, S801 detects the original input data of the user operating the Bluetooth keyboard. S802, the Bluetooth keyboard encapsulates the original input data into HID data.

[0163] Then execute S803, the Bluetooth keyboard passes the HID data to the HID device of the Pad, S804, the HID device determines that the HID data acts on the local business interface of the Pad, so it can be determined that the data type of the HID data is local control data, then execute S805, the HID device sends the HID data to the input event management module (Framework Input ManagerService, referred to as FIM), S806, the FIM module decapsulates the HID data to obtain the corresponding input event, and executes S807 to pass the input event to the local business interface process.

[0164] As an optional example, when the above-mentioned input operation is a local input operation, the original input data corresponding to the above-mentioned local input operation is determined; the original input data corresponding to the above-mentioned local input operation is converted into input events to obtain the interaction events corresponding to the above-mentioned local input operation; the screen projection display interface process of the above-mentioned display terminal device is used to encapsulate the interaction events corresponding to the above-mentioned local input operation to obtain the above-mentioned human-computer interaction data.

[0165] It should be noted that since the interface displayed on the current display device is a projection display interface, when the input device is a local display screen, the human-computer interaction data input by the user on the display screen is generally projection reverse control data, regardless of the case where the data type is local control data. When the input device is a peripheral (for example, a Bluetooth device, a stylus, etc.), the specific subdivision considers the data type as local control data and reverse control data.

[0166] Optionally, the above-mentioned local input operation refers to an operation performed directly by a user on the touch screen of the display device, such as clicking, sliding, dragging, etc.

[0167] Optionally, the above-mentioned original input data refers to the data generated by the display device according to the user's operation of the touch screen, which is used to represent the state and parameters of the touch screen, such as the position of the touch point, the number of touch points, the pressure of the touch point, etc.

[0168] Optionally, input event conversion processing can be used to convert raw input data into Input events that are recognized and received by the source device, such as converting the position of a touch point into the position of a mouse click, converting the number of touch points into the number of mouse clicks, converting the pressure of a touch point into the intensity of a mouse click, etc.

[0169] Optionally, in this example, encapsulation is to convert the interaction event into a format and protocol that can be recognized and received by the source device, such as converting the position of the mouse click into coordinates relative to the display interface of the source device, converting the distance and direction of the mouse movement into displacement and angle relative to the display interface of the source device, etc.

[0170] Optionally, the above-mentioned interaction event can be understood to refer to an event generated by the display terminal device according to the input event conversion processing and used to represent the user's operation intention.

[0171] Still taking the display end device as the Pad end and the source end device as the PC end as an example, Fig. 9 The schematic diagram of the reverse control process based on the touch screen is shown in FIG. Fig. 9 As shown, in the case where the input operation is a local input operation on the Pad side (i.e., the input device is a touch screen), S901 is executed to detect the original input data of the user operating the touch screen. Since the original input data of the touch screen (which may involve user clicks, slides, gestures, etc. on the touch screen) is processed by the Pad side itself and does not require the driver, S902 is executed to send the original input data of the touch screen to the FIM module. In S903, the FIM module processes the original input data and directly converts it into a more specific interaction event (i.e., input event, for example, converting a sliding gesture on the touch screen into scrolling, zooming, and other operation events). In S904, the FIM module passes the input event to the projection display interface process.

[0172] S905, the screen projection display interface process determines that the data type of the HID data is screen projection reverse control data, and the input device is a touch screen, then execute S906, call the UIBC encapsulation module to encapsulate the input event into HID data, no other decapsulation or re-encapsulation processing is required, S907, the UIBC encapsulation module directly sends the HID data to the PC. S908, the PC uses the internal HID device to parse and process the received HID data to obtain the Key event, so that the process can be simplified to complete the reverse control of the screen projection from the Pad to the PC.

[0173] In one example, the display end device may adopt the following method: before sending the target human-computer interaction data to the source end device according to the data type and input device corresponding to the human-computer interaction data, determine the data type corresponding to the human-computer interaction data, that is, distinguish different data types according to the object of the human-computer interaction data, so as to indicate the purpose and function of the human-computer interaction data, such as projection screen reverse control data or local control data:

[0174] When the human-computer interaction data acts on the projection display interface of the display terminal device, it is determined that the data type corresponding to the human-computer interaction data is projection reverse control data.

[0175] In the case where the human-computer interaction data acts on the local service interface of the display terminal device, it is determined that the data type corresponding to the human-computer interaction data is local control data.

[0176] In an optional implementation, the screen projection reverse control data refers to the data generated by the display device based on the user's reverse control operation to represent the user's operation intention, such as touch coordinates, sliding direction, dragging distance, etc. This data is used to reverse control the display interface of the source device.

[0177] In an optional implementation, local control data refers to data generated by the display device based on the user's local operation that represents the user's operating intention, such as clicking, selecting, and opening an application. This data is used to control the local business of the display device and only affects the local display without involving the control of the source device, such as the local business interface or local application.

[0178] Through the above examples, the display device can identify and distinguish the user's reverse control operation or local operation by determining the data type corresponding to the human-computer interaction data, thereby improving the intelligence and adaptability of the reverse control.

[0179] In one example, the display device may also determine the input device corresponding to the human-computer interaction data in the following manner:

[0180] In the case where the input operation received by the display terminal device is a peripheral input operation, it is determined that the input device corresponding to the human-computer interaction data is a peripheral input device, and the peripheral input device includes at least one of the following: a stylus pen and a Bluetooth device.

[0181] In the case that the input operation received by the display terminal device is a local input operation, it is determined that the input device corresponding to the human-computer interaction data is a local input device, and the local input device includes: a touch screen.

[0182] In the example of this application, the external input operation is the operation performed by the user through the external input device connected to the display terminal device, and the local input operation is the operation performed by the user directly on the touch screen of the display terminal device, such as clicking, sliding, dragging, etc. The display terminal device can distinguish the device type of the input device used by the user according to the input method of the user's reverse control operation. For example, the external input device can be a stylus, a Bluetooth device (for example, a Bluetooth mouse, a Bluetooth keyboard, etc.), and the local input device can be, for example, a touch screen.

[0183] By identifying the input device corresponding to the human-computer interaction data according to the input operation received by the display terminal device, it is possible to identify and distinguish the input device corresponding to the human-computer interaction data input by the user to the display terminal device.

[0184] As an optional embodiment, when the above data type is screen projection reverse control data and the above input device is a peripheral input device, the above human-computer interaction data is determined as the target human-computer interaction data. Afterwards, the target human-computer interaction data is reported to the screen projection display interface process of the display end device through the peripheral driver corresponding to the peripheral input device. Finally, the target human-computer interaction data is output to the source end device through the screen projection display interface process.

[0185] When the data type is screen projection reverse control data, and the input device is a peripheral input device, the human-computer interaction data is determined as the target human-computer interaction data, and the driver is no longer used to decapsulate the HID data, nor is the interface used to recapsulate the HID data. Instead, the peripheral driver corresponding to the peripheral input device used by the peripheral input operation is directly used to report the target human-computer interaction data to the screen projection display interface process of the display device, which can realize the reverse control of the source device by the display device, and improve the user's operating convenience and experience.

[0186] In one example, when the above-mentioned peripheral input device is a Bluetooth device, by detecting the key value to be responded contained in the above-mentioned human-computer interaction data, it can be determined whether the key value to be responded is processed by the display end device (such as Pad) or the source end device (such as PC).

[0187] Optionally, the Bluetooth device is an external device wirelessly connected to the display device via the Bluetooth protocol, such as a Bluetooth mouse, a Bluetooth keyboard, etc. The key value to be responded to is used to indicate the status and parameters of the Bluetooth device, such as the left button, right button, scroll wheel, etc. of a Bluetooth mouse, and letters, numbers, symbols, etc. of a Bluetooth keyboard.

[0188] Assume that the display device is a Pad and the source device is a PC. Figure 7The flowchart of local control based on Bluetooth keyboard is shown. After executing S705, the HID device sends the HID data to the projection display interface process. In S706, the projection display interface process determines that the input device corresponding to the HID data is a Bluetooth keyboard, and the data type is projection reverse control data. The process also detects whether the key value to be responded to contained in the HID data is the key value to be responded to by the source device. In S707, if it is the key value to be responded to by the source device, the projection display interface process does not need to re-encapsulate or decapsulate the HID data, and can directly forward it to the source device for parsing or conversion into Key events.

[0189] In the case where it is determined that the key value to be responded to is the key value to be responded to by the source device, the human-computer interaction data is determined to be the above-mentioned target human-computer interaction data. Since the target human-computer interaction data is used to reversely control the display interface of the source device, that is, the key value to be responded to by the source device requires the source device to make corresponding changes to its display interface according to the user's intention of reverse control operation, for example, the left-click of the Bluetooth mouse corresponds to the mouse click of the display interface of the source device, and the letter input of the Bluetooth keyboard corresponds to the text input of the display interface of the source device.

[0190] As an optional example, when it is determined that the key value to be responded to is the key value to be responded to by the display device, that is, the source device needs to respond to the human-computer interaction data, then the human-computer interaction data needs to be unpacked to obtain the interaction event corresponding to the human-computer interaction data; and the interaction event is sent to the local business interface of the display device.

[0191] In this implementation method, for the implementation scenario of processing human-computer interaction data and acting on the local business interface of the display device, when the key value to be responded to is determined to be the key value to be responded to by the display device, that is, the display device makes corresponding changes to its local business interface (that is, the non-projected display interface, which at least includes the content of the display device itself, such as settings, applications, menus, icons, etc.) according to the user's local operation intention. For example, the left click of the Bluetooth mouse corresponds to the mouse click of the local business interface of the display device, and the letter input of the Bluetooth keyboard corresponds to the text input of the local business interface of the display device, etc.

[0192] Optionally, decapsulation can be understood as converting human-computer interaction data into interaction events that can be recognized and received by the display device, for example, converting the position of a mouse click into coordinates relative to the local business interface of the display device, converting letters entered on the keyboard into ASCII codes, etc.

[0193] Still taking the display end device as the Pad end and the source end device as the PC end as an example, Fig.10 Another flow chart of local control based on Bluetooth keyboard is shown in FIG. Fig.10As shown, after executing S706, the screen projection display interface process determines that the input device corresponding to the HID data is a Bluetooth keyboard and the data type is screen projection reverse control data, the screen projection display interface process also detects whether the key value to be responded to contained in the HID data is the key value to be responded to by the source device, S709, if it is the key value to be responded to by the display device, the screen projection display interface sends the HID data to the UIBC encapsulation module for decapsulation processing, S710, the UIBC encapsulation module forwards the obtained input event to the FIM module. In this example, the FIM module does not need to continue to decapsulate the HID data, but executes S711 to directly pass the input event to the local business interface process.

[0194] It is understandable that in the example of this application, when the key value to be responded to is the key value to be responded to by the display terminal device, it can be no different from other local operations without screen projection. It is still necessary to decapsulate the human-computer interaction data to obtain the interaction event corresponding to the human-computer interaction data. The interaction event acts on the display terminal device itself, and does not need to be sent to the source terminal device. Then, the decapsulated interaction event is directly sent to the local business interface process of the display terminal device, and the interaction event is applied to the corresponding local business through the local business interface process. For example, in the specific implementation, it can be the settings, applications, menus, icons, etc. in the local business interface. The display terminal device can respond to and feedback the user's local operation, and improve the accuracy and sensitivity of the local operation.

[0195] As an optional example, when the data type is screen projection reverse control data and the input device is a local input device, the human-computer interaction data is determined to be the target human-computer interaction data; the screen projection display interface process of the display end device is used to output the above target human-computer interaction data to the source end device.

[0196] As explained above, the data types involved in the examples of this application refer to the display end device distinguishing different data types according to the object of human-computer interaction data, which are used to indicate the purpose and function of human-computer interaction data, such as projection screen reverse control data or local control data. Input device refers to the display end device distinguishing different data types according to the user input operation method, such as a stylus, Bluetooth device (mouse, keyboard) or touch screen.

[0197] When the data type is screen projection reverse control data, and the input device is a local input device (e.g., display screen, touch screen), since the original input data of the touch screen is processed by the Pad itself and does not require a driver, the original input data of the touch screen can be directly converted into an interactive event (input event) and passed to the screen projection display interface. The screen projection display interface then encapsulates the above interactive events into HID data, and directly sends the HID data to the PC without any other decapsulation or re-encapsulation processing. The PC uses an internal HID device to parse and process the received HID data to obtain a Key event, completing the reverse control of the screen projection of the PC by the Pad.

[0198] As mentioned above, the display device uses different drivers or modules to receive and process the operation data of the corresponding input device. For example, the FIM module reports the input events of the touch screen of the display device to the projection display interface process, the TP driver reports the HID events (HID data) of the stylus of the display device to the projection display interface process, and the HID device reports the HID events (HID data) of the stylus of the display device to the projection display interface process.

[0199] like Fig.11 A logic diagram of screen projection control is shown in FIG. Fig.11 As shown, S1101, when the projection display interface process receives the reported human-computer interaction data, it first determines whether a reverse control channel has been opened between the display end device and the source end device.

[0200] Still Fig.11 As shown, S1102, if the reverse control channel has been opened, a specific analysis is performed to determine how to process the human-computer interaction data or event based on the data type and input device.

[0201] For example, in S1103, when the data type is screen projection reverse control data and the input device is a local input device (display screen, input tool is finger), determine to use the human-computer interaction data as the target human-computer interaction data; in S1104, use the screen projection display interface process to output the target human-computer interaction data to the source device.

[0202] For example, in S1105, when the data type is screen projection reverse control data and the input device is a peripheral input device, the driver is not used for decapsulation, and the interface is no longer used for recapsulation, and the human-computer interaction data is determined to be the target human-computer interaction data. S1106, the peripheral driver corresponding to the peripheral input device is used to report the target human-computer interaction data to the screen projection display interface process of the display terminal device; S1107, the screen projection display interface process is used to output the target human-computer interaction data to the source terminal device.

[0203] Specifically, when the peripheral input device is a Bluetooth device, the key value to be responded to contained in the human-computer interaction data is detected; when it is determined that the key value to be responded to is a key value to be responded to by the source device, the human-computer interaction data is determined to be the target human-computer interaction data; when it is determined that the key value to be responded to is a key value to be responded to by the display device, the human-computer interaction data is unpacked to obtain the interaction event corresponding to the human-computer interaction data; and the interaction event is sent to the local business interface process of the display device.

[0204] For another example, in S1108, when the data type is local control data and the input device is a peripheral input device, the human-computer interaction data is unpacked to obtain the corresponding interaction event; in S1109, the interaction event is sent to the local business interface process of the display device.

[0205] For another example, S1110, if the reverse control channel is not opened between the display end device and the source end device, determine whether the data type of the human-computer interaction data or event is a HID event. If so, execute S1111 to decapsulate the human-computer interaction data. If not, execute S1112 and directly send it to the display end device for local processing.

[0206] It can be understood that since the interface displayed on the current display device is a projection display interface, when the input device is a local display screen, the human-computer interaction data input by the user on the display screen is generally projection reverse control data, regardless of the case where the data type is local display data. When the input device is a peripheral (for example, a Bluetooth device, a stylus, etc.), the specific subdivision considers the data type as local control data and reverse control data.

[0207] An embodiment of a screen projection control device provided in an embodiment of the present application can be implemented by software, hardware or a combination of both to become part or all of a computer device, and the computer device can be Figure 1 The screen projection control device is applied to a display terminal device, and includes:

[0208] A request unit, configured to establish a reverse control channel with a source device in response to a reverse control operation triggered by a user;

[0209] A determination unit, configured to determine the human-computer interaction data of the display terminal device when a reverse control channel is established with the source terminal device;

[0210] The output unit is used to send target human-computer interaction data to the source device according to the data type and input device corresponding to the human-computer interaction data. The target human-computer interaction data is used to reversely control the display interface of the source device. The input device is the device used for the input operation corresponding to the human-computer interaction data.

[0211] It should be noted that the screen projection control device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0212] The functional units and modules in the above embodiments may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above integrated units may be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0213] The screen projection control device and the screen projection control method embodiments provided in the above embodiments belong to the same concept. The specific working process of the units and modules in the above embodiments and the technical effects brought about can be found in the method embodiment part and will not be repeated here.

[0214] An embodiment of the present application further provides an electronic device, the electronic device comprising one or more processors and a memory;

[0215] The memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the electronic device to execute the screen projection control method shown above.

[0216] The electronic device may be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, or a communication device such as a server, a storage device, a base station, or a smart car, etc. The embodiments of the present application do not impose any restrictions on the specific type of the electronic device.

[0217] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium is run on an electronic device, the electronic device executes the screen projection control method shown above.

[0218] The above-mentioned computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The above-mentioned computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integrated therein. The above-mentioned available media may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0219] An embodiment of the present application also provides a computer program product including computer instructions, which, when executed on an electronic device, enables the electronic device to execute the screen projection control method shown above.

[0220] The computer storage medium and computer program product provided in the above-mentioned embodiments of the present application are used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the method provided above, and will not be repeated here.

[0221] In the above embodiments, it can also be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above-mentioned computer program product includes one or more computer instructions. When the above-mentioned computer instructions are loaded and executed on a computer, the above-mentioned process or function according to the embodiment of the present application is generated in whole or in part. The above-mentioned computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the above-mentioned computer instructions can be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The above-mentioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, digital versatile discs (DVD)), or semiconductor media (eg, solid state disks (SSD)).

[0222] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

[0223] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A screen projection control method, characterized in that: Applied to display devices, including: In response to a reverse control operation triggered by a user, a reverse control channel is established with the source device; Under the condition that a reverse control channel is established between the source terminal device and the display terminal device, determining the human-computer interaction data of the display terminal device; According to the data type and input device corresponding to the human-computer interaction data, the target human-computer interaction data is sent to the source device, the target human-computer interaction data is used to reversely control the display interface of the source device, and the input device is the device used for the input operation corresponding to the human-computer interaction data.

2. The method according to claim 1, characterized in that Before establishing a reverse control channel with the source device in response to the reverse control operation triggered by the user, the method further includes: In response to the screen projection request sent by the source device, establish a wireless screen projection connection with the source device; When a wireless screen projection connection is established with the source device, a screen projection display interface is displayed, wherein the screen projection display interface at least includes the content requested by the source device for screen projection display.

3. The method according to claim 1, characterized in that The step of determining the human-computer interaction data of the display terminal device when a reverse control channel is established with the source terminal device includes: In a case where a reverse control channel is established with the source device, determining an input operation received by the display device; Corresponding human-computer interaction data is determined based on the input operation, wherein the input operation includes: a local input operation and a peripheral input operation.

4. The method according to claim 3, characterized in that The determining corresponding human-computer interaction data based on the input operation includes: In the case where the input operation is the peripheral input operation, determining original input data corresponding to the peripheral input operation and a peripheral driver corresponding to the peripheral input device adopted by the peripheral input operation; The peripheral device driver is used to encapsulate the original input data corresponding to the peripheral device input operation to obtain the human-computer interaction data.

5. The method according to claim 3, characterized in that The determining corresponding human-computer interaction data based on the input operation includes: In the case where the input operation is the local input operation, determining original input data corresponding to the local input operation; Performing input event conversion processing on the original input data corresponding to the local input operation to obtain an interaction event corresponding to the local input operation; Through the screen projection display interface process of the display terminal device, the interaction event corresponding to the local input operation is encapsulated to obtain the human-computer interaction data.

6. The method according to claim 1, characterized in that Before sending the target human-computer interaction data to the source device according to the data type and input device corresponding to the human-computer interaction data, the method further includes: In a case where the human-computer interaction data acts on the projection display interface of the display terminal device, determining that the data type corresponding to the human-computer interaction data is projection reverse control data; In the case where the human-computer interaction data acts on the local service interface of the display terminal device, it is determined that the data type corresponding to the human-computer interaction data is local control data.

7. The method according to claim 3, characterized in that Before sending the target human-computer interaction data to the source device according to the data type and input device corresponding to the human-computer interaction data, the method further includes: In the case where the input operation received by the display terminal device is an external input operation, determining that the input device corresponding to the human-computer interaction data is an external input device, the external input device includes at least one of the following: a stylus pen and a Bluetooth device; In a case where the input operation received by the display terminal device is a local input operation, it is determined that the input device corresponding to the human-computer interaction data is a local input device, and the local input device includes a touch screen.

8. The method according to any one of claims 1 to 7, characterized in that The sending the target human-computer interaction data to the source device according to the data type and input device corresponding to the human-computer interaction data includes: When the data type is screen projection reverse control data and the input device is an external input device, determining that the human-computer interaction data is the target human-computer interaction data; Sending the target human-computer interaction data to the screen projection display interface process of the display terminal device through the peripheral driver corresponding to the peripheral input device; The target human-computer interaction data is sent to the source device through the screen projection display interface process.

9. The method according to claim 8, characterized in that When the data type is screen projection reverse control data and the input device is an external input device, determining that the human-computer interaction data is the target human-computer interaction data includes: In the case where the peripheral input device is a Bluetooth device, detecting a key value to be responded to contained in the human-computer interaction data; In a case where the key value to be responded to is a key value to be responded to by the source device, the human-computer interaction data is determined to be the target human-computer interaction data.

10. The method according to claim 9, characterized in that In the case where the external input device is a Bluetooth device, after detecting the key value to be responded to contained in the human-computer interaction data, the method further includes: In a case where the key value to be responded to is a key value to be responded to by the display terminal device, decapsulating the human-computer interaction data to obtain an interaction event corresponding to the human-computer interaction data; Sending the interaction event to the local service interface process of the display terminal device; The interaction event is applied to the corresponding local service through the local service interface process.

11. The method according to any one of claims 1 to 7, characterized in that The sending the target human-computer interaction data to the source device according to the data type and input device corresponding to the human-computer interaction data includes: When the data type is screen projection reverse control data and the input device is a local input device, determining that the human-computer interaction data is the target human-computer interaction data; The target human-computer interaction data is sent to the source device through the screen projection display interface process of the display device.

12. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When the data type is local control data and the input device is an external input device, decapsulating the human-computer interaction data to obtain a corresponding interaction event; Sending the interaction event to the local service interface process of the display terminal device; The interaction event is applied to the corresponding local service through the local service interface process.

13. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when executed on an electronic device, enable the electronic device to perform the method as claimed in any one of claims 1 to 12.

Images