Face orientation based attention detection method, electronic device and chip system

Abstract

The face orientation-based attention detection method, the electronic device and the chip system provided by the embodiments of the present application relate to the technical field of terminals. The method comprises the following steps: a plurality of electronic devices sharing an input device can respectively calculate an angle with a user's face, comprehensively judge the face orientation of the user, and control the switching of the focus point of the input device on different electronic devices. In this way, the focus point of the input device can be quickly switched on different electronic devices by judging the face orientation, without the user having to tediously click a mouse or a screen to switch the focus point of the input device, thereby improving the user experience.

Description

[0001] This application claims priority to the Chinese Patent Application No. 202510592245.6, filed on May 08, 2025, entitled "Attention detection method based on face orientation, electronic device and chip system", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of terminal, and in particular to an attention detection method based on face orientation, an electronic device and a chip system. BACKGROUND

[0003] With the development of the network era, there may be a scenario in which a user uses multiple electronic devices simultaneously. In order to facilitate the use of the user, the multiple electronic devices can support a keyboard and mouse sharing function, that is, the multiple electronic devices can share the use of input devices such as a keyboard, a mouse or a touchpad.

[0004] However, in some scenarios, when the keyboard and mouse sharing function is implemented between the multiple electronic devices, the user needs to tediously click the mouse or the screen to switch the focus of the input device, which reduces the user experience. SUMMARY

[0005] The present application provides an attention detection method based on face orientation, an electronic device, a storage medium and a chip system. The multiple electronic devices sharing the input device can respectively calculate the included angle with the face of the user, comprehensively judge the face orientation of the user, and thereby control the switching of the focus of the input device on different electronic devices. In this way, the focus of the input device can be quickly switched on different electronic devices by judging the face orientation, without the user tediously switching the focus of the input device by clicking the mouse or the screen, thereby improving the user experience.

[0006] In a first aspect, an attention detection method based on face orientation is provided, applied to a first electronic device, and the method comprises:

[0007] The first electronic device and the first angle of the face of the user are determined, and the second angle from the second electronic device is obtained; the second angle is the angle between the second electronic device and the face of the user, and the second electronic device responds to the input event of the input device; in the case where the first angle is less than the second angle, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device; and in response to the user triggering the first operation of the input device, the input event corresponding to the first operation is executed. It can be understood that the multiple electronic devices (including the first electronic device and the second electronic device) sharing the input device can respectively calculate the angle with the face of the user (including the first angle and the second angle), comprehensively judge the direction of the face of the user, and control the switching of the focus of the input device on different electronic devices. In this way, the focus of the input device can be quickly switched on different electronic devices by judging the direction of the face, without the user having to tediously switch the focus of the input device by clicking the mouse or the screen, thereby improving the user experience.

[0008] In a possible implementation, after the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, the interface of the first electronic device displays first prompt information, and the first prompt information is used to indicate that the first electronic device responds to the input event of the input device. It can be understood that through the first prompt information, such as mouse animation prompt information, the user can know in time that the focus of the keyboard and the mouse has been switched from the second electronic device to the first electronic device. By using the mouse animation prompt information, the user can also quickly find the position of the mouse pointer, thereby improving the user experience.

[0009] In a possible implementation, the first prompt information includes one or more types of prompt information such as icon type prompt information, text type prompt information, and mouse prompt. It can be understood that by using the icon type prompt information, the information can be conveyed through visual symbols, so that the user can more easily identify and understand the meaning expressed by the first prompt information. By using the text type prompt information, the user can better understand on which electronic device the focus of the input device is currently on, so that the interface is more interactive and personalized. By using the mouse prompt, the user can quickly find the position of the mouse pointer or the mouse cursor, and can make further editing or other operations. That is, through the first prompt information, the user can know in time that the focus of the keyboard and the mouse has been switched from the second electronic device to the first electronic device, thereby improving the user experience of using the hand-eye coordination function.

[0010] In a possible implementation, the mouse prompt includes that the interface of the first electronic device displays an animation effect of a mouse pointer, and the animation effect of the mouse pointer includes one or more of the following animation effects: a size change of the mouse pointer, a color change of the mouse pointer, a light-emitting effect change of an edge of the mouse pointer, and a change around the mouse pointer. In this way, by taking the animation effect of the mouse pointer and the animation effect of the water ripples as the prompt information of the focus switching, the user can learn in a timely manner that the focus of the keyboard and the mouse has been switched to the first electronic device, so as to facilitate the user to quickly find the position of the mouse pointer, and improve the user experience.

[0011] In a possible implementation, before the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, the method further includes: displaying, by the interface of the first electronic device, second prompt information, the second prompt information being used to indicate that the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device by triggering a second operation; and switching, by the electronic device, the electronic device responding to the input event of the input device from the second electronic device to the first electronic device, including: in response to the second operation, switching, by the electronic device, the electronic device responding to the input event of the input device from the second electronic device to the first electronic device. In this way, by the second prompt information, the user can quickly learn how to switch the focus of the input device between different electronic devices, improve the accuracy of operation, and thus make the user more skilled in using the keyboard-mouse sharing function and the hand-eye coordination function.

[0012] In a possible implementation, the second operation includes a key operation on the input device. It can be understood that the first electronic device responds to the input event of the input device by triggering the second operation by the user. In this way, the user can be allowed to select whether to switch the focus of the input device according to actual needs, increase the selection and setting of some functions of the electronic device by the user, improve the interactivity with the user, and thus improve the user experience.

[0013] In a possible implementation, before the first angle between the first electronic device and the face of the user is determined and the second angle from the second electronic device is acquired, the method further includes: receiving an operation of setting the second electronic device as an electronic device sharing the input device with the first electronic device. It can be understood that by setting the second electronic device as the electronic device sharing the input device with the first electronic device, the keyboard-mouse sharing function and the hand-eye coordination function can be implemented between the first electronic device and the second electronic device. In this way, the user can move the mouse or share the input device between multiple electronic devices, and thus perform cross-screen operation more smoothly.

[0014] In a possible implementation, the input device is a device included in the first electronic device, or the input device is independent of the first electronic device. It can be understood that the input device is a device included in the first electronic device, and it can also be understood that the input device is a built-in device of the first electronic device. The built-in device can be tightly integrated with other parts or other modules in the first electronic device, without the need to prepare an additional input device or perform additional settings or connections, which is convenient and fast. The input device is independent of the first electronic device, and it can also be understood that the input device is an external device of the first electronic device. The user can select a type of input device that meets personal preferences, and the user can also adjust the position and angle of the external device as needed, to provide a more flexible input environment.

[0015] In a possible implementation, the input device includes a keyboard or a mouse.

[0016] In a possible implementation, the first electronic device includes a first camera, the second electronic device includes a second camera, the first included angle is an included angle between the first camera and the face orientation, and the second included angle is an included angle between the second camera and the face orientation. It can be understood that the face orientation of the user can be comprehensively determined by the included angles related to the plurality of electronic devices (including the first included angle and the second included angle). Compared with the determination of the face orientation of the user by a single electronic device, the determination based on the included angles of the plurality of electronic devices can reduce the fluctuation of data and the inaccuracy of the calculation result, reduce the determination error, and thus more accurately control the switching of the focus of the input device on different electronic devices.

[0017] In a possible implementation, the method further includes: displaying, by the first electronic device, a first interface; determining a third included angle between the first electronic device and the face of the user; and if the third included angle is greater than or equal to a preset angle threshold, adjusting a display content of the first interface or not displaying the first interface. It can be understood that adjusting the display content of the first interface or not displaying the first interface can make the content displayed by the first interface become blurred or be blocked, so that other people cannot clearly identify the content displayed by the first interface, thereby better protecting the information of the user.

[0018] In one possible implementation, the method further includes: a first electronic device displaying a second interface; determining a fourth angle between the first electronic device and the user's face; and if the fourth angle is greater than or equal to a preset angle threshold for a duration exceeding a preset time, entering a screen-off state or a screen-locked state. It is understood that, even if the electronic device does not receive any user-triggered operation within the preset time, it can still determine the user's facial orientation. If the user's face is not facing the electronic device within the preset time, it indicates that the user is not viewing the content displayed on the screen, and the electronic device can turn off or lock the screen. If the user's face is facing the electronic device within the preset time, it indicates that the user may be viewing the content displayed on the screen, and the electronic device will not turn off or lock the screen. In this way, the electronic device can more accurately determine the timing for turning off or locking the screen, thereby improving the user experience.

[0019] In one possible implementation, the method further includes: a first electronic device displaying a call interface; determining a fifth angle between the first electronic device and the user's face; and, if the fifth angle is greater than or equal to a preset angle threshold, the first electronic device activating a call mute function. The call mute state may include the electronic device not using audio data, and also the electronic device not sending the collected audio data to the other party's electronic device. This allows for reasonable protection of the user's information based on their head orientation, thereby improving the user experience.

[0020] In one possible implementation, the method further includes: establishing a communication connection between a first electronic device and a third electronic device, with the third electronic device displaying an interface of a first application; if the distance between the first and third electronic devices is less than a preset distance and the sixth angle is less than a preset angle threshold, the first electronic device displays a third prompt message, which includes the icon of the first application; wherein the sixth angle is the angle between the first electronic device and the user's face. It is understandable that the first electronic device can first determine the user's face orientation when displaying the third prompt message. When the first electronic device determines that the user's face is facing it, i.e., when the user's attention shifts from the third electronic device to the first electronic device, the first electronic device then begins displaying the third prompt message. This allows the user to promptly notice the application continuation prompt message, enabling the user to better utilize the application continuation function on the first electronic device, thereby improving the user experience.

[0021] In a possible implementation manner, before the first included angle between the first electronic device and the face of the user is determined, the method further includes: obtaining an image containing the face of the user, determining key point information of the face of the user based on the image; determining head posture information of the user based on the key point information, the head posture information including a yaw angle of the head of the user; and determining the first included angle between the first electronic device and the face of the user based on the head posture information of the user, a distance between the camera and the face of the user in a horizontal direction, and a distance between the camera and the face of the user in a vertical direction. It can be understood that the sparse feature is a relatively small number of face key points, and the face key points are sufficient to describe the basic structure and features of the face. By calculating the sparse feature, the face features can be effectively described while reducing the calculation complexity of the electronic device, and the processing speed and the recognition efficiency of the electronic device can be improved while maintaining the recognition accuracy.

[0022] In a second aspect, an attention detection method based on a face orientation is provided, and the method is applied to a second electronic device, and includes the following steps.

[0023] In response to a third operation of the input device triggered by the user, an input event corresponding to the third operation is executed; a second included angle between the second electronic device and the face of the user is determined, and the second included angle is sent to the first electronic device; and in a case where the first included angle is less than the second included angle, a fourth operation of the input device triggered by the user is not responded, where the first included angle is an included angle between the first electronic device and the face of the user. It can be understood that a plurality of electronic devices (including the first electronic device and the second electronic device) sharing the input device can respectively calculate the included angle with the face of the user (including the first included angle and the second included angle), comprehensively determine the face orientation of the user, and control the switching of the focus of the input device on different electronic devices. In this way, the focus of the input device can be quickly switched on different electronic devices by determining the face orientation, without the user tediously switching the focus of the input device by clicking the mouse or the screen, and the user experience is improved.

[0024] In a third aspect, an attention detection method based on a face orientation is provided, and the method is applied to a communication system, and the communication system includes a first electronic device and a second electronic device, and the method includes the following steps.

[0025] The second electronic device responds to the input event of the input device; the first electronic device determines a first included angle between the first electronic device and the face of the user, and the second electronic device determines a second included angle between the second electronic device and the face of the user; in a case where the first electronic device determines that the first included angle is smaller than the second included angle, the first electronic device responds to the input event of the input device; and the first electronic device performs an input event corresponding to a first operation of the input device in response to the user triggering the first operation of the input device. It can be understood that the first electronic device can calculate the first included angle, and the second electronic device can calculate the second included angle, and the face orientation of the user can be comprehensively determined through the first included angle and the second included angle, so as to control the switching of the focus of the input device on different electronic devices. In this way, the focus of the input device can be quickly switched on different electronic devices by determining the face orientation, and the user experience is improved.

[0026] In a fourth aspect, an apparatus for attention detection based on face orientation is provided. The apparatus can be an electronic device, or a chip or chip system in the electronic device. The apparatus can include a processing unit. The processing unit can be configured to implement any method described in the first aspect or any possible implementation of the first aspect, or any method described in the implementation of the second aspect, or any method described in the implementation of the third aspect.

[0027] When the apparatus is an electronic device, the processing unit can be a processor. The apparatus can further include a storage unit, which can be a memory. The storage unit can be configured to store instructions, and the processing unit can be configured to execute the instructions stored in the storage unit, so that the electronic device implements any method described in the first aspect or any possible implementation of the first aspect, or any method described in the implementation of the second aspect, or any method described in the implementation of the third aspect.

[0028] When the apparatus is a chip or chip system in an electronic device, the processing unit can be a processor. The processing unit can be configured to execute instructions stored in a storage unit, so that the electronic device implements any method described in the first aspect or any possible implementation of the first aspect, or any method described in the implementation of the second aspect, or any method described in the implementation of the third aspect. The storage unit can be a storage unit (e.g., a register, a cache, etc.) in the chip, or a storage unit (e.g., a read-only memory, a random access memory, etc.) in the electronic device and located outside the chip.

[0029] In a fifth aspect, an electronic device is provided, which includes one or more processors and a memory coupled to the one or more processors. The memory is configured to store computer program codes comprising computer instructions. The one or more processors are configured to invoke the computer instructions to perform the method described in the first aspect or any possible implementation manner of the first aspect, or perform the method described in the implementation manner of the second aspect, or perform the method described in the implementation manner of the third aspect.

[0030] In a sixth aspect, a computer readable storage medium is provided, which stores computer programs or instructions. When the computer programs or instructions are run on a computer, the computer is caused to perform the method described in the first aspect or any possible implementation manner of the first aspect, or perform the method described in the implementation manner of the second aspect, or perform the method described in the implementation manner of the third aspect.

[0031] In a seventh aspect, a computer program product including a computer program is provided. When the computer program is run on a computer, the computer is caused to perform the method described in the first aspect or any possible implementation manner of the first aspect, or perform the method described in the implementation manner of the second aspect, or perform the method described in the implementation manner of the third aspect.

[0032] In an eighth aspect, a chip or chip system is provided, which includes at least one processor and a communication interface. The communication interface and the at least one processor are interconnected by a line. The at least one processor is configured to run computer programs or instructions to perform the method described in the first aspect or any possible implementation manner of the first aspect, or perform the method described in the implementation manner of the second aspect, or perform the method described in the implementation manner of the third aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0033] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, which stores instructions. The memory can be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0034] It should be understood that the fourth aspect to the eighth aspect of the present application correspond to the technical solutions of the first aspect, the second aspect and the third aspect of the present application. The beneficial effects achieved by each aspect and the corresponding possible implementation manners are similar, and will not be described again. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1A hardware structure schematic diagram of an electronic device provided by an embodiment of the present application;

[0036] Figure 2 A software architecture schematic diagram of an electronic device provided by an embodiment of the present application;

[0037] Figure 3 A use scenario schematic diagram of an attention detection method provided by an embodiment of the present application;

[0038] Figure 4 A schematic diagram of displaying prompt information in a mobile phone interface provided by an embodiment of the present application;

[0039] Figure 5 A schematic diagram of displaying an animation effect of a mouse pointer in a mobile phone interface provided by an embodiment of the present application;

[0040] Figure 6 A schematic diagram of displaying a water ripple animation effect around a mouse pointer provided by an embodiment of the present application;

[0041] Figure 7 An interface schematic diagram of setting an application window in a PC provided by an embodiment of the present application;

[0042] Figure 8 An interface schematic diagram of displaying a device list in a keyboard and mouse sharing interface provided by an embodiment of the present application;

[0043] Figure 9 A schematic diagram of a keyboard and mouse sharing interface in a PC provided by an embodiment of the present application;

[0044] Figure 10 An interface schematic diagram of setting an application in a mobile phone provided by an embodiment of the present application;

[0045] Figure 11 A module interaction schematic diagram of an attention detection method based on face orientation provided by an embodiment of the present application;

[0046] Figure 12 A schematic diagram of the relationship between face orientation and the included angle of an electronic device provided by an embodiment of the present application;

[0047] Figure 13 A schematic diagram of judging face orientation based on multiple electronic devices provided by an embodiment of the present application;

[0048] Figure 14 A schematic diagram of face orientation judgment in an application connection scenario provided by an embodiment of the present application;

[0049] Figure 15 A schematic diagram of protecting screen content based on face orientation provided by an embodiment of the present application;

[0050] Figure 16 A schematic diagram of face orientation judgment in a call scenario provided by an embodiment of the present application;

[0051] Figure 17 A schematic diagram of an attention detection method based on face orientation provided by an embodiment of the present application;

[0052] Figure 18 A structural schematic diagram of a chip provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to clearly describe the technical solutions of the embodiments of the present application, the following briefly introduces some terms and technologies involved in the embodiments of the present application:

[0054] 1. Terms

[0055] In the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc. For example, the first chip and the second chip are only used to distinguish different chips, and do not limit the sequence. Those skilled in the art can understand that "first", "second", etc. do not limit the number and execution order, and "first", "second", etc. do not necessarily mean different.

[0056] It should be noted that in the embodiments of the present application, "exemplary" or "for example" is used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0057] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship between the associated objects is described by "and / or", which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c, can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0058] With the development of the network era, a user can use multiple electronic devices at the same time. In order to improve the input experience of the user, the multiple electronic devices can support a keyboard and mouse sharing function, that is, the multiple electronic devices can share the use of input devices such as a keyboard, a mouse, or a touchpad.

[0059] Taking the sharing of a keyboard between a personal computer (PC) and a mobile phone as an example, in some scenarios, the user can edit a document, reply to an email, and the like on the PC through the keyboard, and the user can also reply to social information on the mobile phone through the keyboard. When the user uses the PC and the mobile phone alternately, the focus of the keyboard input needs to be switched between the PC and the mobile phone. For example, when the user uses the PC, the focus of the keyboard input needs to be switched to the PC by clicking a mouse. When the user uses the mobile phone, the focus of the keyboard input needs to be switched to the mobile phone by clicking the screen of the mobile phone. That is, when the keyboard and mouse sharing function is implemented between the multiple electronic devices, the user needs to click the mouse or the screen to switch the focus of the input device complicatedly, which reduces the user experience.

[0060] In view of this, the face orientation based attention detection method provided in the embodiments of the present application can be used to calculate the included angle between the face of the user and the multiple electronic devices (including the first electronic device and the second electronic device in the following) sharing the input device (including the first included angle and the second included angle in the following), comprehensively determine the face orientation of the user, and control the switching of the focus of the input device on different electronic devices. In this way, the focus of the input device can be quickly switched on different electronic devices by determining the face orientation, without the user clicking the mouse or the screen to switch the focus of the input device complicatedly, which improves the user experience.

[0061] It can be understood that the function of controlling the switching of the focus on different electronic devices by determining the face orientation provided in the embodiments of the present application can also be called a hand-eye same function or a hand-eye coordination function. In order to facilitate description, the function is called a hand-eye same function in the following.

[0062] The electronic device of the embodiments of the present application can also be a terminal device of any form having a keyboard and mouse sharing function and a hand-eye coordination function. For example, the electronic device can include a mobile phone, a tablet computer, a palm computer, a notebook computer, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, an electronic device in a 5G network, or an electronic device in a future evolved public land mobile network (PLMN), etc., and the embodiments of the present application are not limited thereto.

[0063] In addition, in the embodiments of the present application, the electronic device can also be an electronic device in an internet of things (IoT) system. The IoT is an important component of future information technology development, and its main technical feature is to connect objects through communication technology and network, so as to realize the intelligent network of man-machine interconnection and object-object interconnection.

[0064] The electronic device in the embodiments of the present application can also be referred to as a user equipment (UE), a mobile station (MS), a mobile terminal (MT), an access terminal, a subscriber unit, a subscriber station, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a terminal, a wireless communication device, a user agent, or a user apparatus, etc.

[0065] In the embodiments of the present application, the electronic device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also referred to as a main memory). The operating system can be any one or more computer operating systems that implement business processing through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. The application layer includes a browser, an address book, word processing software, instant messaging software, and the like.

[0066] Exemplary, Figure 1 A hardware structure schematic diagram of the electronic device is shown.

[0067] The electronic device can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, and the like. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0068] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can include hardware, software, or a combination of software and hardware.

[0069] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated in one or more processors. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0070] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instructions or data again, it can directly call from the above-mentioned memory. Avoiding repeated access, reducing the waiting time of the processor 110, thus improving the efficiency of the system. For example, in the embodiments of the present application, the processor 110 can be used to support the face key point detection module to perform face key point detection, and support the comprehensive sensing information platform to perform head pose estimation and face orientation judgment, etc.

[0071] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function, etc. The data storage area can store data created during the use of the electronic device, etc. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various function applications and data processing of the electronic device by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor. For example, in the embodiments of the present application, the internal memory 121 can be used to store related codes of face key point detection, head pose estimation, and user face orientation judgment, etc.

[0072] In some embodiments, the electronic device can include 1 or N cameras 193, N being a positive integer greater than 1. The camera 193 can be used to capture still images or videos. For example, in the embodiments of the present application, the camera can be used to collect images containing the user's face, etc.

[0073] Figure 2 is a schematic diagram of a software architecture of an electronic device according to an embodiment of the present application.

[0074] For example, the following is an example of a software structure of an electronic device using the Harmony system It should be noted that the embodiments of the present application can also be applied to other operating systems, such as the Android system, the iOS system, and the like, as long as the functions of the various functional modules are similar to those of the embodiments of the present application and the solutions of the present application can be implemented.

[0075] The layered architecture of the software can divide the software into several layers, each layer having a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Harmony system can include an application layer, a framework layer, a system basic service layer, and a kernel layer, etc.

[0076] The application layer can include system applications and third-party applications, such as keyboard and mouse sharing applications, settings applications, camera applications, gallery applications, video applications, etc. In some scenarios, the application layer can also be referred to as the application program layer. In the embodiments of the present application, the keyboard and mouse sharing application can instruct the electronic device to display a prompt message for switching the screen focus. In order to realize the switching of the screen focus between multiple electronic devices, the user can open the keyboard and mouse sharing function and the hand-eye coordination function in the settings application, and the specific opening method can refer to the related description in the corresponding embodiments, which will not be described here. Figures 7 to 10

[0077] The framework layer provides a user program framework for the application programs of the Harmony system in multiple languages such as Java / C / C++ / JavaScript, and a multi-language framework application programming interface opened by various software and hardware services. In some scenarios, the framework layer can also be referred to as the application program framework layer. In the embodiments of the present application, the framework layer can include a multimodal sensor data platform (MSDP). The MSDP can be used to estimate the head posture of the user and calculate the included angle between the face of the user and the electronic device, so as to determine the orientation of the face of the user.

[0078] ​The system foundation service layer is a core capability set of the HarmonyOS system, and supports the HarmonyOS system to provide services to application services through the framework layer in a multi-device deployment scenario. The system foundation service layer can include, but is not limited to, one or more of the following parts: a system basic capability subsystem set, a basic software service subsystem set, an enhanced software service subsystem set, a hardware service subsystem set, a proprietary hardware service subsystem set, a Harmony driver framework, and a hardware abstraction layer.

[0079] The system basic capability subsystem set provides basic capabilities for distributed applications to run, schedule, and the like on the HarmonyOS multi-device. The system basic capability subsystem set can include, but is not limited to, one or more of the following parts: a distributed task scheduling, a distributed storage and data management, a distributed software bus, an Ark runtime, and a multi-modal input subsystem.

[0080] For example, in the embodiment of the present application, the distributed software bus provides communication-related capabilities for data transmission and information interconnection between multiple electronic devices, including wireless local area network (WLAN) service capabilities, Bluetooth service capabilities, a software bus, remote procedure call (RPC) communication capabilities, and the like.

[0081] The multi-modal input subsystem can integrate inputs of multiple dimensions together. For example, the multi-modal input subsystem can receive input events of input devices such as keyboards, mice, touch screens, touchpads, and the like based on the kernel subsystem and the driver framework, and deliver the input events to an application (such as a keyboard and mouse sharing application). The input event can be understood as a signal or data generated by an input device, which can be used to represent a certain operation of a user. The input event is the basis for the software system to interact with the user. The multi-modal input subsystem can achieve interaction with the user by receiving and processing input events. For example, when a user presses a key on a keyboard, the multi-modal input subsystem can receive a keyboard event and perform a corresponding function according to the keyboard event.

[0082] The basic software service subsystem set provides common and universal software services for the HarmonyOS system, which can be composed of a graphics image software service, a distributed media software service, and the like. The enhanced software service subsystem set provides differentiated capability-enhanced software services for different devices for the HarmonyOS system, which can be composed of a tablet business software service, a smart screen business software service, and the like. The hardware service subsystem set provides common and adaptive hardware services for the HarmonyOS system, which can include a power service and the like. The proprietary hardware service subsystem set provides differentiated hardware services for different devices for the HarmonyOS system, which can include a tablet proprietary hardware service, a wearable proprietary hardware service, and the like.

[0083] It can be understood that each of the above service subsystem sets can be deployed according to different device form factors, and each subsystem can be trimmed according to functional granularity.

[0084] The Hongmeng driving framework and the hardware abstraction layer are the basis for the hardware ecological opening of the Hongmeng system, which provides hardware capability abstraction to the hardware and provides a development framework and a running environment for various peripheral drivers. For example, in the embodiment of the present application, the hardware abstraction layer can include a face key point detection module. The face key point detection module can be used to obtain key point information of a face based on an image collected by a camera.

[0085] The Hongmeng system adopts a multi-core design, which can include a Linux kernel, a Hongmeng microkernel, and LiteOS, for example. Through this design, electronic devices with different device capabilities can select appropriate system kernels.

[0086] The method of the embodiments of the present application will be described in detail below through specific embodiments. The following embodiments can be combined or implemented independently, and for the same or similar concepts or processes, they can not be described again in some embodiments.

[0087] For example, the following takes a PC and a mobile phone as an example to illustrate a use scenario of the attention detection method of the embodiments of the present application. It can be understood that the method of the embodiments of the present application can be used to detect attention between two or more electronic devices, and the number of electronic devices and the type of electronic devices involved are not limited by the embodiments of the present application.

[0088] In a possible scenario, a user can view or edit a document on a PC and chat on a mobile phone. As shown in a of FIG. 3, the interface 300 of the PC can include a document interface 301. The content displayed by the document interface 301 is not limited by the embodiments of the present application. Figure 3 The interface 310 of the mobile phone can display a chat window. The chat window can include a title bar 311, a chat content display area 312, and a chat content input area 313. The title bar 311 can include a return control and a friend name. The return control can be used to exit the current chat window. The chat content display area 312 can be used to display chat records of the user and the friend. The content of the chat records is not limited by the embodiments of the present application. The chat content input area 313 can be used to input voice, text, and emoticons, or send pictures, videos, etc. The chat content input area 313 can include an input bar 314, which can be used to input voice, text, and emoticons.

[0089]

[0090] ​It should be noted that the interfaces provided by the embodiments of the present application are exemplary interfaces. More or fewer contents and / or controls can be included in each interface, and the contents and / or controls included in each interface and the display styles of the contents and / or controls are not limited by the embodiments of the present application.

[0091] When the user opens the keyboard-mouse sharing function and the hand-eye following function in the PC and the mobile phone respectively, the keyboard 320 and / or the mouse 330 can be shared between the PC and the mobile phone. The specific manner of opening the keyboard-mouse sharing function and the hand-eye following function in the PC and the mobile phone can be referred to the following Figures 7 to 10 The related description of the corresponding embodiments is not repeated here. The PC and the mobile phone can control the focus of the keyboard 320 and / or the mouse 330 on which electronic device by judging the face orientation of the user. The PC and the keyboard 320 can establish a wireless connection therebetween, and the PC and the mouse 330 can establish a wireless connection therebetween. The mobile phone and the keyboard 320 can establish a wireless connection therebetween, and the mobile phone and the mouse 330 can establish a wireless connection therebetween. The wireless connection includes, for example, a Bluetooth connection, a Wi-Fi connection, and the like.

[0092] When the user edits or views the document content in the document interface 301 of the PC, the user can input the content to be edited in the document interface 301 through the keyboard 320, and the user can view the content in the document interface 301 by sliding or dragging the mouse 330. At this time, the focus of the keyboard 320 and the focus of the mouse 330 are both on the interface 300 of the PC. The cursor can be displayed in the document interface 301, which is used to indicate the position of the mouse pointer or the position of the input content, and the user can input the content to be edited at the position of the cursor through the keyboard 320.

[0093] When the head of the user turns from the PC to the mobile phone, the focus of the keyboard 320 and the focus of the mouse 330 are switched to the interface 310 of the mobile phone without the user dragging or clicking the mouse 330. The user can input the chat content in the input box 314 through the keyboard 320, and the user can view the chat record in the chat content display area 312 by sliding or dragging the mouse 330. At this time, the cursor can be displayed in the input box 314, which is used to indicate the position of the mouse pointer or the position of the input content, and the user can input the chat content at the position of the cursor through the keyboard 320.

[0094] When the head of the user turns from the mobile phone to the PC again, the focus of the keyboard 320 and the focus of the mouse 330 are switched to the interface 300 of the PC without the user dragging or clicking the mouse 330, and the cursor can be displayed in the document interface 301. The user can input the content to be edited in the document interface 301 through the keyboard 320 again, and the user can view the content in the document interface 301 by sliding or dragging the mouse 330.

[0095] That is, without the need to frequently drag or click the mouse 330, the embodiment of the present application can control the switching of the focus of the keyboard 320 and the focus of the mouse 330 between the interface 300 of the PC and the interface 310 of the mobile phone by judging the face orientation of the user. In this way, the operation of the user to switch the focus between different electronic devices can be saved, and the user experience is improved.

[0096] As shown in FIG. 3b, the camera 302 is installed on the PC, and the camera 315 is installed on the mobile phone. When the head of the user turns from the PC to the mobile phone, the camera 302 can capture an image containing the face of the user, and then the PC can calculate the included angle (hereinafter referred to as the first included angle) between the straight line between the camera 302 and the face of the user and the face orientation of the user based on the image captured by the camera 302. The camera 315 can also capture an image containing the head of the user, and then the mobile phone can calculate the included angle (hereinafter referred to as the second included angle) between the straight line between the camera 315 and the face of the user and the face orientation of the user based on the image captured by the camera 315. Figure 3

[0097] After the first included angle and the second included angle are obtained respectively, the PC or the mobile phone can judge the face orientation of the user by comparing the first included angle and the second included angle. Taking the PC as an example to judge the first included angle and the second included angle, the mobile phone can send the calculated second included angle to the PC. Further, the PC can judge the face orientation of the user by comparing the sizes of the first included angle and the second included angle.

[0098] As can be seen from FIG. 3b, the first included angle is greater than the second included angle, which indicates that the included angle between the face of the user and the PC is relatively large, and the included angle between the face of the user and the mobile phone is relatively small. It can also be understood that the face of the user is oriented to the mobile phone, and the attention of the user is turned to the mobile phone. Therefore, the focus of the keyboard 320 and the focus of the mouse 330 can be switched from the interface 300 of the PC to the interface 310 of the mobile phone. Figure 3 Optionally, when the face of the user is oriented to the mobile phone, the interface 310 of the mobile phone can display a prompt information. The prompt information is used to prompt the user that the current gaze is the mobile phone, and can also be used to instruct the user to switch the focus. Optionally, the prompt information can be presented in the form of an icon and / or a prompt sentence. The prompt information can be displayed at any possible position of the interface 310, and the embodiment of the present application does not limit the display position of the prompt information.

[0099] Exemplarily, as shown in FIG. 3c, the interface 310 of the mobile phone can display a prompt information 311. The prompt information 311 is used to prompt the user that the current gaze is the mobile phone, and can also be used to instruct the user to switch the focus. The prompt information 311 can be presented in the form of an icon and / or a prompt sentence. The prompt information 311 can be displayed at any possible position of the interface 310, and the embodiment of the present application does not limit the display position of the prompt information 311.

[0100] Figure 4 ​​As shown in Fig. 4a, the interface 400 of the mobile phone displays a prompt information 401. The prompt information is used to prompt that the user is currently gazing at the mobile phone, and can also be used to instruct the user to switch the focus. The prompt information 401 can include an icon and a prompt sentence. The icon can be a smiling face icon, or other icons, which are not limited in the embodiments of the present application. The prompt sentence can include "press the Ctrl key to switch the cursor to the current screen", and the prompt sentence can include more or less information, which are not limited in the embodiments of the present application. When the user presses the Ctrl key, the interface 400 of the mobile phone displays a prompt information of focus switching in response to the operation of the user triggering the Ctrl key. The prompt information of focus switching can be presented in the form of an icon, a prompt sentence, and / or a display of a mouse pointer.

[0101] Optionally, the user can also switch the cursor by triggering other keys (such as the Enter key, the shift key, or the space bar, etc.) or other operations (such as the operation of clicking the mouse, etc.). It can be understood that the operations mentioned in the embodiments of the present application can include but are not limited to touch (such as clicking, etc.), voice control, gestures, etc., which are not limited in the embodiments of the present application.

[0102] For example, taking the prompt information of focus switching as a display of a mouse pointer, in response to the operation of the user triggering the Ctrl key, the mobile phone can display the mouse pointer in the form of an animation effect as shown in Fig. 4b. Figure 5

[0103] As shown in Fig. 5a, the interface 500 of the mobile phone displays a mouse pointer 501. Optionally, the mouse pointer 501 can be displayed at any possible position of the interface 500, which is not limited in the embodiments of the present application, for example, the mouse pointer 501 can be displayed at the midpoint of the interface 500 of the mobile phone, the input box 314 or the position near the input box 314, or the last position of the mouse pointer 501 when the last focus of the mobile phone was located at the interface 500. Figure 5 Figure 5 For example, the magnification process of the mouse pointer 501 can include the change from a to b, and then the change from b to c. It can be understood that the interface shown in Fig. 4b can be an interface in the magnification process of the mouse pointer 501. Figure 5 Figure 5 For example, the magnification process of the mouse pointer 501 can include the change from a to b, and then the change from b to c. It can be understood that the interface shown in Fig. 4b can be an interface in the magnification process of the mouse pointer 501. Figure 5 Figure 5 For example, the magnification process of the mouse pointer 501 can include the change from a to b, and then the change from b to c. It can be understood that the interface shown in Fig. 4b can be an interface in the magnification process of the mouse pointer 501.

[0104] For example, the magnification process of the mouse pointer 501 can include the change from a to b, and then the change from b to c. It can be understood that the interface shown in Fig. 4b can be an interface in the magnification process of the mouse pointer 501. Figure 5 Figure 5 For example, the magnification process of the mouse pointer 501 can include the change from a to b, and then the change from b to c. It can be understood that the interface shown in Fig. 4b can be an interface in the magnification process of the mouse pointer 501. Figure 5 Figure 5 ​​​​​​the process of a. Alternatively, the reduced mouse pointer 501 can have the same size as the mouse pointer 501 in a, and of course can have different sizes, which are not limited in the embodiments of the present application. Figure 5

[0105] Alternatively, the style of the mouse pointer 501 can also change during the display of the animation effect of the mouse pointer 501. The style change of the mouse pointer 501 can include color change of the mouse pointer 501 and / or light effect change around the mouse pointer 501, etc. The color change can include color tone change, color brightness change and / or color transparency change, etc., which are not limited in the embodiments of the present application.

[0106] Taking the style change of the mouse pointer 501 including color brightness change and light effect change as an example. In a possible implementation, as the mouse pointer 501 gradually enlarges, the color brightness of the mouse pointer 501 can become brighter and brighter, and the light effect around the mouse pointer 501 can become more and more obvious. As the mouse pointer 501 gradually reduces, the color brightness of the mouse pointer 501 can become dimmer and dimmer, and the light effect around the mouse pointer 501 can become less and less obvious, or even no light effect is displayed.

[0107] Alternatively, other animation effects can also be displayed around the mouse pointer 501 during the display of the animation effect of the mouse pointer 501. The display style of the other animation effects is not limited in the embodiments of the present application.

[0108] Taking the water ripple animation effect as an example, as shown in a, a schematic diagram of displaying the water ripple animation effect around the mouse pointer is shown. As shown in b and c, the water ripple 602 can be displayed around the mouse pointer 601 during the gradual enlargement of the mouse pointer 601. Figure 6 Figure 6 As shown in a, the mouse pointer 601 can be displayed in the interface 600 of the mobile phone. Alternatively, the mouse pointer 601 can be displayed at any possible position of the interface 600, which is not limited in the embodiments of the present application, for example, can be displayed at the midpoint of the interface 600 of the mobile phone, the input bar 314 or the position nearby, or can be displayed at the last position of the mouse pointer 601 when the last focus is located at the interface 600 of the mobile phone. During the display of the animation effect of the mouse pointer 601, the water ripple animation effect can be displayed around the mouse pointer. As shown in b and c, the water ripple 602 can be displayed around the mouse pointer 601 during the gradual enlargement of the mouse pointer 601. Figure 6 Figure 6 As shown in b and c, the water ripple 602 can be displayed around the mouse pointer 601 during the gradual enlargement of the mouse pointer 601. And the water ripple 602 can gradually increase and become more and more obvious with the increase of the mouse pointer 601.

[0109] After the mouse pointer 601 is enlarged to a certain extent, the mouse pointer 601 will gradually reduce. During the gradual reduction of the mouse pointer 601, as shown in d, the water ripple 602 around the mouse pointer 601 can gradually reduce and become less and less obvious with the reduction of the mouse pointer 601. Figure 6 ​​​As shown in d of FIG. 6, the water ripples 602 are still displayed around the mouse pointer 601. Moreover, the water ripples 602 continue to increase and become less and less obvious. Such a display manner can better simulate the change process of the water ripples, and make the animation effect of the water ripples more realistic. It can be understood that, Figure 6 b of FIG. 6, Figure 6 c of FIG. 6, and Figure 6 d of FIG. 6 can be an interface in the display process of the animation effect of the water ripples.

[0110] When the animation effect display of the mouse pointer 601 is completed, the animation effect of the water ripples 602 can also be completed. As shown in e of FIG. 6, the interface 600 displays the mouse pointer 601, but no longer displays the water ripples 602. Figure 6

[0111] In this way, by taking the change of the mouse pointer and the animation effect of the water ripples as the prompt information of the focus switching, the user can know in time that the focus of the keyboard and the mouse has been switched to the mobile phone, so as to facilitate the user to quickly find the position of the mouse pointer, and improve the user experience.

[0112] The following will combine Figures 7 to 10 to explain how to open the keyboard and mouse sharing function and the hand-eye coordination function in the PC and the mobile phone. Among them, Figures 7 to 9 shows a way of opening the keyboard and mouse sharing function and the hand-eye coordination function in the PC, Figure 10 shows a way of opening the keyboard and mouse sharing function and the hand-eye coordination function in the mobile phone. It can be understood that the way of opening the keyboard and mouse sharing function and the hand-eye coordination function on other electronic devices is similar to the PC or the mobile phone, and will not be described here.

[0113] Exemplarily, as shown in Figure 7 , it is an interface schematic diagram of a window 701 of a setting application in a PC interface. The window 701 can include a first display area 702 and a second display area 703. The first display area 702 is used to display icons of setting applications, a setting title, a search box, account information, and a plurality of setting items, etc. The second display area 703 is used to display a detailed interface of a certain setting item.

[0114] Among them, in the first display area 702, the search box can be used to quickly search the content that the user wants to find. The search box can include a search icon and a search prompt. The search prompt is used to prompt the user that the user can quickly find the content that the user wants to find by inputting a keyword. In response to the operation of triggering the search icon by the user, the PC can search the related content based on the keyword in the search box.

[0115] ​The account information can include an account avatar of the user, an account name of the user, and an account identifier (ID) of the user, and the like.

[0116] The plurality of setting items can include, for example, WLAN, Bluetooth, network, multi-device collaboration, desktop and personalization, display and brightness, sound, notification and status bar, system, and the like. Each setting item can include a setting item icon and a setting item name. For example, taking the multi-device collaboration setting item 704 as an example, the multi-device collaboration setting item 704 can include a multi-device collaboration icon 7041 and a multi-device collaboration name 7042.

[0117] In response to the user triggering an operation of the multi-device collaboration setting item 704, the PC can display a keyboard and mouse sharing interface 705 in the second display area 703. The triggering operation can include an operation of triggering any position in the multi-device collaboration setting item 704.

[0118] Optionally, in response to the user triggering an operation of the multi-device collaboration setting item 704, the multi-device collaboration setting item 704 can be displayed differently from other setting items. For example, the background (such as color) of the multi-device collaboration setting item 704 can be different from other setting items, or the multi-device collaboration icon 7041 and / or the multi-device collaboration name 7042 can be highlighted.

[0119] It can be understood that in the embodiments of the present application, the highlighting can include, but is not limited to, one or more of the following display modes: changing the color of the font and / or icon, changing the size of the font and / or icon, changing the gray scale of the font and / or icon, and bolding the font.

[0120] In the second display area 703, the keyboard and mouse sharing interface 705 can include a return control, a keyboard and mouse sharing title, a full-screen control, a minimize control, a close control, a keyboard and mouse sharing icon, a keyboard and mouse sharing prompt, a keyboard and mouse sharing option 706, prompt information corresponding to the option 706, a device option 707 of keyboard and mouse sharing, an option 708 of allowing the keyboard and mouse to move to the screen of a nearby device to connect the device, prompt information corresponding to the option 708, a hand-eye coordination option 709, and prompt information corresponding to the option 709.

[0121] The return control is used to exit the current keyboard and mouse sharing interface or return to the previous interface. The maximize control is used to display the window 701 of the setting application in full screen. The minimize control is used to hide or minimize the display window 701, for example, including reducing the window 701 to the taskbar of the PC, so that the window 701 does not occupy the screen space of the PC. The close control is used to close the window 701.

[0122] The keyboard and mouse sharing prompt can include, for example, "Move the mouse on the computer to the edge of the screen, and then continuously move to the screen of another device nearby to experience the keyboard and mouse sharing and file dragging across devices. Learn more." The keyboard and mouse sharing prompt can include more or less information, which is not limited in the embodiments of the present application. The "Learn more" in the keyboard and mouse sharing prompt can be displayed in the form of a hyperlink. In response to the user triggering the operation of "Learn more", the PC can open and display the details content in the hyperlink corresponding to "Learn more". Optionally, "Learn more" can be highlighted, so that the user can quickly find the location of the hyperlink, and the user can view the details content in the hyperlink, thereby improving the user experience.

[0123] The keyboard and mouse sharing option 706 can include the option name (for example, "keyboard and mouse sharing") of the option 706, and a switch control 7061 of the keyboard and mouse sharing function. The switch control 7061 can be used to open (which can also be understood as starting or enabling) or close (which can also be understood as disabling) the keyboard and mouse sharing function. For example, in the case where the switch control 7061 is in the closed state, the keyboard and mouse sharing function is in the disabled state. In response to the user triggering the operation of the switch control 7061, the switch control 7061 can be displayed in the open state (as shown), and the PC can enable the keyboard and mouse sharing function. In the case where the switch control 7061 is in the open state, the keyboard and mouse sharing function is in the enabled state. In response to the user triggering the operation of the switch control 7061, the switch control 7061 can be displayed in the closed state, and the PC can disable the keyboard and mouse sharing function. Figure 7

[0124] The prompt information corresponding to the option 706 can include, for example, "This service needs to use Bluetooth or WLAN function to share mouse and keyboard related operations between devices. Turn on the switch to indicate that you agree to the above content." The prompt information corresponding to the option 706 can include more or less information, which is not limited in the embodiments of the present application.

[0125] The device option 707 of the keyboard and mouse sharing can include the option name (for example, "keyboard and mouse sharing device") of the option 707, prompt information 7071, and a drop-down control 7072.

[0126] ​The prompt information 7071 is used to prompt the connection status of the device currently sharing the keyboard and mouse with the PC. For example, if the PC is not connected to any device sharing the keyboard and mouse, the prompt information 7071 can be displayed as "No connected device". If the PC is connected to a device sharing the keyboard and mouse, the prompt information 7071 can be used to display the relevant information of the device sharing the keyboard and mouse. It can be understood that the relevant information of the device can be used to uniquely identify the device, so as to facilitate the user to distinguish the connected device. For example, the relevant information of the device can include one or more information such as, but not limited to, the device model, the device icon, the device name, the device ID, the account name logged in the device, and the like. The content displayed by the prompt information 7071 is not limited in the embodiments of the present application, as long as the content displayed by the prompt information 7071 corresponds to the information of the device sharing the keyboard and mouse connected to the PC.

[0127] The pull-down control 7072 can be used to display the device list of the devices that can share the keyboard and mouse with the PC.

[0128] The option 708 can include the option name of the option 708 (for example, "Allow the keyboard and mouse to move into the screen of the nearby device to connect the device") and the switch control 7081 of the option 708. The switch control 7081 can be used to open or close the function of allowing the keyboard and mouse to move into the screen of the nearby device to connect the device. For example, when the switch control 7081 is in the closed state, the PC does not allow the keyboard and mouse to move into the screen of the nearby device to connect the device. In response to the operation of the switch control 7081 triggered by the user, the switch control 7081 can be displayed in the open state (as shown in FIG. 7B), and the PC allows the keyboard and mouse to move into the screen of the nearby device to connect the device. Figure 7 When the switch control 7081 is in the open state, the PC allows the keyboard and mouse to move into the screen of the nearby device to connect the device. In response to the operation of the switch control 7081 triggered by the user, the switch control 7081 can be displayed in the closed state, and the PC does not allow the keyboard and mouse to move into the screen of the nearby device to connect the device.

[0129] The prompt information corresponding to the option 708 includes, for example, "When the mouse is allowed to move to the edge of the screen, it can continue to move into the screen of another device logged in the same account to connect the device". The prompt information corresponding to the option 708 can also include more or less information, which is not limited in the embodiments of the present application.

[0130] The hand-eye co-travel option 709 can include an option name (e.g., "hand-eye co-travel") of the option 709, and a switch control 7091 of the hand-eye co-travel function. The switch control 7091 can be used to turn on or turn off the hand-eye co-travel function. For example, when the switch control 7091 is in the off state, the hand-eye co-travel function is in the disabled state. In response to the user triggering the operation of the switch control 7091, the switch control 7091 can be displayed in the on state (as shown in FIG. 7B), and the PC can enable the hand-eye co-travel function. When the switch control 7091 is in the on state, the hand-eye co-travel function is in the enabled state. In response to the user triggering the operation of the switch control 7091, the switch control 7091 can be displayed in the off state, and the PC can disable the hand-eye co-travel function. Figure 7

[0131] The prompt information corresponding to the option 709 includes, for example, "When the keyboard and mouse are shared, the cursor can be switched to the screen being gazed at by using the Ctrl key while gazing at the screen. The function needs to be enabled on both devices. How to use?". The prompt information corresponding to the option 709 can also include more or less information, which is not limited by the embodiments of the present application. Among them, "How to use?" in the prompt information corresponding to the option 709 can be displayed in the form of a hyperlink. In response to the user triggering the operation of "How to use?", the PC can open and display the details in the hyperlink corresponding to "How to use?". Optionally, "How to use?" can be highlighted, so that the user can quickly find the location of the hyperlink, and the user can view the details in the hyperlink, thereby improving the user experience.

[0132] In response to the user triggering the operation of the drop-down control 7072 in the device option 707 of the keyboard and mouse sharing, the PC can display a device list, which can be used to display one or more options of devices that can share the keyboard and mouse with the PC.

[0133] It can be understood that, since one or more devices that can share the keyboard and mouse are selected on the PC, the PC can serve as a master device, and the other devices that can share the keyboard and mouse can serve as slave devices or secondary devices, and the PC can determine the face orientation of the user by judging the included angle reported by the slave devices. Optionally, the master device can also be any electronic device in the multiple devices that can share the keyboard and mouse, which is not limited by the embodiments of the present application. For ease of description, the PC is taken as the master device in the following description.

[0134] For example, Figure 8 ​As shown in a of FIG. 8, the PC can display the device list 803 in the vicinity of the pull-down control 802 in the mouse and keyboard sharing interface 801. Alternatively, the device list 803 can be displayed in any possible position in the mouse and keyboard sharing interface 801, which is not limited in the embodiments of the present application. Taking a tablet and a mobile phone as examples of the devices that can perform mouse and keyboard sharing with the PC, the tablet option 8031 and the mobile phone option 8032 can be displayed in the device list 803. The tablet option 8031 can include the icon corresponding to the tablet and the account name of the tablet, for example, the account name of the tablet is "My Tablet". The mobile phone option 8032 can include the icon corresponding to the mobile phone and the account name of the mobile phone. For example, the account name of the mobile phone is "My Mobile Phone".

[0135] The user can select one or more devices from the device list 803 that the user wants to perform mouse and keyboard sharing with the PC. For example, in response to the user triggering the operation of the tablet option 8031, as shown in b of FIG. 8, the check icon 8033 can be displayed in the tablet option 8031. Then the PC and the tablet can establish a communication connection to perform mouse and keyboard sharing. Figure 8

[0136] Alternatively, in response to the user triggering the operation of the tablet option 8031, the tablet option 8031 can be displayed differently from other device options. For example, the background (such as color) of the tablet option 8031 can be different from other device options, or the icon and / or text in the tablet option 8031 can be highlighted.

[0137] Similarly, in response to the user triggering the operation of the mobile phone option 8032, as shown in b of FIG. 8, the check icon 8034 can be displayed in the mobile phone option 8032. Then the PC and the mobile phone can establish a communication connection to perform mouse and keyboard sharing. Figure 8

[0138] After the user selects one or more devices to perform mouse and keyboard sharing with the PC, in response to the user triggering the operation of the area in the mouse and keyboard sharing interface 801 other than the device list 803, the PC can close (or can be understood as not display) the device list 803, and display the mouse and keyboard sharing interface 901 as shown in a of FIG. 9. Alternatively, in the case that all device options in the device list 803 are selected, the PC can close the device list 803, and display the mouse and keyboard sharing interface 901 as shown in a of FIG. 9. Figure 9 Figure 9

[0139] As shown in a of FIG. 9, in the mouse and keyboard sharing interface 901, the prompt information 9021 of the mouse and keyboard sharing device option 902 is no longer displayed as the prompt information of "unconnected device" in c of FIG. 8, but displays the relevant information of the connected mouse and keyboard sharing device. The relevant information of the specific device can refer to the description of the relevant information of the connected mouse and keyboard sharing device in the mouse and keyboard sharing interface 901 in the embodiments of the present application. Figure 9 Figure 7 Figure 7 ​​​​​​The related information of the device involved in the prompt information 7071 in the corresponding embodiment is not described again. Taking the account name of the logged-in device as an example, the prompt information 7071 can include "my tablet" and "my mobile phone". The related information of the device displayed by the specific prompt information 7071 is not limited in the embodiment of the application, as long as the device can be uniquely identified and the user can distinguish the connected device.

[0140] In the case that the PC establishes a communication connection with the shared device of the keyboard and the mouse, the screen arrangement option 903 can also be displayed in the keyboard and mouse sharing interface 901. The screen arrangement option 903 can include the option name (for example, "screen arrangement") of the option 903, and the editing control 9031. In response to the operation of the user triggering the editing control 9031, the PC can display the screen arrangement pop-up window 904 in the keyboard and mouse sharing interface 901, as shown in b of FIG. 9. Figure 9

[0141] Among them, in the screen arrangement area 9041, the user can adjust the relative position and screen arrangement order between the PC and the other shared device of the keyboard and the mouse. In response to the operation of the user triggering the completion control 9042, the PC can display the keyboard and mouse sharing interface 901 as shown in a of FIG. 9, and the PC can implement the keyboard and mouse sharing function according to the adjusted screen arrangement order. For example, if the user sets the screen of the PC on the left side of the tablet, when the user moves the mouse to the right edge of the PC screen, the mouse will automatically appear on the screen of the tablet. In this way, based on the arrangement order of the screen, the user can naturally move the mouse between electronic devices, so as to more smoothly perform cross-screen operation. Figure 9

[0142] Figure 10 The way of opening the keyboard and mouse sharing function and the hand-eye coordination function in the mobile phone is shown.

[0143] As shown in a of FIG. 10, the interface 1000 for setting the application in the mobile phone can include a search bar, an account area, and a plurality of setting items. Figure 10

[0144] Among them, the search bar can be used to indicate that the mobile phone quickly searches for the search content input by the user, and the search bar can include a search icon and a search prompt, etc. Among them, the search prompt is used to prompt the user that the user can quickly find the content that the user wants to find by inputting a keyword. In response to the operation of the user triggering the search icon, the mobile phone can search for related content based on the keyword in the search box.

[0145] ​​​The account area can display account information, account avatar, cloud storage information, etc. It can also include cloud storage controls and a device finder control. Responding to user actions with the cloud storage controls, users can store and manage their data in the cloud, including pictures, videos, contacts, memos, etc., and keep data synchronized across multiple electronic devices logged into the user's account. Responding to user actions with the device finder control, users can quickly locate their mobile phone.

[0146] Multiple settings can include WLAN, Bluetooth, mobile network, satellite network, multi-device collaboration, desktop and personalization, display and brightness, sound and vibration, notification and status bar, general settings, etc. Each setting can include a setting icon, a setting name, and a control to access the setting details page. For example, taking the multi-device collaboration setting as an example, the multi-device collaboration setting can include a multi-device collaboration icon 1001, a multi-device collaboration name 1002, and a control 1003 to access the multi-device collaboration interface.

[0147] In response to user-triggered access to the multi-device collaboration interface, such as Figure 10 As shown in b, the mobile phone can display a multi-device collaboration interface 1004. The triggering operation can include triggering any location within the multi-device collaboration settings, such as triggering the control 1003 of the multi-device collaboration details page. The multi-device collaboration interface 1004 can include multiple settings, such as sharing, connection, cross-device clipboard, cross-device interoperability, and keyboard / mouse sharing. Each setting can include a setting name and a control to access the setting details page. For example, taking the keyboard / mouse sharing setting as an example, the keyboard / mouse sharing setting can include a keyboard / mouse sharing name 1005 and a control 1006 to access the keyboard / mouse sharing interface.

[0148] In response to a user's action of entering the keyboard and mouse sharing interface, such as Figure 10 As shown in Figure c, the mobile phone can display the keyboard and mouse sharing interface 1007. This triggering operation can include triggering any location within the keyboard and mouse sharing settings, such as triggering control 1006. The keyboard and mouse sharing interface 1007 can include a return control, a keyboard and mouse sharing title, a keyboard and mouse sharing icon, a keyboard and mouse sharing prompt, a keyboard and mouse sharing option 1008, and corresponding prompt information for option 1008.

[0149] The return control is used to exit the current keyboard and mouse sharing interface or return to the previous interface.

[0150] Keyboard and mouse sharing prompts may include, for example, "By dragging the mouse to the edge of the screen and across to another device's screen, you can share your keyboard and mouse across devices and drag and drop files." Keyboard and mouse sharing prompts may also include more or less information, which is not limited in this embodiment.

[0151] The keyboard and mouse sharing option 1008 may include an option name (e.g., "Keyboard and Mouse Sharing") and a switch control 1009 for the keyboard and mouse sharing function. The switch control 1009 can be used to turn the keyboard and mouse sharing function on or off. For example, when the switch control 1009 is in the off state (e.g., ...), ... Figure 10 In the case shown in c), the keyboard and mouse sharing function is disabled. In response to the user triggering the switch control 1009, the switch control 1009 can be displayed as open (e.g., ...). Figure 10 As shown in d), the mobile phone can enable the keyboard and mouse sharing function. When the switch control 1009 is in the open state, the keyboard and mouse sharing function is enabled. In response to the user triggering the switch control 1009, the switch control 1009 can be displayed as closed, and the mobile phone can disable the keyboard and mouse sharing function.

[0152] The prompt message corresponding to option 1008 may include, for example, "This service requires the use of Bluetooth or WLAN for sharing mouse and keyboard operations between devices. By turning on the switch, you agree to the above." The prompt message corresponding to option 1008 may also include more or less information, which is not limited in this embodiment.

[0153] In response to the user triggering the operation of turning on the switch control 1009, the mobile phone can display as follows: Figure 10 The keyboard and mouse sharing interface shown in d is 1007. Figure 10 Compared to the keyboard and mouse sharing interface shown in C (1007), Figure 10 In the keyboard and mouse sharing interface 1007 shown in d, the switch control 1009 is displayed as open, and the keyboard and mouse sharing interface 1007 may also include the hand-eye coordination option 1010.

[0154] The hand-eye coordination option 1010 may include an option name (e.g., "Hand-eye Coordination") and a toggle control 1011 for the hand-eye coordination function. The toggle control 1011 can be used to turn the hand-eye coordination function on or off. For example, when the toggle control 1011 is in the off state, the hand-eye coordination function is disabled. In response to a user triggering the toggle control 1011, the toggle control 1011 may be displayed in the on state (e.g., ...). Figure 10 As shown in d), the mobile phone can enable the hand-eye coordination function. When the switch control 1011 is in the open state, the hand-eye coordination function is enabled. In response to a user triggering the switch control 1011, the switch control 1011 can be displayed in the closed state, and the mobile phone can disable the hand-eye coordination function.

[0155] The prompt information corresponding to option 1010 includes, for example, "When sharing the keyboard and mouse, you can switch the cursor to the screen you are looking at by looking at the screen and pressing the Ctrl key at the same time. This function needs to be enabled on both devices. How to use?". The prompt information corresponding to option 1010 can also include more or less information, which is not limited by the embodiments of the application. Among them, "How to use?" in the prompt information corresponding to option 1010 can be displayed in the form of a hyperlink. In response to the user's operation of triggering "How to use?", the phone can open and display the details in the hyperlink corresponding to "How to use?". Optionally, "How to use?" can be highlighted, so that the user can quickly find the location of the hyperlink, making it convenient for the user to view the details in the hyperlink, thereby improving the user experience.

[0156] Optionally, in Figure 10 In b, in response to the user's operation of triggering the keyboard and mouse sharing interface (such as triggering the operation of control 1006), the phone can also display the keyboard and mouse sharing interface 1007 as shown in d of Figure 10 Instead of c of Figure 2 That is, when displaying the keyboard and mouse sharing interface, the keyboard and mouse sharing interface can simultaneously include the keyboard and mouse sharing option and the hand-eye coordination option, instead of displaying the hand-eye coordination option only when the keyboard and mouse sharing function is enabled.

[0157] The following describes the module interaction diagram of the face orientation-based attention detection method of the embodiments of the application with reference to the software architecture of Figure 11 PC and mobile phone. It can be understood that the PC and the mobile phone can each include various functional modules or applications for implementing the method of the embodiments of the application, such as a face key point detection module, a comprehensive sensing information platform, a multi-mode input subsystem, and a keyboard and mouse sharing application.

[0158] For ease of description and distinction, the PC can be taken as a first electronic device and the mobile phone can be taken as a second electronic device hereinafter. The PC can include a first camera, a first face key point detection module, a first comprehensive sensing information platform, a first multi-mode input subsystem, and a first keyboard and mouse sharing application. The mobile phone includes a second camera, a second face key point detection module, a second comprehensive sensing information platform, a second multi-mode input subsystem, and a second keyboard and mouse sharing application.

[0159] It can be understood that when the PC and the mobile phone respectively enable the keyboard and mouse sharing function and the hand-eye coordination function, the PC and the mobile phone can determine the face orientation of the user in real time or periodically. In this way, whether the user's attention has shifted between the electronic devices can be determined in a timely manner, thereby better implementing the method of the embodiments of the application.

[0160] As shown in Figure 12As shown, the method of the embodiments of the present application mainly includes the following parts: (1) face key point detection, (2) head pose estimation and calculation of the angle between the user's face and the electronic device, (3) face orientation judgment, and (4) keyboard and mouse crossing management. The following will describe these parts respectively.

[0161] (1) Face key point detection.

[0162] Taking the scenario of the user's face turning from the mobile phone to the PC as an example, on the one hand, the first camera in the PC can capture the user's face image and transmit it to the first face key point detection module. The first face key point detection module can detect face key points based on the captured image. On the other hand, the second camera in the mobile phone can also capture the user's face image and transmit it to the second face key point detection module. The second face key point detection module can detect face key points based on the captured image. Wherein, the face key point can also be referred to as the facial feature point, and the face key point refers to a set of specific points identified in the face image, which can correspond to some facial features such as eyes, nose, mouth, and facial contour.

[0163] Taking the first face key point detection module as an example, in a possible implementation, the first face key point detection module can obtain the image captured by the first camera. Optionally, the first camera can be a low-power camera (also referred to as a low-power camera). The image captured by the low-power camera can be a single-channel image, for example, a grayscale image, and the resolution of the image is relatively low, but it does not affect the detection of face key points. In this way, by using a low-power camera, the power consumption of the electronic device can be reduced, and the detection of face key points can also be realized.

[0164] After obtaining the image, the first face key point detection module can calculate the face key points based on the face key point detection model. The face key points can be sparse features. It can be understood that sparse features refer to a set of limited and representative face key points, which can also be understood as a relatively small number of face key points. These key points are sufficient to describe the basic structure and features of the face. In this way, by calculating sparse features, the computational complexity of the electronic device can be reduced while maintaining an effective description of the facial features. That is, the recognition accuracy is maintained while the processing speed and recognition efficiency of the electronic device are improved.

[0165] In the embodiments of the present application, the sparse features can include 6 facial key points, such as the inner and outer corners of the two eyes, and the two corners of the mouth. Of course, the sparse features can include other facial key points, such as one or more of the following, but not limited to: the midpoint of the forehead, the brow, the peak of the eyebrow, the tail of the eyebrow, the pupil center of the eye, the tip of the nose, the two sides of the nose, the upper lip midpoint of the mouth, the lower lip midpoint of the mouth, the chin, and the cheekbone. The embodiments of the present application are not limited.

[0166] In the embodiments of the present application, the face key point detection model can output a rotation matrix representing the sparse features. The rotation matrix can be used to identify the change of the face coordinates relative to the first camera coordinates. For example, the rotation matrix can be a 3x4 matrix. The rotation matrix includes the three-dimensional (x-axis, y-axis, and z-axis) coordinates of the face with the first camera as the origin. The size and form of the specific rotation matrix are not limited in the embodiments of the present application. After the face key point detection model outputs the rotation matrix, the first face key point detection module can pass the rotation matrix to the first comprehensive sensing information platform for head pose estimation.

[0167] It can be understood that the specific implementation of the second face key point detection module is similar to that of the first face key point detection module, which will not be repeated. After the second face key point detection module obtains the rotation matrix corresponding to the sparse features of the user, the second face key point detection module can pass the rotation matrix to the second comprehensive sensing information platform for head pose estimation.

[0168] (2) Head pose estimation and calculation of the angle between the user's face and the electronic device.

[0169] For the first comprehensive sensing information platform, in a possible implementation, the first comprehensive sensing information platform can calculate the head pose in the world coordinate system based on the rotation matrix obtained in the first face key point detection module. The head pose in the world coordinate system can be represented by Euler angles. That is, the first comprehensive sensing information platform can convert the rotation matrix to Euler angles. The conversion of the rotation matrix to Euler angles can refer to related technologies, and will not be repeated. The Euler angles can include a yaw angle, a pitch angle, and a roll angle. These three angles describe the rotation of the face around its own coordinate axis, and can effectively describe the head pose in the three-dimensional space.

[0170] The Yaw angle describes the rotation of a face around its vertical axis (z-axis), representing left or right rotation. A negative Yaw angle indicates leftward rotation, while a positive Yaw angle indicates rightward rotation. For example, a Yaw angle of 0° indicates the face is facing forward; a positive Yaw angle indicates rightward rotation; and a negative Yaw angle indicates leftward rotation.

[0171] The pitch angle describes the rotation of a face around its horizontal axis (y-axis), indicating whether the face is tilted forward or backward. A positive pitch angle occurs when the face is tilted upward, and a negative pitch angle occurs when the face is tilted downward.

[0172] The Roll angle describes the rotation of a face around its vertical axis (x-axis), representing the face rolling along a horizontal plane. The Roll angle is positive when the face rolls to the right and negative when it rolls to the left.

[0173] It is understandable that when a user shifts their attention between multiple electronic devices, it typically corresponds to a scenario where the user's face rotates to the left or right. Therefore, this embodiment of the application can calculate the user's facial orientation based on the yaw angle. This is more in line with the usage scenario of this embodiment of the application, making the calculation more accurate.

[0174] The first integrated sensor information platform can determine the user's face position and orientation based on head posture estimation, and then calculate the angle between the user's face orientation and the PC. The specific relationship between face position, face orientation, and angle can be found in the relevant information of the first angle in Table 1 below.

[0175] For example, such as Figure 12 Figure 'a' illustrates the relationship between the angle between the user's face orientation and the electronic device. In one possible scenario, the user's face is located to the right of the first camera and to the left of the second camera, and the user's face is turning from the phone towards the PC (i.e., the user is looking to the left). In this scenario, assuming the user's yaw angle is 'Yaw', the PC detects a distance of x1 between the first camera and the user's face along the x-axis, a distance of z1 between the first camera and the user's face along the z-axis, and the angle between the line connecting the first camera and the user's face and the user's face orientation is the first angle A1.

[0176] It is understood that, in this embodiment of the application, when the user's head is on the right side of the camera, such as... Figure 12 In parameter 'a', the position of the user's head relative to the first camera, and the distance between the camera and the user's head along the x-axis can be set to a negative value. When the user's head is to the left of the camera, such as... Figure 12In point 'a', the position of the user's head relative to the second camera, and the distance between the camera and the user's head along the x-axis, can be set to a positive value. Of course, values ​​with opposite signs can also be used; this embodiment of the application does not limit this.

[0177] from Figure 12 From the positional and triangular relationships within 'a', we can see that the first included angle A1 satisfies the following formula:

[0178] A1 = arctan(|x1| / z1) - |Yaw|.

[0179] Since the user's face is located to the right of the first camera, the distance x1 between the first camera and the user's face along the x-axis is negative, meaning x1 is less than 0. However, the distance z1 between the first camera and the user's face along the z-axis is greater than 0, therefore arctan(x1 / z1) is less than 0. Because the user's face is facing left, i.e., towards PC, the Yaw angle is negative, meaning Yaw is less than 0. Removing the absolute value from the above formula for the first included angle A1, the first included angle A1 satisfies the following formula:

[0180] A1 = -arctan(x1 / z1) + Yaw.

[0181] Similar to the calculations performed on the first integrated sensor information platform, the second integrated sensor information platform can also calculate the head pose in the world coordinate system based on sparse features, and calculate the angle between the user's head and the mobile phone. The specific relationship between the face position, face orientation, and angle can be found in the relevant information for the second angle in Table 1 below.

[0182] like Figure 12 As shown in Figure a, when the user's face is located to the left of the second camera, and the user's face is turning from the phone to the PC, assuming the user's yaw angle is Yaw, the phone detects a distance of x2 between the second camera and the user's face along the x-axis, a distance of z2 between the second camera and the user's face along the z-axis, and the angle between the line connecting the second camera and the user's face and the direction the user's face is facing is the second angle B1. From Figure 12 From the trigonometric relationships in triangle a, we can see that the second included angle B1 satisfies the following formula:

[0183] B1 = arctan(|x2| / z2) + |Yaw|.

[0184] Wherein, since the user's face is located at the left of the second camera, the distance x2 between the second camera and the user's face in the x-axis direction is positive, i.e. x2 is greater than 0. And the distance z2 between the second camera and the user's face in the z-axis direction is greater than 0, then arctan(x2 / z2) is greater than 0. Since the face looks left, i.e. the user's face is towards the PC, the Yaw angle is negative, i.e. Yaw is less than 0. Removing the absolute value in the above-mentioned second included angle B1 formula, the second included angle B1 satisfies the following formula:

[0185] B1 = arctan(x2 / z2) - Yaw.

[0186] In another possible scenario, the user's face can also turn from the PC to the mobile phone, and the user's face is located at the right of the first camera and at the left of the second camera. Then the first included angle A2 satisfies the following formula:

[0187] A2 = arctan(|x1| / z1) + |Yaw|.

[0188] Wherein, since the user's face is located at the right of the first camera, the distance x1 between the first camera and the user's face in the x-axis direction is negative, i.e. x1 is less than 0. And the distance z1 between the first camera and the user's face in the z-axis direction is greater than 0, then arctan(x1 / z1) is less than 0. Since the face looks right, i.e. the user's face is towards the mobile phone, the Yaw angle is positive, i.e. Yaw is greater than 0. Therefore, removing the absolute value in the above-mentioned first included angle A2 formula, the first included angle A2 satisfies the following formula:

[0189] A2 = -arctan(x1 / z1) + Yaw.

[0190] The second included angle B2 satisfies the following formula:

[0191] B2 = arctan(|x2| / z2) - |Yaw|.

[0192] Wherein, since the user's face is located at the left of the second camera, the distance x2 between the second camera and the user's face in the x-axis direction is positive, i.e. x2 is greater than 0. And the distance z2 between the second camera and the user's face in the z-axis direction is greater than 0, then arctan(x2 / z2) is greater than 0. Since the face looks right, i.e. the user's face is towards the mobile phone, the Yaw angle is positive, i.e. Yaw is greater than 0. Removing the absolute value in the above-mentioned second included angle B2 formula, the second included angle B2 satisfies the following formula:

[0193] B2 = arctan(x2 / z2) - Yaw.

[0194] Table 1

[0195]

[0196] It's understandable that PCs and mobile phones might not be placed on the same horizontal plane, such as... Figure 12 As shown in b, but the calculation method is the same. Figure 12 The same applies to point 'a', so I won't elaborate further. It's important to note that because the PC and mobile phone are not placed on the same horizontal line, the Yaw angle calculated by the PC will differ from that calculated by the mobile phone. For example... Figure 4 In step b, the PC calculates the Yaw angle as Yaw1, and the phone calculates the Yaw angle as Yaw2. Therefore, when calculating the first included angle, the PC needs to use Yaw1. When calculating the second included angle, the phone needs to use Yaw2.

[0197] (3) Judging by the direction of the face.

[0198] Since the PC is the master device and the mobile phone is the slave device, the second included angle calculated by the mobile phone needs to be sent to the PC, which then judges the first and second included angles. Therefore, after the second integrated sensor information platform determines the second included angle between the mobile phone and the user's face, it can instruct the mobile phone to send the second included angle to the PC. In this embodiment, the PC and the mobile phone can communicate and transmit data based on a distributed soft bus. Optionally, when the mobile phone sends the second included angle to the PC, it can also send information such as the mobile phone's device identifier. That is, when the slave device sends the included angle information to the master device, it can also send the slave device's device identifier, such as device ID. This makes it easier for the master device to distinguish between the multiple included angles reported and the correspondence between different slave devices, facilitating subsequent focus switching control.

[0199] After the PC obtains the second angle from the mobile phone, the first integrated sensor information platform can determine the user's face orientation based on the first and second angles.

[0200] If the first included angle is smaller than the second included angle, the first integrated sensor information platform can determine that the user's face is facing the PC.

[0201] If the first included angle is greater than or equal to the second included angle, the first integrated sensor information platform can determine that the user's face is facing the mobile phone, which can also be understood as the focus of the input device still being on the mobile phone screen, and no focus switching is required.

[0202] (4) Keyboard and mouse crossover management.

[0203] In the case that the first included angle is less than the second included angle, the user's face is towards the PC, and the first comprehensive sensing information platform can deliver indication information to the first keyboard-mouse sharing application, which can be used to instruct the first keyboard-mouse sharing application to display focus switching information. After the first keyboard-mouse sharing application obtains the indication information, it can display prompt information of focus switching in the PC interface. The specific way of displaying prompt information of focus switching in the PC interface can refer to Figure 5 The related description of prompt information 401 in the corresponding embodiment will not be repeated.

[0204] In response to the user triggering the operation of the Ctrl key, the first multi-mode input subsystem can deliver a keyboard-mouse crossing request to the first keyboard-mouse sharing application, and the first keyboard-mouse sharing application can send a keyboard-mouse crossing request to the second keyboard-mouse sharing application based on the device identifier and other information of the mobile phone. After the second keyboard-mouse sharing application receives the keyboard-mouse crossing request, it can deliver the keyboard-mouse crossing request to the second multi-mode input subsystem.

[0205] After the second multi-mode input subsystem obtains the keyboard-mouse crossing request, it can deliver indication information to the second keyboard-mouse sharing application, which is used to instruct the second keyboard-mouse sharing application to hide the keyboard-mouse focus. After the second keyboard-mouse sharing application obtains the indication information, it can hide the keyboard-mouse focus in the mobile phone interface. After the keyboard-mouse focus is hidden, the second multi-mode input subsystem can return a response (referred to as a request response) of the keyboard-mouse crossing request to the second keyboard-mouse sharing application, and the second keyboard-mouse sharing application can further return the request response to the first keyboard-mouse sharing application. After the first keyboard-mouse sharing application receives the request response, it can deliver the request response to the first multi-mode input subsystem.

[0206] After the first multi-mode input subsystem obtains the request response, it can deliver indication information to the first keyboard-mouse sharing application, which is used to instruct the first keyboard-mouse sharing application to display the keyboard-mouse focus. After the first keyboard-mouse sharing application obtains the indication information, it can display the keyboard-mouse focus in the PC interface. The specific way of displaying the keyboard-mouse focus in the PC can use Figure 6 and Figure 4 The related description in the corresponding embodiment will not be repeated.

[0207] In the case that the first included angle is greater than or equal to the second included angle, the user's face is towards the mobile phone, and no focus switching is needed. In a possible scenario, if the user's face turns from the PC to the mobile phone, the first comprehensive sensing information platform can send indication information to the second comprehensive sensing information platform, which is used to instruct the second keyboard-mouse sharing application to display focus switching information. The specific way of displaying prompt information of focus switching in the mobile phone interface can refer to Figure 11 The related description of prompt information 401 in the corresponding embodiment will not be repeated.

[0208] According to Figure 12Corresponding embodiments and Figure 13 As can be known from the above description of the corresponding embodiments, when implementing the method of the embodiments of the present application, it is necessary to determine the face orientation of the user based on multiple electronic devices. As shown in Figure 14 , taking a first electronic device (such as a PC) and a second electronic device (such as a mobile phone) as an example, the first electronic device and the second electronic device need to respectively acquire images containing the face of the user, and respectively calculate the face key points and perform head pose estimation. In this way, the first electronic device can obtain a first included angle with the face of the user, and the second electronic device can obtain a second included angle with the face of the user. Taking the first electronic device as the master device and the second electronic device as the slave device as an example, the second electronic device can send the second included angle to the first electronic device. The second electronic device can determine the face orientation of the user according to the relationship between the first included angle and the second included angle. Optionally, the determination of the first included angle and the second included angle can be performed by the slave device (i.e., the second electronic device). That is, the process of determining the face orientation of the user based on multiple included angles can be performed by any one of the multiple keyboard-mouse sharing devices, and the embodiments of the present application are not limited in this regard.

[0209] In a possible implementation, the attention detection method based on the face orientation of the embodiments of the present application can also be applied in other use scenarios. Some possible extended scenarios are exemplarily introduced below. In actual applications, the method of the embodiments of the present application can also be applied to more scenarios, which are not limited in this regard.

[0210] Extended scenario 1: application continuation scenario.

[0211] In a possible scenario, the application continuation function can be implemented between multiple electronic devices. The application continuation function can be understood as a function of seamlessly switching and continuing to use a certain application program between multiple electronic devices. Exemplarily, taking a PC and a mobile phone as an example, as shown in Figure 14 , the mobile phone can display an editing interface 1400 of a document application, and the user can edit the document content in the editing interface 1400. The embodiments of the present application do not limit the specific document content.

[0212] When the mobile phone and the PC are close to each other, as shown in Figure 14 , the mobile phone can display the editing interface 1400 of the document application, and the user can continue to edit the document content in the editing interface 1400.As shown in b of FIG. 14, the PC can display a continuation icon 1402 in the interface 1401. For example, the PC can display the continuation icon 1402 in a taskbar, a shortcut window or other interface entry of the interface 1401, which is not limited in the embodiment of the present application. The continuation icon 1402 is used to prompt the user that the document application can be continued by the PC. The continuation icon 1402 can include one or more of the following information, but is not limited to: an application icon of the document application, a device type icon of the mobile phone, a device type name of the mobile phone. For example, the continuation icon 1402 can display the device type icon 1403 of the mobile phone in the upper right corner of the document application icon. The display style of the continuation icon 1402 is not limited in the embodiment of the present application.

[0213] As shown in c of FIG. 14, the PC can display an editing interface 1404 of the document application in the interface 1401. It can be understood that the content in the editing interface 1404 of the PC and the content and editing progress in the editing interface 1400 of the mobile phone are the same. In this way, the user can continue to edit the document content on the PC. Figure 15

[0214] In order to better prompt the user, the PC can play an animation effect of the continuation icon 1402, for example, the PC can display the continuation icon 1402 in a flashing manner for a period of time, and the like. The display style of the animation effect is not limited in the embodiment of the present application.

[0215] However, in some scenarios, when the PC plays the animation effect of the continuation icon 1402, the user's attention may still be on the mobile phone. When the user shifts his attention from the mobile phone to the PC, the animation effect of the continuation icon 1402 on the PC has already been played. This can cause the user to miss the prompt information of the application continuation, so that the user cannot use the application continuation function on the PC well, and the user experience is reduced.

[0216] In the embodiment of the present application, in combination with the face orientation attention detection method, when the PC displays the continuation icon 1402, the PC can first determine the face orientation of the user. When the PC determines that the face of the user is oriented to the PC, that is, the user shifts his attention from the mobile phone to the PC, the PC starts to display the animation effect of the continuation icon 1402. In this way, the user can timely notice the prompt information of the application continuation, so that the user can use the application continuation function on the PC well, thereby improving the user experience.

[0217] Extended scenario 2: scenario of protecting screen content.

[0218] In possible scenarios, the user can edit a document, watch a video, chat on social media, and the like using an electronic device. Taking that the electronic device is a PC as an example, as shown in a of FIG. 14, the user can edit a document on the PC, and as shown in b of FIG. 14, the PC can display a continuation icon 1402 in the interface 1401. For example, the PC can display the continuation icon 1402 in a taskbar, a shortcut window or other interface entry of the interface 1401, which is not limited in the embodiment of the present application. The continuation icon 1402 is used to prompt the user that the document application can be continued by the PC. The continuation icon 1402 can include one or more of the following information, but is not limited to: an application icon of the document application, a device type icon of the mobile phone, a device type name of the mobile phone. For example, the continuation icon 1402 can display the device type icon 1403 of the mobile phone in the upper right corner of the document application icon. The display style of the continuation icon 1402 is not limited in the embodiment of the present application. Figure 15 ​As shown in a of FIG. 15, the PC can display a document interface 1501 in the interface 1500, and the user can edit or view the document in the document interface 1501. When other people are close to the user, for example, communicate with the user, the PC can determine whether the content displayed in the interface 1500 needs to be adjusted or not displayed by detecting the head orientation of the user.

[0219] In a possible implementation, as shown in b of FIG. 15, when the user does not look at the PC, the PC can detect that the included angle formed by the camera and the head orientation of the user is greater than or equal to a preset angle threshold, and then the PC can adjust the content displayed in the interface 1500. The adjustment of the content displayed in the interface 1500 can include but is not limited to the following display modes: adding a mask on the interface 1500 (as shown in c of FIG. 15), blurring the interface 1500, and displaying preset content on the interface 1500. The preset content can include content preset by the electronic device or the user (such as a preset picture). Figure 15 Figure 15

[0220] Alternatively, when the PC detects that the included angle formed by the camera and the head orientation of the user is greater than or equal to the preset angle threshold, the interface 1500 of the PC can not display the document interface 1501. The non-display of the document interface 1501 can include but is not limited to the following display modes: minimizing the document interface 1501, closing the document application, displaying the desktop of the PC, displaying the lock screen interface of the PC (as shown in d of FIG. 15), and turning off the screen of the PC. Figure 16

[0221] In this way, the adjustment of the display content of the interface or the non-display of the interface can make the content displayed in the interface blurred or blocked, so that other people cannot clearly identify the content displayed in the interface, and the information of the user is better protected.

[0222] Optionally, in the screen-off scenario of the PC, when the PC detects that no operation triggered by the user is received within a preset time period, the PC can turn off the screen. The screen-off can also be referred to as screen-out, and the screen-off can be understood as a state in which the screen of the PC is turned off or darkened to save battery power or prevent accidental touch. After the screen is turned off, the screen of the PC no longer displays content, but the PC can still be running.

[0223] However, in some scenarios, although the user does not trigger any operation, the user can actually be watching the content displayed in the interface 1500 all the time, for example, the user is watching the content in the document interface 1501. In this scenario, if the PC turns off the screen, it will affect the user to watch the document, thereby reducing the user experience.

[0224] ​​​Therefore, in this embodiment, if the electronic device does not receive any user-triggered operation within a preset time period, the electronic device can also determine the user's face orientation. If the user's face is not facing the electronic device within the preset time period, it indicates that the user is not viewing the content displayed on the screen, and the electronic device can turn off the screen. If the user's face is facing the electronic device within the preset time period, it indicates that the user may be viewing the content displayed on the screen, and the electronic device will not turn off the screen. In this way, the electronic device can more accurately determine the timing of screen turning off, thereby improving the user experience. It is understood that the above optimization for the screen-off scenario also applies to the lock screen scenario, and will not be elaborated further.

[0225] Extended Scenario 3: Call Scenario.

[0226] In possible scenarios, users can use electronic devices to make voice or video calls. Taking a PC as an example... Figure 16 As shown in Figure a, the PC interface 1600 can display the call interface 1601. The call interface 1601 can include the call duration, the other party's avatar, the other party's name, a microphone control 1602, a hang-up control, and a speaker control. During the call, the microphone control 1602 is in an on state (e.g., ...). Figure 16 As shown in Figure a), the electronic device can send the user's audio data to the other party's electronic device. When someone else is near the user, such as when communicating with the user, the electronic device can determine whether to mute the call by detecting the user's head orientation in order to prevent the other party from hearing the user's conversation with other people.

[0227] In possible implementations, such as Figure 16 As shown in b, when the user is not looking at the PC, the PC can detect that the angle formed by the camera 1603 and the user's head orientation is greater than or equal to a preset angle threshold. In this case, the PC can set the microphone control 1602 to a call mute state (e.g., ...). Figure 17 (As shown in c). The call muting state can include the electronic device not using audio data, and also the electronic device not sending the collected audio data to the other party's electronic device. This allows for the reasonable protection of user information based on the user's head orientation, thereby improving the user experience.

[0228] Figure 11 A schematic diagram of an attention detection method based on face orientation according to an embodiment of this application is shown. The method is applied to a first electronic device and includes:

[0229] S1701. Determine the first angle between the first electronic device and the user's face, and obtain the second angle from the second electronic device; wherein the second angle is the angle between the second electronic device and the user's face, and the second electronic device responds to the input event of the input device.

[0230] In the embodiments of the present application, the first electronic device can be any electronic device with the keyboard-mouse sharing function and the hand-eye coordination function. Moreover, when the first electronic device and other devices implement the method of the embodiments of the present application, the first electronic device can act as the master device. The master device can be understood as the initiator of the keyboard-mouse sharing function, and one or more devices that can perform keyboard-mouse sharing can be selected in the master device. The master device can also receive and process data from other keyboard-mouse sharing devices. For example, the first electronic device can include the PC in the above embodiments.

[0231] The second electronic device can be any electronic device with the keyboard-mouse sharing function and the hand-eye coordination function. Moreover, when the second electronic device and other devices implement the method of the embodiments of the present application, the second electronic device can act as the slave device (or secondary device). The slave device can send the second included angle, device identifier and other related information to the master device. For example, the second electronic device can include the mobile phone in the above embodiments.

[0232] The input device can include a keyboard, mouse or touch screen and other devices that can be used for sharing between multiple electronic devices. It can be understood that in the case where the second electronic device responds to the input event of the input device, the interface of the second electronic device can also display the focus of the input device. The focus can include a cursor or mouse pointer.

[0233] The specific determination of the first included angle between the first electronic device and the face of the user can refer to the related description in Figure 11 The related description in the (2) head pose estimation and the calculation of the included angle between the face and the device of the corresponding embodiments will not be repeated.

[0234] S1702, in the case where the first included angle is less than the second included angle, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device.

[0235] In the embodiments of the present application, the first included angle is less than the second included angle, which means that the included angle between the face of the user and the first electronic device is smaller, the included angle between the face of the user and the second electronic device is larger, i.e. the face of the user is towards the first electronic device. In this case, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, which can also be understood as the focus of the input device is switched from the second electronic device to the first electronic device. The specific focus switching method can refer to the related description in Figure 17 The related description in the (4) keyboard-mouse crossing management of the corresponding embodiments will not be repeated.

[0236] It can be understood that if the first included angle is greater than or equal to the second included angle, it indicates that the face of the user is facing the second electronic device. That is, the attention of the user is still on the second electronic device, and the face of the user is not turned to the first electronic device. Therefore, in this case, the focus of the input device is still on the second electronic device, that is, the electronic device responding to the input event of the input device is still the second electronic device, and there is no need to switch to the first electronic device.

[0237] S1703, in response to the user triggering the first operation of the input device, performing an input event corresponding to the first operation.

[0238] In a possible scenario, after step S1703, if the face of the user is again facing the second electronic device, the electronic device responding to the input event of the input device can be switched from the first electronic device to the second electronic device. In response to the user triggering the operation of the input device, the input event corresponding to the operation is performed by the second electronic device.

[0239] In the embodiments of the present application, the plurality of electronic devices (including the first electronic device and the second electronic device) sharing the input device can respectively calculate the included angle with the face of the user (including the first included angle and the second included angle), comprehensively judge the direction of the face of the user, and control the switching of the focus of the input device on different electronic devices. In this way, by judging the direction of the face, the focus of the input device can be quickly switched on different electronic devices, without the user tediously switching the focus of the input device by clicking the mouse or the screen, thereby improving the user experience.

[0240] Optionally, in the embodiments of the corresponding embodiments, Figure 5 Based on the corresponding embodiments, after the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, the embodiments can further include: the interface of the first electronic device displays first prompt information, and the first prompt information is used to indicate that the first electronic device responds to the input event of the input device.

[0241] In the embodiments of the present application, the first prompt information can also be understood as prompt information of the switching of the focus (from the second electronic device to the first electronic device). The first prompt information can include but is not limited to one or more of the following information: icon type prompt information, text type prompt information, mouse prompt. The mouse prompt can include mouse cursor prompt and mouse pointer prompt. The mouse pointer prompt can include animation effect of the mouse pointer and / or animation effect of water ripples around the mouse pointer. For specific mouse pointer prompt mode, refer to the description of the mouse pointer prompt in the corresponding embodiments and Figure 6 For the related description in the corresponding embodiments and Figure 17 For the related description in the corresponding embodiments, no longer be repeated.

[0242] It can be understood that through the first prompt information, for example, the mouse animation prompt information, the user can learn in time that the focus of the keyboard and the mouse has been switched from the second electronic device to the first electronic device. By using the mouse animation prompt information, the user can also quickly find the position of the mouse pointer, thereby improving the user experience.

[0243] Optionally, in the Figure 17 According to the corresponding embodiment, the first prompt information comprises one or more types of prompt information, such as icon type prompt information, text type prompt information, and mouse prompt.

[0244] It can be understood that the explanation of the first prompt information can refer to the related description of the first prompt information in the above optional embodiments, which will not be repeated here.

[0245] In the embodiments of the present application, the icon type prompt information can convey information through visual symbols, making it easier for the user to identify and understand the meaning expressed by the first prompt information. By using the text type prompt information, the user can better understand which electronic device the focus of the current input device is on, making the interface more interactive and humanized. By using the mouse prompt, the user can quickly find the position of the mouse pointer or the mouse cursor, making it convenient for the user to make further editing or other operations. That is, through the first prompt information, the user can learn in time that the focus of the keyboard and the mouse has been switched from the second electronic device to the first electronic device, thereby improving the user experience of using the hand-eye coordination function.

[0246] Optionally, in the Figure 5 According to the corresponding embodiment, the mouse prompt comprises that the interface of the first electronic device displays an animation effect of the mouse pointer, and the animation effect of the mouse pointer comprises one or more animation effects, such as size change of the mouse pointer, color change of the mouse pointer, light effect change of the edge of the mouse pointer, and change around the mouse pointer.

[0247] In the embodiments of the present application, the animation effect of the mouse pointer can refer to the related description in the corresponding embodiment, which will not be repeated here. The color change of the mouse pointer can include color tone change, color brightness change, and / or color transparency change, etc., which are not limited in the embodiments of the present application. Figure 6

[0248] The change around the mouse pointer can include a water ripple animation effect around the mouse pointer. The specific water ripple animation effect around the mouse pointer can refer to the related description in the corresponding embodiment, which will not be repeated here. Figure 17

[0249] ​​It can be understood that, by taking the animation effect of the mouse pointer and the animation effect of the water ripples as the prompt information of the focus switching, the user can know in time that the focus of the keyboard and the mouse has been switched to the first electronic device, so as to facilitate the user to quickly find the position of the mouse pointer, and improve the user experience.

[0250] Optionally, in the Figure 4 On the basis of the corresponding embodiment, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, and the method can further include: displaying second prompt information on the interface of the first electronic device, the second prompt information being used to indicate that the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device by triggering a second operation; and switching the electronic device responding to the input event of the input device from the second electronic device to the first electronic device in response to the second operation.

[0251] In the embodiment, the second prompt information can also be understood as information used to prompt the user that the current gaze is the first electronic device, and can also be used to indicate that the user can perform the focus switching. The second prompt information can include icon prompt and / or text prompt. For example, the second prompt information can be Figure 17 The prompt information 401 in the corresponding embodiment.

[0252] The second operation can include a key operation on the input device, where the key can include a Ctrl key, an Enter key, a shift key, a space key, or a click on a mouse, and the embodiment is not limited in this regard.

[0253] It can be understood that, by the second prompt information, the user can quickly understand how to switch the focus of the input device between different electronic devices, improve the accuracy of operation, and thus make the user more skilled in using the keyboard-mouse sharing function and the hand-eye coordination function.

[0254] Optionally, in the Figure 17 On the basis of the corresponding embodiment, the second operation includes a key operation on the input device.

[0255] It can be understood that the explanation of the second operation can refer to the related description of the second operation in the above optional embodiment, and will not be repeated here.

[0256] In the embodiment, the first electronic device responds to the input event of the input device by triggering the second operation by the user. In this way, the user can be allowed to select whether to switch the focus of the input device according to actual needs, increase the selection and setting of certain functions of the electronic device by the user, improve the interactivity with the user, and thus improve the user experience.

[0257] Optionally, in theFigure 8 On the basis of the corresponding embodiment, before determining the first included angle between the first electronic device and the face of the user and obtaining the second included angle from the second electronic device, the method can further include: receiving an operation of setting the second electronic device as an electronic device sharing an input device with the first electronic device.

[0258] In the embodiment of the application, the operation of receiving the operation of setting the second electronic device as an electronic device sharing an input device with the first electronic device can also be understood as an operation of setting the second electronic device as a slave device, or an operation of adding the second electronic device for keyboard and mouse sharing in the first electronic device. For example, taking the first electronic device as a PC and the second electronic device as a tablet or a mobile phone, the operation can include Figure 17 In the corresponding embodiment, the operation of triggering the tablet option 8031 by the user or the operation of triggering the mobile phone option 8032 by the user.

[0259] By setting the second electronic device as an electronic device sharing an input device with the first electronic device, the keyboard and mouse sharing function and the hand-eye coordination function can be realized between the first electronic device and the second electronic device. The user can conveniently move the mouse or share the input device between multiple electronic devices, so that the cross-screen operation is more smooth.

[0260] Optionally, in the embodiment of the application, Figure 17 On the basis of the corresponding embodiment, the input device is a device contained in the first electronic device, or the input device is a device independent of the first electronic device.

[0261] In the embodiment of the application, the input device is a device contained in the first electronic device, which can also be understood as that the input device is a built-in device in the first electronic device. The built-in device can be closely integrated with other parts or other modules in the first electronic device, and there is no need to prepare an additional input device or to make additional settings or connections, which is convenient and fast. For example, if the first electronic device is a notebook computer, the input device can be a keyboard, a touchpad, etc. of the notebook computer.

[0262] The input device is a device independent of the first electronic device, which can also be understood as that the input device is an external device of the first electronic device. The user can select a type of input device that meets personal preferences, and the user can also adjust the position and angle of the external device according to needs, to provide a more flexible input environment.

[0263] Optionally, in the embodiment of the application, Figure 17 On the basis of the corresponding embodiment, the input device includes a keyboard or a mouse.

[0264] Optionally, in the embodiment of the application, Figure 11Corresponding to the embodiment, the first electronic device includes a first camera, the second electronic device includes a second camera, the first included angle is an included angle between the first camera and the face orientation, and the second included angle is an included angle between the second camera and the face orientation.

[0265] In the embodiment, the first included angle can include Figure 11 The second included angle can include Figure 11 The second included angle B1 or the second included angle B2 in the corresponding embodiment. For specific first included angles and second included angles, refer to Figure 12 The corresponding embodiment and Figure 17 The related description in the corresponding embodiment will not be repeated.

[0266] It can be understood that the face orientation of the user can be comprehensively determined through the included angles related to the plurality of electronic devices (including the first included angle and the second included angle). Compared with the determination of the face orientation of the user by a single electronic device, the determination based on the included angles of the plurality of electronic devices can reduce the fluctuation of data and the inaccuracy of the calculation result, reduce the determination error, and thus more accurately control the switching of the focus of the input device on different electronic devices.

[0267] Optionally, in Figure 15 Corresponding to the embodiment, the method can further include: the first electronic device displays a first interface; determining a third included angle between the first electronic device and the face of the user; and if the third included angle is greater than or equal to a preset angle threshold, adjusting the display content of the first interface or not displaying the first interface.

[0268] In the embodiment, the first interface can include Figure 15 The interface 1500 in the corresponding embodiment.

[0269] The third included angle greater than or equal to the preset angle threshold can be understood as a scenario in which the user does not gaze at the first electronic device. The preset angle threshold can be a preset angle in the first electronic device, for example, the preset angle threshold can be greater than or equal to 45°, and the specific value of the preset angle threshold is not limited in the embodiment.

[0270] The adjustment of the display content of the first interface can refer to the related description of the adjustment of the content displayed by the interface 1500 in the corresponding embodiment. Figure 15 The adjustment of the display content of the first interface can refer to the related description of the adjustment of the content displayed by the interface 1500 in the corresponding embodiment. Figure 17 The adjustment of the display content of the first interface can refer to the related description of the adjustment of the content displayed by the interface 1500 in the corresponding embodiment.

[0271] Optional, in Figure 17 Based on the corresponding embodiments, the method may further include: the first electronic device displaying the second interface; determining the fourth angle between the first electronic device and the user's face; if the duration of the fourth angle being greater than or equal to a preset angle threshold continues to exceed a preset duration, entering a screen-off state or a screen-locked state.

[0272] In this embodiment, the preset duration can be understood as the duration during which the first electronic device determines that it has not received any user-triggered operation before entering a screen-off or screen-locked state. This preset duration can be pre-set by the electronic device or by the user; this embodiment does not impose any limitations on this.

[0273] If the electronic device does not receive any user-triggered actions within a preset time period, it can also determine the user's facial orientation. If the user's face is not facing the electronic device within this preset time period, it indicates that the user is not viewing the content displayed on the screen, and the electronic device can turn off or lock the screen. If the user's face is facing the electronic device within this preset time period, it indicates that the user may be viewing the content displayed on the screen, and the electronic device will not turn off or lock the screen. In this way, the electronic device can more accurately determine when to turn off or lock the screen, thereby improving the user experience.

[0274] Optional, in Figure 16 Based on the corresponding embodiments, the method may further include: the first electronic device displaying a call interface; determining a fifth angle between the first electronic device and the user's face; and, if the fifth angle is greater than or equal to a preset angle threshold, the first electronic device activating a call mute function.

[0275] In this embodiment of the application, the call interface may include... Figure 16 The corresponding embodiment shows the call interface 1601.

[0276] by Figure 16 Taking the corresponding embodiment as an example, the first electronic device activating the call mute function can be understood as the first electronic device setting the microphone control 1602 to a call mute state (e.g., Figure 16 As shown in Figure c), the microphone control 1602 is in the off state at this time. The call mute state can include the electronic device not using audio data, and also the electronic device not sending the collected audio data to the other party's electronic device. This allows for reasonable protection of user information based on the user's head orientation, thereby improving the user experience.

[0277] Understandably, when the fifth included angle is less than a preset angle threshold, the first electronic device can disable the call mute function. At this time, the microphone control 1602 is in the open state (e.g., ...). Figure 17 (as shown in a).

[0278] Optional, inFigure 14 On the basis of the corresponding embodiment, the method can further include: the first electronic device establishes a communication connection with a third electronic device, and the third electronic device displays an interface of the first application; if the distance between the first electronic device and the third electronic device is less than a preset distance, and a sixth included angle is less than a preset angle threshold, the first electronic device displays third prompt information, and the third prompt information includes an icon of the first application; wherein the sixth included angle is an included angle between the first electronic device and the face of the user.

[0279] In the embodiment of the application, the third electronic device can be any electronic device capable of application continuation with the first electronic device. After the first electronic device establishes a communication connection with the third electronic device, application continuation can be achieved. The interface of the first application can include Figure 14 The editing interface 1400 of the document application in the corresponding embodiment. Take Figure 17 For example, the corresponding embodiment, the first electronic device can be a PC, and the third electronic device can be a mobile phone.

[0280] The preset distance can be understood as a distance that enables the first electronic device and the third electronic device to achieve the application continuation function. The third prompt information can be understood as the continuation icon 1402 or the animation effect of the continuation icon 1402.

[0281] It can be understood that when the first electronic device displays the third prompt information, the face orientation of the user can be determined first. When the first electronic device determines that the face of the user is oriented towards the first electronic device, that is, the attention of the user is transferred from the third electronic device to the first electronic device, the first electronic device starts to display the third prompt information. In this way, the user can promptly notice the third prompt information of the application continuation, so that the user can better use the application continuation function on the first electronic device, thereby improving the user experience.

[0282] Optionally, in the Figure 11 On the basis of the corresponding embodiment, before determining the first included angle between the first electronic device and the face of the user, the method can further include: acquiring an image containing the face of the user, determining key point information of the face of the user based on the image; determining head posture information of the user based on the key point information, the head posture information including a yaw angle of the head of the user; and determining the first included angle between the first electronic device and the face of the user, including: determining the first included angle based on the head posture information of the user, the distance between the camera and the face of the user in the horizontal direction, and the distance between the camera and the face of the user in the vertical direction.

[0283] In the embodiment of the application, the face key points of the user can be represented by sparse features of the face. Determining the key point information of the face of the user based on the image can refer to Figure 11The related descriptions of (1) face key point detection in the corresponding embodiments will not be repeated. The face key points can include the inner and outer corners of the two eyes and the two corners of the mouth. The face feature information included by the specific face key points is not limited in the embodiments of the present application.

[0284] The determination of the head pose information of the user based on the key point information and the determination of the first included angle can refer to Figure 11 The related descriptions of (2) head pose estimation and calculation of the included angle between the face and the device in the corresponding embodiments will not be repeated. The yaw angle of the head of the user can be understood as the rotation angle of the head of the user around the vertical axis (such as the z-axis) of the coordinate axis of the head of the user, which is used to represent the left and right rotation of the head of the user. For details, refer to Figure 18 The related descriptions in the corresponding embodiments. It can be understood that the sparse features are a relatively small number of face key points, which are sufficient to describe the basic structure and features of the face. By calculating the sparse features, the computational complexity of the electronic device can be reduced while maintaining effective description of the features of the face, and the processing speed and recognition efficiency of the electronic device can be improved while maintaining the recognition accuracy.

[0285] The above detailed description of the specific embodiments has described the purposes, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application. The module names involved in the embodiments of the present application can be defined as other names, as long as the functions of the modules can be realized, and the names of the modules are not limited specifically.

[0286] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for the user to choose authorization or refusal.

[0287] The above describes the technical solutions provided by the embodiments of the present application from the method aspect. To implement the above functions, the technical solutions comprise hardware structures and / or software modules corresponding to each function. Those skilled in the art should easily realize that the method steps described in the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and software. Whether a certain function is implemented in the form of hardware or software driven hardware depends on the specific application of the technical solution and design constraints. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0288] The embodiments of the present application provide an electronic device, which can comprise one or more processors and a memory. The memory is coupled to the one or more processors. The memory can be used to store computer program code comprising computer instructions. The one or more processors can invoke the computer instructions to cause the electronic device to perform the technical solutions in the above embodiments.

[0289] The embodiments of the present application provide a chip system. The chip system is applied to an electronic device, and the chip system can comprise one or more processors. The one or more processors can be used to invoke instructions to cause the electronic device to perform the technical solutions in the above embodiments.

[0290] ​ A structural schematic diagram of a chip system provided by the embodiments of the present application is shown in FIG. 18. The chip system 1800 comprises one or more processors 1801, a communication line 1802, a communication interface 1803 and a memory 1804.

[0291] In some embodiments, the memory 1804 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0292] The above method described in the embodiments of the present application can be applied to the processor 1801 or implemented by the processor 1801. The processor 1801 can be an integrated circuit chip having a processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits or instructions in software form of the processor 1801. The above processor 1801 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a DSP, an application specific integrated circuit, a ready programmable gate array or other programmable logic device, a discrete gate, a transistor logic device or a discrete hardware component. The processor 1801 can implement or execute the methods, steps and logic block diagrams related to processing in the embodiments of the present application.

[0293] The method of the embodiments of the present application can be implemented by using a hardware decoding processor to complete, or by using a combination of hardware and software modules in the decoding processor to complete. The software modules can be located in the memory 1804, and the processor 1801 can read the information in the memory 1804 to complete the steps of the above method in combination with the hardware. The processor 1801, the memory 1804, and the communication interface 1803 can communicate through the communication line 1802.

[0294] The embodiments of the present application also provide a readable storage medium (also referred to as a computer readable storage medium). The readable storage medium includes computer instructions. When the computer instructions run on an electronic device, the electronic device executes the technical solutions in the above embodiments.

[0295] The method described in the above embodiments can be implemented by software, hardware, firmware, or any combination thereof, in whole or in part. If implemented in software, the functions can be stored as one or more instructions or programs on a readable medium or transmitted on a readable medium. The readable medium can include a storage medium and a communication medium, and can also include any medium that can carry a program from one place to another. The storage medium can be any target medium that can be accessed.

[0296] In a possible implementation, the readable medium can include random access memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, optical disc read-only memory, or other optical disc memory, etc. The readable medium can also include magnetic disc memory or other magnetic disc storage device, or any other medium targeted for carrying, or storing the required program in the form of instructions or data structures, and can be accessed.

[0297] The embodiments of the present application also provide a program product (also referred to as a computer program product). The program product includes computer program code. When the computer program code runs on an electronic device, the electronic device executes the technical solutions in the above embodiments.

[0298] In various embodiments of the present application, the terms and / or descriptions of various embodiments are consistent and can be mutually referred to if there is no special description and no logical conflict. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0299] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An attention detection method based on facial orientation, characterized in that, The method is applied to a first electronic device, and the method includes: Determine a first angle between the first electronic device and the user's face, and obtain a second angle from the second electronic device; wherein the second angle is the angle between the second electronic device and the user's face, and the second electronic device responds to an input event from an input device; When the first included angle is smaller than the second included angle, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device; In response to a user triggering a first operation on the input device, the input event corresponding to the first operation is executed.

2. The method as described in claim 1, characterized in that, After the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, it further includes: The interface of the first electronic device displays a first prompt message, which is used to instruct the first electronic device to respond to the input event of the input device.

3. The method as described in claim 2, characterized in that, The first prompt message includes one or more of the following types of prompt messages: icon-type prompt messages, text-type prompt messages, and mouse tooltips.

4. The method as described in claim 3, characterized in that, The mouse tooltip includes an animation effect of the mouse pointer displayed on the interface of the first electronic device. The animation effect of the mouse pointer includes one or more of the following: changes in the size of the mouse pointer, changes in the color of the mouse pointer, changes in the glowing effect of the edge of the mouse pointer, and changes around the mouse pointer.

5. The method according to any one of claims 1-4, characterized in that, Before the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, it further includes: The interface of the first electronic device displays a second prompt message, which indicates that by triggering a second operation, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device. The electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device, including: In response to the second operation, the electronic device responding to the input event of the input device is switched from the second electronic device to the first electronic device.

6. The method as described in claim 5, characterized in that, The second operation includes key operations on the input device.

7. The method as described in claim 6, characterized in that, The key operations include any of the following: pressing the Ctrl key, pressing the Enter key, pressing the Shift key, pressing the space bar, or clicking the mouse.

8. The method according to any one of claims 1-4 and 6-7, characterized in that, Before determining the first angle between the first electronic device and the user's face, and before obtaining the second angle from the second electronic device, the method further includes: The system receives an operation to configure the second electronic device as an electronic device that shares the input device with the first electronic device.

9. The method according to any one of claims 1-4 and 6-7, characterized in that, The input device is either a device included in the first electronic device or a device independent of the first electronic device.

10. The method according to any one of claims 1-4 and 6-7, characterized in that, The input device includes a keyboard or a mouse.

11. The method according to any one of claims 1-4 and 6-7, characterized in that, The first electronic device includes a first camera, the second electronic device includes a second camera, the first included angle is the angle between the first camera and the direction the face is facing, and the second included angle is the angle between the second camera and the direction the face is facing.

12. The method according to any one of claims 1-4 and 6-7, characterized in that, The method further includes: The first electronic device displays the first interface; Determine the third angle between the first electronic device and the user's face; If the third included angle is greater than or equal to a preset angle threshold, adjust the displayed content of the first interface, or do not display the first interface.

13. The method according to any one of claims 1-4 and 6-7, characterized in that, The method further includes: The first electronic device displays the second interface; Determine the fourth angle between the first electronic device and the user's face; If the duration for which the fourth included angle is greater than or equal to a preset angle threshold continues to exceed a preset duration, the screen will enter an off state or a locked state.

14. The method according to any one of claims 1-4 and 6-7, characterized in that, The method further includes: The first electronic device displays the call interface; Determine the fifth angle between the first electronic device and the user's face; When the fifth included angle is greater than or equal to a preset angle threshold, the first electronic device activates the call mute function.

15. The method according to any one of claims 1-4 and 6-7, characterized in that, The method further includes: The first electronic device establishes a communication connection with the third electronic device, and the third electronic device displays the interface of the first application; If the distance between the first electronic device and the third electronic device is less than a preset distance, and the sixth included angle is less than a preset angle threshold, the first electronic device displays a third prompt message, which includes the icon of the first application; wherein, the sixth included angle is the angle between the first electronic device and the user's face.

16. The method according to any one of claims 1-4 and 6-7, characterized in that, Before determining the first angle between the first electronic device and the user's face, the method further includes: Acquire an image containing the user's face, and determine key point information of the user's face based on the image; The user's head posture information is determined based on the key point information, and the head posture information includes the yaw angle of the user's head; Determining the first angle between the first electronic device and the user's face includes: The first included angle is determined based on the user's head posture information and the distance between the camera and the user's face in the horizontal and vertical directions.

17. An attention detection method based on facial orientation, characterized in that, The method is applied to a second electronic device, and the method includes: In response to a third operation triggered by the user on the input device, the input event corresponding to the third operation is executed; Determine a second angle between the second electronic device and the user's face, and send the second angle to the first electronic device; If the first included angle is less than the second included angle, the fourth operation triggered by the user on the input device will not be responded to, wherein the first included angle is the angle between the first electronic device and the user's face.

18. An attention detection method based on facial orientation, characterized in that, The method is applied to a communication system, the communication system including a first electronic device and a second electronic device, the method comprising: The second electronic device responds to input events from the input device; The first electronic device determines a first angle between itself and the user's face, and the second electronic device determines a second angle between itself and the user's face; If the first electronic device determines that the first included angle is less than the second included angle, the first electronic device responds to the input event of the input device; The first electronic device responds to a first operation triggered by the user on the input device by executing the input event corresponding to the first operation.

19. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-16, or the method as described in claim 17, or the method performed by the first electronic device in the method as described in claim 18, or the method performed by the second electronic device in the method as described in claim 18.

20. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-16, or to perform the method as described in claim 17, or to perform the method performed by a first electronic device in the method as described in claim 18, or to perform the method performed by a second electronic device in the method as described in claim 18.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-16, or the method as described in claim 17, or the method as described in claim 18 performed by a first electronic device, or the method as described in claim 18 performed by a second electronic device.

22. A computer program product, characterized in that, The computer program product includes computer program code that, when executed on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-16, or the method as described in claim 17, or the method performed by the first electronic device in the method as described in claim 18, or the method performed by the second electronic device in the method as described in claim 18.

Images