Interaction method between devices based on pointing operation and electronic device

Abstract

The invention provides an interaction method between devices based on pointing operation and an electronic device, the electronic device can be a mobile phone and the like, the method is based on an acceleration sensor, a gyroscope, an IMU, a camera and the like of a handheld device, the pointing operation of a user using the handheld device to a target device is detected, and a wireless positioning function of the handheld device is triggered; when it is recognized that the axis of the handheld device intersects with or approximately intersects with the target device, at least one feedback of visual feedback, sound feedback, vibration and the like can be provided on the handheld device and / or the target device, account information and device information logged in by the target device are transmitted to the handheld device, and a control window of the target device is displayed on the handheld device. A user can control the target equipment through the control window on the handheld equipment. According to the method, the user is helped to select farther equipment according to the pointing operation of the user, the process of interactive operation between the equipment is simplified, and the user experience is improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202010781046.7, and the original application date is August 5, 2020. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to an interaction method between devices based on pointing operations and an electronic device. Background Art

[0003] In an environment with massive terminals, each user may have different types of electronic devices, such as smartphones, personal computers, smart TVs, tablets, and speakers. In home scenarios, users may also have smart audio and video equipment, routers, wireless fidelity (WIFI) boxes, smart cleaning equipment, smart kitchen appliances, smart lighting systems, and other home appliances. With the development of technology, in the scenario of the Internet of Everything, users can control more and more devices, and at the same time, the demand for interconnection between multiple devices is increasing.

[0004] When multiple devices are interconnected and interacting at the same time, users are often required to select one or more specific target devices, and then perform different operations on the selected target devices, such as device discovery, pairing, data transmission, screen projection, etc. This process is cumbersome, has many interactive processes, and poor user experience. Summary of the invention

[0005] The present application provides an interaction method between devices based on pointing operations and an electronic device. The method helps a user select and control a farther device according to the user's pointing operations, simplifies the process of interaction operations between devices, and improves the user experience.

[0006] In a first aspect, a method for interaction between devices is provided, which is applied to a first electronic device, and the method includes: obtaining acceleration of the first electronic device in a first axis direction, an acceleration in a second axis direction, and an acceleration in a third axis direction, wherein the first axis is parallel to the long side of the first electronic device, the second axis is parallel to the short side of the first electronic device, and the third axis is perpendicular to a plane defined by the first axis and the second axis; determining that during a movement of a user holding the first electronic device and moving from a first position as a starting position to a second position, the acceleration in the first axis direction is greater than or equal to a first threshold value or the duration of the acceleration in the first axis direction being greater than or equal to the first threshold value is greater than or equal to a first preset duration, the acceleration in the second axis direction is less than or equal to a second threshold value, and the acceleration in the third axis direction is less than or equal to a third threshold value; detecting that at the second position, the first axis of the first electronic device intersects with the second electronic device, or the first axis of the first electronic device approximately intersects with the second electronic device within a preset accuracy range; receiving account information and device information of the second electronic device sent by the second electronic device; and displaying a first window according to the account information and device information of the second electronic device, wherein the first window displays an interface for controlling the second electronic device.

[0007] Optionally, the user's pointing operation can be described as a pointing operation in which the user holds the first electronic device and moves from the first position as the starting position to the second position, and at the second position, the axis of the long side of the first electronic device intersects or approximately intersects with one or more other devices. Here, intersection can be understood as other devices just existing in the direction pointed by the user; approximately intersection can be understood as other devices existing within a preset accuracy range in the direction pointed by the user, and both situations can be understood as the user's pointing operation being detected.

[0008] In the process of determining the pointing operation of the user holding the first electronic device to point to the second electronic device, a coordinate system O-XYZ is established on the first electronic device. Specifically, the coordinate system O-XYZ takes the center of gravity of the first electronic device as the coordinate origin O, the straight line where the long frame of the first electronic device passing through the coordinate origin O is located as the Y axis, that is, the first axis; the straight line where the short frame of the first electronic device is located is the X axis, that is, the second axis; and the straight line perpendicular to the ground or the XOY plane is the Z axis, that is, the third axis.

[0009] It should be understood that the axis of the first electronic device is related to the antenna layout of the first electronic device. When the antenna is arranged at the front frame of the first electronic device, the user can point the axis where the long side is located to the second electronic device, or when the antenna is arranged inside the shell of the first electronic device, the first electronic device can be stood up in a similar posture to taking a photo, with the normal line perpendicular to the display screen of the first electronic device as the axis line, so that the axis line where the normal line is located points to the second electronic device, and the embodiments of the present application are not limited to this.

[0010] In a possible implementation, the user's pointing operation may be identified by means of one or more sensors of the first electronic device, or a camera of the first electronic device, etc. In the description of the subsequent embodiments, the user's pointing operation may be referred to as a "pointing operation".

[0011] Optionally, the sensor of the first electronic device may include an accelerometer, a gyroscope, a magnetometer (M-sensor), an inertial measurement unit (IMU), etc. The camera of the first electronic device is not limited to the front camera or the rear camera of the first electronic device; or the camera of the first electronic device is not limited to one or more of the main camera, the wide-angle camera, and the telephoto camera of the first electronic device. The pointing operation of the user is recognized according to the picture obtained by the camera, and this embodiment of the present application is not limited to this.

[0012] In combination with the first aspect, in certain implementations of the first aspect, obtaining the acceleration of the first electronic device in the direction of a first axis, the acceleration on a second axis, and the acceleration on a third axis includes: based on one or more of an acceleration sensor, an inertial detection unit IMU, and a gyroscope, obtaining the acceleration in the direction of the first axis, the acceleration on the second axis, and the acceleration on the third axis.

[0013] It should be understood that when the user's arm is extended forward, the first electronic device is accelerated along the straight line where the Y axis is located, and the acceleration sensor of the first electronic device detects that the acceleration of the first electronic device along the straight line where the Y axis is located is greater than or equal to the first preset threshold value a. A At the same time, it is detected that the acceleration of the first electronic device on the straight line where the X axis is located and the straight line where the Z axis is located is less than or equal to the second preset threshold a B .

[0014] Optionally, the acceleration of the first electronic device in different directions can be detected by the IMU of the first electronic device. Alternatively, the acceleration of the first electronic device in different directions can be detected based on the picture captured by the front camera of the first electronic device and the normally-on optical flow algorithm. Specifically, the front camera of the first electronic device is used to obtain adjacent frames in the picture, wherein the two-dimensional vector field of the feature points in the adjacent frames during the translation process is a velocity field that represents the three-dimensional motion of the object point through a two-dimensional image. That is, the image changes formed by the movement of the object point within a certain time interval can be reflected according to the velocity field of the three-dimensional motion of the object point, so as to determine the motion direction and motion rate of the object point on the image, and then determine whether the motion trajectory of the first electronic device is a straight line or a quasi-straight line.

[0015] In one possible implementation, during the operation of a user using a first electronic device to point at a second electronic device, the first electronic device can guide the user to perform the correct pointing operation through different methods such as guide icons, sound feedback, vibration feedback, etc., so as to implement the control of the second electronic device through pointing operation provided in an embodiment of the present application.

[0016] Optionally, the sound feedback may be issued by the first electronic device, or by the second electronic device being pointed at, for example, the first electronic device prompts the user in a voice message “please move right”.

[0017] Optionally, the vibration feedback may be emitted by the first electronic device so as to be perceived by the user more quickly and sensitively. It should be understood that the embodiments of the present application are not limited to this.

[0018] Alternatively, after the user successfully points the first electronic device at the second electronic device, feedback may be provided on the first electronic device or the second electronic device being pointed at, and the feedback is used to inform the user that the pointing operation is successful. The feedback information may include one or more of visual feedback, sound feedback, and vibration feedback. For example, after the user successfully points the first electronic device at the speaker, the first electronic device may vibrate to inform the user that the pointing operation is complete.

[0019] It should be understood that after the user successfully points to the second electronic device using the first electronic device, the second electronic device can transmit the currently logged-in account information and device information to the first electronic device, and then display the second electronic device control window on the first electronic device.

[0020] It should also be understood that when determining whether the first electronic device is pointing to the second electronic device, when it is identified that the axis of the first electronic device and the position of the second electronic device are within a preset range, the second electronic device pointed to by the user can be determined. Specifically, when it is identified that the axis of the first electronic device substantially intersects with the physical position of the second electronic device with a certain predetermined accuracy, or when it is identified that the first electronic device is aligned with the second electronic device, the second electronic device pointed to by the user can be determined, that is, feedback can be provided on the first electronic device or the pointed second electronic device, and the login account information and device information of the second electronic device are transmitted to the first electronic device, and the second electronic device control window or the interface related to the second electronic device is displayed on the first electronic device.

[0021] In summary, the embodiment of the present application provides a method for realizing cross-device interaction based on pointing operation, in which a user can use a first electronic device to point to a second electronic device, and based on one or more sensors and / or cameras of the first electronic device, such as an accelerometer, a gyroscope, an IMU, etc., detect the user's pointing operation, and trigger the wireless positioning function of the first electronic device. The first electronic device determines the distance between the first electronic device and other devices, as well as the position and direction of other devices according to the wireless positioning function. When it is identified that the axis of the first electronic device and the physical position of the second electronic device are judged to intersect or substantially intersect with a predetermined accuracy, at least one feedback such as visual feedback, sound feedback, and vibration can also be provided on the first electronic device and / or the second electronic device, and the account information and device information of the second electronic device logged in are transmitted to the first electronic device, and the control window of the second electronic device is displayed on the first electronic device, and the user can control the second electronic device through the control window on the first electronic device. When it is identified that the axis of the first electronic device does not intersect with the physical position of the second electronic device, at least one guidance method such as visual guidance, sound guidance, and vibration can also be provided on the first electronic device and / or the second electronic device to guide the user to perform the correct pointing operation, and further realize the display of the control window of the second electronic device on the first electronic device and the control function of the second electronic device.

[0022] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, before detecting that the first axis of the first electronic device intersects with the second electronic device at the second position, or the first axis of the first electronic device and the second electronic device approximately intersect within a preset accuracy range, the method also includes: capturing a picture during the movement through the camera of the first electronic device; determining that the picture captured by the first electronic device at the first position includes the facial feature information of the user, and the picture captured at the second position does not include the facial feature information of the user.

[0023] Optionally, the first electronic device can detect a state that is far away from the user's body or far away from the user's body through an image acquired by a camera. Specifically, the camera of the first electronic device can acquire images in real time and perform facial feature detection based on the acquired images. Within a certain period of time, the first electronic device first detects facial information in the acquired images and determines it as a state close to the user's body; then, no facial information is detected in the acquired images and it is determined to be a state far away from the user's body. When the above-mentioned change process occurs, it can be determined that the first electronic device gradually changes from being close to the user's body to being far away from the user's body.

[0024] In combination with the first aspect and the above-mentioned implementation manners, in some implementation manners of the first aspect, before detecting that the first axis of the first electronic device intersects with the second electronic device at the second position, or the first axis of the first electronic device and the second electronic device approximately intersect within a preset accuracy range, the method also includes: collecting the magnetic induction intensity during the movement by the magnetometer of the first electronic device; determining that the magnetic induction intensity collected by the first electronic device at the first position is greater than or equal to a fourth threshold, and the magnetic induction intensity collected at the second position is less than or equal to a fifth threshold, and the fourth threshold is greater than the fifth threshold.

[0025] Optionally, the first electronic device can detect the body of the user far away or close to the body of the user through a magnetometer. Since the human body itself is a magnetic field, when the first electronic device is far away from the human body or close to the human body, the magnetic field strength detected by the magnetometer will change significantly. Therefore, it can be determined that the first electronic device is far away from the user's body or close to the user's body based on the magnetic field strength detected by the magnetometer.

[0026] In combination with the first aspect and the above-mentioned implementations, in some implementations of the first aspect, before detecting that the first axis of the first electronic device intersects with the second electronic device at the second position, or that the first axis of the first electronic device and the second electronic device approximately intersect within a preset accuracy range, the method also includes: determining that the duration for which the first electronic device hovers at the second position is greater than or equal to a second preset duration.

[0027] Specifically, when the IMU of the first electronic device detects that the first electronic device stops accelerating and the hovering time of the first electronic device reaches the second preset time length t dwell When the user points to the second electronic device, it can be determined that the user is pointing to the second electronic device.

[0028] When a pointing operation is detected, the first electronic device is triggered to determine the location of the device pointed by the user through wireless positioning technology. This process can combine data collected by multiple sensors to more accurately determine whether the user has performed a pointing operation.

[0029] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold, the method also includes: displaying a first prompt message, the first prompt message being used to guide the user to continue accelerating along the direction of the first axis, the first prompt message including one or more of text, icon, sound and vibration.

[0030] It should be understood that, because the user's hand-eye separation occurs when pointing to the second electronic device, the eyes cannot see the interface of the first electronic device. Therefore, the embodiment of the present application does not limit the prompt method of any of the above prompt information (the aforementioned first prompt information, second prompt information, third prompt information, fourth prompt information, etc.), for example, it can be a visual or non-visual prompt that is recognizable to the user, such as different interface prompts, vibrations, indicator lights, voices, and other prompt methods.

[0031] In one possible implementation, when a user uses a first electronic device to point at a second electronic device, a prompt message may be displayed on the first electronic device. The prompt message is used to guide the user to perform a correct pointing operation to implement the control of the second electronic device through a pointing operation provided in an embodiment of the present application.

[0032] Optionally, the prompt information can be displayed through a window displayed on the first electronic device, or displayed in the form of an icon on the first electronic device, or in other visual guidance methods such as arrows, to guide the user to perform the correct pointing operation, so as to achieve the control of the second electronic device by pointing operation provided in the embodiment of the present application. Optionally, the display position of the arrow or icon can change according to the different positions of the user.

[0033] In combination with the first aspect and the above-mentioned implementation methods, in certain implementation methods of the first aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold and the duration is greater than or equal to the first preset duration, the method also includes: displaying a second prompt message, the second prompt message being used to prompt the user that the acceleration duration along the direction of the first axis reaches the first preset duration, the second prompt message including one or more of text, icon, sound and vibration.

[0034] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, when the duration of the hovering of the first electronic device at the second position is greater than or equal to a second preset duration, the method also includes: displaying a third prompt information, the third prompt information being used to indicate that the first electronic device detects whether the first axis intersects with the second electronic device at the second position, the third prompt information including one or more of text, icon, sound and vibration.

[0035] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, after detecting that at the second position, the first axis of the first electronic device intersects with the second electronic device, or the first axis of the first electronic device and the second electronic device approximately intersect within a preset accuracy range, the method also includes: displaying fourth prompt information, the fourth prompt information is used to indicate that the first electronic device is pointing to the second electronic device, and the fourth prompt information includes one or more of text, icon, sound and vibration.

[0036] In a possible implementation, in order to accurately identify the user's pointing operation and reduce the false touch rate, the embodiment of the present application can trigger the first electronic device to detect the user's pointing operation only under a preset scenario, and the preset scenario includes: the first electronic device is in a light screen state; and / or, the first electronic device is in an unlocked state; and / or, the first electronic device is displayed as the main interface; and / or, it is detected that the normal direction of the display screen perpendicular to the first electronic device has a certain angle with the ground. Among them, the discovery direction of the display screen of the first electronic device has a certain angle with the ground, which can be understood as the normal direction of the display screen is not perpendicular to the ground.

[0037] In another possible implementation, when the IMU of the first electronic device detects that the first electronic device stops accelerating and the hovering time of the first electronic device reaches a second preset time length t dwell When the user points to a second electronic device, a third prompt message may appear on the first electronic device, and the third prompt message prompts the user to point to the second electronic device. When at least two devices are detected in the direction pointed by the user, the third prompt message may display information of the at least two devices to the user. The user may select the second electronic device from them.

[0038] Alternatively, optionally, a fourth prompt message may also appear on the pointed device at this time, prompting the user to perform the correct pointing operation to point to the device. For example, if the user points to the smart screen through the first electronic device, since the user will be looking at the smart screen during the pointing process, a pop-up window may appear on the smart screen, and the pop-up window includes the fourth prompt message, indicating the correct pointing operation that the user needs to perform to control the smart screen.

[0039] In the above, by using visual and non-visual prompting methods, the pointing operation is guided and fed back on the first electronic device and the pointed second electronic device, thereby increasing the success rate of the pointing operation and improving the user experience.

[0040] Through the setting of the above preset scenarios, the first electronic device is limited to recognize whether the user has performed a pointing operation only in one preset scenario or in multiple scenarios at the same time. For example, the first electronic device recognizes whether the user has performed a pointing operation only in the screen-on state and / or unlocked state, or the first electronic device is limited to recognize whether the user has performed a pointing operation only when it is displayed as the main interface, or the first electronic device is limited to recognize whether the user has performed a pointing operation only when it is in the screen-on state and unlocked state and displays the main interface of the first electronic device. The setting of the preset scenarios can increase the accuracy of identifying the user's pointing operation and avoid detection in possible scenarios such as the user just handing the first electronic device to other people, thereby reducing the false touch rate and improving the user experience.

[0041] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, detecting that at the second position, the first axis of the first electronic device intersects with the second electronic device, or the first axis of the first electronic device approximately intersects with the second electronic device within a preset accuracy range, includes: according to a positioning device, detecting that at the second position, the first axis of the first electronic device intersects with the second electronic device, or the first axis of the first electronic device approximately intersects with the second electronic device within a preset accuracy range, wherein the positioning device is an independently installed positioning sensor, or the positioning device is a positioning chip installed on the first electronic device and the second electronic device, and the positioning chip includes any one of a Bluetooth positioning chip, an ultra-wideband UWB positioning chip, and a wireless fidelity WIFI positioning chip.

[0042] In a possible implementation, in a home scenario, at least three fixed positioning devices may be installed, and it is ensured that the three positioning devices are powered on and in working state, and the at least three positioning devices can communicate with the user's first electronic device.

[0043] Optionally, the positioning device may be a sensor with a positioning function or a structure with a positioning function, for example, the sensor may be a laser sensor, an infrared sensor, etc.; the structure with a positioning function may be a positioning chip. For example, the structure with a positioning function may be a Bluetooth positioning chip, an ultra-wideband UWB positioning chip, a wireless fidelity WIFI positioning chip, etc.

[0044] Optionally, in addition to pre-arranging the positioning devices listed above in the home scene, the positioning device can also be installed on smart TVs (smart screens), air conditioners, speakers, routers and other devices in the home scene. For example, if the smart screen, air conditioner, speaker and other devices themselves have UWB chips, there is no need to additionally arrange at least 3 fixed positioning devices in the home scene. The positioning function of any device in the home scene can be achieved through the interaction between the first electronic device and the smart screen, air conditioner, speaker and other devices. For the specific positioning algorithm, please refer to the process described later. It should be understood that the embodiments of the present application do not limit the number and implementation form of the positioning devices.

[0045] Through the above scheme, the first electronic device, as the first electronic device, can detect the second electronic device pointed to by the user through the three positioning devices arranged in the home scene, or the first electronic device detects the second electronic device pointed to by the user through the positioning chip, and further after detecting the user's quick gesture of pointing, the second electronic device can be controlled by the user's first electronic device.

[0046] In combination with the first aspect and the above-mentioned implementations, in some implementations of the first aspect, when the first axis of the first electronic device intersects or approximately intersects with one or more of the second electronic devices, the method also includes: displaying a second window, the second window including information of the one or more second electronic devices; in the second window, detecting a first operation; and in response to the first operation, the first electronic device displaying the first window.

[0047] Through the above process, for multiple second electronic devices in close proximity, the first electronic device may recognize multiple devices pointed to by the user, or when the user desires to control the multiple adjacent devices, multiple device cards may be displayed for the user, allowing the user to select at least one second electronic device according to their needs. This design is more user-friendly, can better meet the different needs of users, and improves user experience.

[0048] In combination with the first aspect and the above-mentioned implementations, in some implementations of the first aspect, the method also includes: in the first window, detecting a second operation; in response to the second operation, the first electronic device sends a control instruction to the second electronic device, and the control instruction is used to control the behavior of the second electronic device.

[0049] In one possible case, the second threshold is equal to the third threshold.

[0050] In a second aspect, a first electronic device is provided, comprising: a processor and a memory, wherein the memory stores one or more instructions, and when the one or more instructions are executed by the processor, the electronic device performs the following steps: obtaining acceleration in the direction of a first axis, acceleration on a second axis, and acceleration on a third axis, wherein the first axis is parallel to the long side of the first electronic device, the second axis is parallel to the short side of the first electronic device, and the third axis is perpendicular to the plane defined by the first axis and the second axis; determining that during a movement of a user holding the first electronic device from a first position as a starting position to a second position, the acceleration in the direction of the first axis is greater than or equal to a first threshold The value or the duration of the acceleration in the direction of the first axis being greater than or equal to the first threshold value is greater than or equal to the first preset time length, the acceleration in the direction of the second axis being less than or equal to the second threshold value, and the acceleration in the direction of the third axis being less than or equal to the third threshold value; detecting that at the second position, the first axis of the first electronic device intersects with the second electronic device, or the first axis of the first electronic device and the second electronic device approximately intersect within a preset accuracy range; receiving the account information and device information of the second electronic device sent by the second electronic device for logging in; displaying a first window according to the account information and device information of the second electronic device for logging in, the first window displaying an interface for controlling the second electronic device.

[0051] In combination with the second aspect, in certain implementations of the second aspect, when the one or more instructions are executed by the processor, the electronic device performs the following steps: based on one or more of an acceleration sensor, an inertial sensing unit IMU, and a gyroscope, obtain the acceleration in the direction of the first axis, the acceleration on the second axis, and the acceleration on the third axis.

[0052] In combination with the second aspect and the above-mentioned implementation methods, in certain implementation methods of the second aspect, when the one or more instructions are executed by the processor, the electronic device performs the following steps: capturing images during movement through the camera of the first electronic device; determining that the image captured at the first position includes the user's facial feature information, and the image captured at the second position does not include the user's facial feature information.

[0053] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, when the one or more instructions are executed by the processor, the electronic device performs the following steps: collecting the magnetic induction intensity during the movement through the magnetometer of the first electronic device; determining that the magnetic induction intensity collected at the first position is greater than or equal to a fourth threshold, the magnetic induction intensity collected at the second position is less than or equal to a fifth threshold, and the fourth threshold is greater than the fifth threshold.

[0054] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, when the one or more instructions are executed by the processor, the electronic device performs the following steps: determining that the duration of the hovering of the first electronic device at the second position is greater than or equal to a second preset duration.

[0055] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold, when the one or more instructions are executed by the processor, the electronic device is also used to perform the following steps: display a first prompt message, the first prompt message is used to guide the user to continue accelerating along the direction of the first axis, and the first prompt message includes one or more of text, icon, sound and vibration.

[0056] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold and the duration is greater than or equal to the first preset duration, when the one or more instructions are executed by the processor, the electronic device is also used to perform the following steps: display a second prompt message, the second prompt message is used to prompt the user that the acceleration duration along the direction of the first axis reaches the first preset duration, and the second prompt message includes one or more of text, icon, sound and vibration.

[0057] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, when the duration of the hovering of the first electronic device at the second position is greater than or equal to the second preset duration, when the one or more instructions are executed by the processor, the electronic device is also used to perform the following steps: display a third prompt information, the third prompt information is used to instruct the first electronic device to detect whether the first axis intersects with the second electronic device at the second position, and the third prompt information includes one or more of text, icon, sound and vibration.

[0058] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, when the one or more instructions are executed by the processor, the electronic device performs the following steps: displaying a fourth prompt information, the fourth prompt information being used to indicate that the first electronic device is pointing to the second electronic device, the fourth prompt information including one or more of text, icon, sound and vibration.

[0059] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, when the one or more instructions are executed by the processor, the electronic device performs the following steps: according to the positioning device, detect that at the second position, the first axis of the first electronic device and the second electronic device intersect, or the first axis of the first electronic device and the second electronic device approximately intersect within a preset accuracy range, wherein the positioning device is an independently installed positioning sensor, or the positioning device is a positioning chip installed on the first electronic device and the second electronic device, and the positioning chip includes any one of a Bluetooth positioning chip, an ultra-wideband UWB positioning chip, and a wireless fidelity WIFI positioning chip.

[0060] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, when the first axis of the first electronic device intersects or approximately intersects with one or more of the second electronic devices, when the one or more instructions are executed by the processor, the electronic device is also used to perform the following steps: displaying a second window, the second window including information of the one or more second electronic devices; in the second window, detecting a first operation; in response to the first operation, the first electronic device displays the first window.

[0061] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, when the one or more instructions are executed by the processor, the electronic device performs the following steps: in the first window, detecting a second operation; in response to the second operation, the first electronic device sends a control instruction to the second electronic device, and the control instruction is used to control the behavior of the second electronic device.

[0062] In combination with the second aspect and the above implementations, in some implementations of the second aspect, the second threshold is equal to the third threshold.

[0063] In a third aspect, the present application provides a device, which is included in an electronic device, and the device has the function of implementing the above aspects and the behavior of the electronic device in the possible implementation methods of the above aspects. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a display module or unit, a detection module or unit, a processing module or unit, etc.

[0064] In a fourth aspect, the present application provides an electronic device, comprising: a touch display screen, wherein the touch display screen comprises a touch-sensitive surface and a display; a positioning chip; one or more cameras; one or more processors; one or more memories; multiple applications; and one or more computer programs. Among them, the one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by one or more processors, the electronic device performs the interaction method between devices based on pointing operations in any possible implementation of any of the above aspects.

[0065] In a fifth aspect, the present application provides an electronic device, comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the interaction method between devices based on pointing operations in any possible implementation of any of the above aspects.

[0066] In a sixth aspect, the present application provides a computer storage medium, including computer instructions, which, when executed on an electronic device, enables the electronic device to execute any possible shortcut function startup method in any of the above aspects.

[0067] In a seventh aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute any one of the above-mentioned methods for interaction between devices based on pointing operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0069] Figure 2 This is a software structure block diagram of an electronic device according to an embodiment of the present application.

[0070] Figure 3 This is a schematic diagram of a graphical user interface for interconnecting devices in an embodiment of the present application.

[0071] Figure 4 This is another example of a graphical user interface schematic diagram of device interconnection in an embodiment of the present application.

[0072] Figure 5 This is a schematic diagram of a graphical user interface for enabling a quick interaction function between devices provided in an embodiment of the present application.

[0073] Figure 6 This is a schematic diagram of an installation of a positioning device provided in an embodiment of the present application.

[0074] Figure 7 This is a schematic diagram of an example of establishing a connection between a handheld device and a positioning device provided in an embodiment of the present application.

[0075] Figure 8 This is a schematic diagram of an example of determining the location of a device in a home scenario provided in an embodiment of the present application.

[0076] Fig. 9 This is a schematic diagram of an example of setting up devices in a home scene provided in an embodiment of the present application.

[0077] Fig.10 This is a schematic diagram of an example of a user pointing to a speaker provided in an embodiment of the present application.

[0078] Fig.11 This is a schematic diagram of an example of a user pointing to a speaker provided in an embodiment of the present application.

[0079] Fig.12 This is a schematic diagram of an example of a user controlling a speaker through a handheld device provided in an embodiment of the present application.

[0080] Fig.13 This is a schematic diagram of an example of a user pointing to a smart screen provided in an embodiment of the present application.

[0081] Fig.14 This is a schematic diagram of an example of a user controlling a smart screen through a handheld device provided in an embodiment of the present application.

[0082] Fig.15 This is a schematic diagram of an example of a user pointing at an air conditioner provided in an embodiment of the present application.

[0083] Fig.16 This is a schematic diagram of an example of a user controlling an air conditioner through a handheld device provided in an embodiment of the present application.

[0084] Fig.17 This is another schematic diagram of a user controlling a speaker through a handheld device provided in an embodiment of the present application.

[0085] Fig.18 This is another schematic diagram of a user controlling a speaker through a handheld device provided in an embodiment of the present application.

[0086] Fig.19 This is a schematic flowchart of an example of an interaction method between pointing-based devices provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0088] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0089] The method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0090] For example, Figure 11 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0091] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0092] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0093] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0094] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0095] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0096] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0097] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0098] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0099] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0100] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0101] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0102] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0103] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0104] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0105] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0106] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0107] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0108] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0109] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0110] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0111] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0112] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0113] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0114] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0115] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0116] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0117] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0118] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0119] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0120] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0121] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0122] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0123] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0124] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0125] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0126] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0127] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0128] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0129] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0130] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0131] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0132] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0133] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0134] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0135] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0136] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0137] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0138] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0139] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0140] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0141] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0142] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0143] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0144] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0145] Figure 2 1 is a software structure diagram of an electronic device 100 in an embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0146] like Figure 2 As shown, the application package may include applications such as camera, music, settings, Bluetooth, search, and smart life.

[0147] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0148] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0149] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0150] The view system includes visual controls, such as controls for displaying text and controls for displaying pictures. The view system can be used to build applications. The display interface can be composed of one or more views. For example, the display of the camera application icon on the mobile phone interface can include a view for displaying text and a view for displaying pictures.

[0151] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0152] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0153] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0154] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0155] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0156] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0157] The system library can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D layers and 3D layers for multiple applications. The media library supports playback and recording of multiple commonly used audio and video formats, as well as static image files, etc. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing. The image processing library can provide analysis for various image data and provide a variety of image processing algorithms, such as image cutting, image fusion, image blurring, image sharpening, etc., which will not be repeated here.

[0158] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0159] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 1 and Figure 2 Taking the mobile phone with the shown structure as an example, the interaction method between pointing-based devices provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0160] Figure 3 This is a diagram of a graphical user interface (GUI) for interconnected devices. Users can Figure 3 The process shown is to discover other electronic devices and establish a connection with the target electronic device. Figure 3 Figure (a) shows the main interface 301 currently displayed on the mobile phone in unlock mode, and the main interface 301 displays multiple applications (applications, Apps), such as music, camera, settings, and smart life applications. It should be understood that the main interface 301 can also include other more applications, which is not limited in the embodiment of the present application.

[0161] like Figure 3 As shown in FIG. 1 (a), the user can perform a sliding operation from the bottom of the mobile phone upward on the main interface 301, such as the sliding track shown by the black arrow. In response to the user's sliding operation, the mobile phone enters the following Figure 3The control center interface 302 shown in FIG. (b) of FIG. The control center interface 302 may include different control menus or control options, such as "Airplay mirroring", "Airdrop reception off", "wireless local area networks (WLAN)", "mute", "auto-rotate", "camera" and other different control menus or control options, which are not limited in the embodiments of the present application.

[0162] For example, the user can execute Figure 3 As shown in FIG. 2 (b), click the "Airplay Mirroring" option on the control center interface 302. In response to the user's click operation, the mobile phone enters the following Figure 3 The Airplay mirroring details interface 303 shown in (c) of FIG. 303. On the Airplay mirroring details interface 303, a list of devices discovered by the mobile phone can be displayed. It should be understood that the device list may include electronic devices that have previously established a connection with the mobile phone, or include electronic devices registered based on the same user account, or include electronic devices connected to the same network, or include electronic devices discovered by the current device scan, etc., and the present application embodiment does not limit this.

[0163] If the user currently wants to connect the mobile phone to and control the smart TV in the home scene, the user can click "userA's TV" and the mobile phone enters the control center interface 403 of the smart TV. The control center interface 403 may include various menus or buttons for controlling the smart TV, such as an exit connection switch, a play progress bar, a pause button, a selection button, etc. The user can control the smart TV in the home scene through the menus or buttons of the control center interface 403.

[0164] The above method performs device discovery and selection of multiple devices on the mobile phone, and then displays the devices that can be controlled by the mobile phone in a list. The user can select a device in the list and perform corresponding control as needed. In this process, when the number of selectable devices in the list exceeds a certain number, the user can identify the device according to the device icon, device name, etc., and select the target device from multiple devices in combination with scrolling operations. In one possible situation, the arrangement order of the devices in the device list is related to the signal strength of the device. When the signal is unstable or the user's location changes, the arrangement order of the devices in the device list may change. The user needs to slide multiple times to find the target device, which increases the user's operating cost; in addition, since the mobile phone system cannot perceive the target device that the user wants to select, it can only display all scanned devices to the user, which also increases the user's operating cost and the time to select the target device; further, when the user selects the target device from the device list, it is easy to make misoperations such as accidental touches.

[0165] Figure 4 This is another example of a graphical user interface diagram of device interconnection. Users can also use Figure 4 The process shown is to discover other electronic devices and establish a connection with the target electronic device. Figure 4 Figure (a) shows the main interface 401 currently displayed on the mobile phone in unlock mode, and the main interface 401 displays multiple applications (applications, Apps), such as music, camera, settings and search applications.

[0166] like Figure 4 As shown in Figure (a), the user clicks to find an application. In response to the user's click, the mobile phone enters the following Figure 4 In the search interface 402 shown in FIG. (b) of FIG. 402, the locations of multiple devices can be displayed in a map manner, for example, Figure 4 As shown in Figure (b), the mobile phone can find device 1, device 2, device 3, device 4 and device 5 and display the location label of each device to mark the location of each device. The user can click the location label of the target device to establish a connection with the target device, or display the control center interface of the target device to further control the target device.

[0167] In the process of displaying multiple devices through the map of the search application, the map needs to occupy most of the area of ​​the mobile phone screen, so the display area of ​​the device list becomes smaller in comparison, and fewer electronic devices can be displayed. In addition, in the map, when the distance between devices is close, the positions of multiple devices will overlap, and the user may accidentally touch other devices when clicking on the target device; or the process requires the user to perform complex interactive operations or auxiliary operations such as zooming, dragging, etc. to select the target device, which increases the user's operating cost and the time to select the target device.

[0168] In another possible implementation, users can also use near field communication (NFC) technology to touch two devices where NFC patches are attached to achieve device discovery and selection. In this process, users need to bring their mobile phones close to the target device and discover and select between devices by physically touching a specific area. However, not all devices support the NFC function, and the mobile phone and the target device must display NFC patches on the appearance of the devices, which affects the appearance of the devices to a certain extent.

[0169] In summary, the existing device discovery, device connection or device control process requires users to perform complex interactive operations, which increases the user's operating costs and the time to select the target device. In view of this, the embodiment of the present application will provide a quick device interaction method to achieve device discovery, connection or device control while reducing user operations.

[0170] It should be understood that in the description of the following embodiments, the mobile phone will be used as the user's handheld device, or referred to as the "control device", to introduce how to control other electronic devices in the home scene through the mobile phone, or connect other electronic devices, and other electronic devices can be referred to as "controlled devices". Figure 1 and Figure 2 The mobile phones, tablets and other devices of the structure shown, the controlled devices may include smart TVs (smart screens), air conditioners, speakers, personal computers, routers, etc., and examples are not given here one by one. The following will take the smart TV (smart screen), air conditioner, and speaker in the home scene controlled by a mobile phone as an example to describe in detail how to use shortcut gestures to achieve the control process between devices in the embodiment of the present application.

[0171] Figure 5 This is a schematic diagram of a graphical user interface for enabling a quick interaction function between devices provided in an embodiment of the present application. Figure 5 FIG. 10 (a) shows a main interface 501 currently displayed on the mobile phone in unlock mode. The user clicks the icon of the setting application on the main interface 501. In response to the user's click operation, the mobile phone displays the following Figure 5 The main interface 502 of the setting application is shown in FIG. (b) in FIG. Figure 5 As shown in Figure (b), the main interface 502 of the setting application includes multiple menus for setting mobile phone network connection, Bluetooth connection, desktop and wallpaper, display and brightness, sound and intelligent assistance functions.

[0172] The user clicks on the Smart Assistance and Gesture Control menu. In response to the user’s click, the phone displays the following Figure 5 The details interface 503 of intelligent assistance and gesture control shown in Figure (c) of FIG. On the interface 503, a variety of submenus for realizing intelligent control of the mobile phone may be further included, such as travel assistant, one-handed operation, gesture control, smart screen recognition, leather case operation, stylus operation and other submenus. The interface 503 may also include switches with different functions such as anti-mistouch mode. It should be understood that the embodiment of the present application does not limit the type and number of menus or switches displayed on the interface. The function of each menu or switch can refer to the implementation of the prior art, which will not be repeated here.

[0173] In a possible implementation, the embodiment of the present application can add a "point-and-click control function" switch in the intelligent assistance and gesture control menu, and the user can perform the following operations: Figure 5 As shown in the operation of Figure (c), click the "one-finger control function" switch to turn on the quick interaction function between devices to be introduced in the embodiment of the present application, that is, to control the interaction between devices through the quick mobile phone.

[0174] It should be understood that the above introduction introduces turning on the “point-and-control function” switch in the settings application, and the “point-and-control function” switch can also be added in other applications, for example, the “point-and-control function” switch can be added in the smart life application. Users can turn on the quick interaction function between devices through the “point-and-control function” switch in smart life, and the embodiment itself does not limit this.

[0175] Alternatively, in addition to enabling the quick interaction function between devices in the settings app or smart life app installed on the phone, you can also add a "one-finger control function" switch in the notification bar of the phone. For example, the user can pull down from the top of the phone on any interface of the phone. In response to the user's pull-down operation, the phone displays the following Figure 3 The notification interface 302 is shown in Figure (b) and a "one-finger control function" switch is added to the notification interface 302. The user can click the "one-finger control function" switch to enable the quick interaction function between devices. The embodiment of the present application does not limit the method of enabling the quick interaction function between devices.

[0176] pass Figure 5 According to the method introduced, the user turns on the function of realizing interaction between devices through quick gestures provided in the embodiment of the present application. Before introducing the quick gestures in detail, the mobile phone, as the user's handheld device, also needs to realize the positioning of smart TVs (smart screens), air conditioners, speakers, routers and other devices in the home scene, that is, the mobile phone needs to determine the position of each device, and then control it more accurately through quick gestures.

[0177] In a possible implementation, at least three fixed positioning devices are installed in a home scene, and it is ensured that the three positioning devices are powered on and in working state, and the at least three positioning devices can communicate with the user's handheld device.

[0178] Optionally, the positioning device may be a sensor with a positioning function or a structure with a positioning function, for example, the sensor may be a laser sensor, an infrared sensor, etc.; the structure with a positioning function may be a chip. For example, the structure with a positioning function may be a positioning structure based on a Bluetooth module, a positioning chip based on ultra wide-band (UWB) wireless sensing capability, a positioning structure based on a global positioning system (GPS), a positioning structure based on a wireless fidelity (WIFI) module, etc.

[0179] Optionally, in addition to pre-arranging the positioning devices listed above in the home scene, the positioning device can also be installed on smart TVs (smart screens), air conditioners, speakers, routers and other devices in the home scene. Exemplarily, if the smart screen, air conditioner, speaker and other devices themselves have UWB chips, there is no need to additionally arrange at least 3 fixed positioning devices in the home scene. The positioning function of any device in the home scene can be realized through the interaction between the mobile phone and the smart screen, air conditioner, speaker and other devices. For the specific positioning algorithm, please refer to the process described later. It should be understood that the embodiments of the present application do not limit the number and implementation form of the positioning devices.

[0180] Figure 6 This is a schematic diagram of an example of positioning device installation provided by an embodiment of the present application. Specifically, taking the installation of three fixed positioning devices in a home scene as an example, the installation rules of the positioning devices are introduced. The specific installation rules may include:

[0181] (1) Barrier-free communication rules.

[0182] Specifically, during the installation of the positioning device, ensure that each positioning device can communicate with any device in the home scene, so that the mobile phone, as a handheld device, can communicate with each positioning device, and then the mobile phone can accurately locate the position of each device in the home scene.

[0183] For example, Figure 6 Figure (a) shows an incorrect installation method. There is an obstacle between the positioning device 1 and the device A, which may cause the communication between the positioning device 1 and the device A to be blocked, and thus the device A in the home scene cannot be located by the three positioning devices. Figure 6 As shown in Figure (b), it is ensured that any device B in the home scene can communicate with positioning device 1, positioning device 2, and positioning device 3 respectively, and there are no obstacles between device B and any positioning device.

[0184] (2) Installation height rules for positioning devices.

[0185] Specifically, during the installation of the positioning devices, ensure that the height of each positioning device from the ground plane is greater than or equal to 2 meters, or the minimum installation height H-min. The minimum installation height H-min can ensure that the three positioning devices have a larger coverage range and can locate any device in the home scene.

[0186] For example, Figure 6 Figure (c) shows an incorrect installation method. The height H2 between positioning device 2 and the ground, and the height H3 between positioning device 3 and the ground are less than 2 meters, which may result in a small coverage range of the three positioning devices, and it is impossible to locate any device in the home scene through the three positioning devices. Figure 6 As shown in Figure (d), ensure that the heights of positioning device 1, positioning device 2, and positioning device 3 above the plane where the ground is located are greater than H-min, that is, positioning device 1, positioning device 2, and positioning device 3 are installed in the shaded area greater than H-min from the plane where the ground is located.

[0187] (3) Large coverage rule of positioning device.

[0188] Specifically, during the installation of the positioning devices, ensure that the combined coverage of the three positioning devices can be maximized, that is, to cover as many smart screens, air conditioners, speakers and other devices in the entire home scene as possible.

[0189] For example, Figure 6 Figure (e) shows an incorrect installation method. The triangular area formed by the connection of positioning device 1, positioning device 2, and positioning device 3 is small, which may result in a small coverage range of the three positioning devices, and it is impossible to locate any device in the home scene through the three positioning devices. Figure 6 As shown in Figure (f), the area of ​​the triangular region formed after the positioning device 1, the positioning device 2, and the positioning device 3 are connected is ensured to be maximized.

[0190] (4) Vertical installation rules for positioning devices.

[0191] Specifically, during the installation of the positioning device, ensure that the three positioning devices are installed on a vertical wall, and avoid installing the positioning device on a wall that has an inclined angle with respect to the plane where the ground is located.

[0192] For example, Figure 6Figure (g) shows an incorrect installation method, where the positioning device 1 is installed on a tilted wall, which may result in a small coverage area of ​​the three positioning devices, and any device in the home scene cannot be located by the three positioning devices. Figure 6 As shown in Figure (h), ensure that positioning device 1, positioning device 2, and positioning device 3 are all installed on the vertical wall.

[0193] (5) Uniform installation rules for positioning devices.

[0194] Specifically, during the installation of the positioning devices, ensure that the three positioning devices are installed on different walls to avoid installing two or more positioning devices on the same wall.

[0195] For example, Figure 6 Figure (i) shows an incorrect installation method, where positioning device 1 and positioning device 2 are installed on the same wall, which may result in a small coverage area of ​​the three positioning devices, and any device in the home scene cannot be located by the three positioning devices. Figure 6 As shown in Figure (h), ensure that positioning device 1, positioning device 2, and positioning device 3 are all installed on different walls.

[0196] (6) There are no obstacles (walls or objects) between the positioning devices.

[0197] Specifically, the barrier-free communication rule introduced in rule (1) ensures that there are no obstacles between any device in the home scene and each positioning device, and communication can be carried out. In addition, during the installation of the positioning device, it should also be ensured that there are no obstacles between the three positioning devices, and there are no obstacles in the direction perpendicular to the plane where the ground is located and in the direction parallel to the plane where the ground is located, further ensuring the largest possible coverage of the three positioning devices, and then any device in the home scene can be located by the three positioning devices.

[0198] For example, Figure 6 Figure (j) shows an incorrect installation method, where there is an obstacle between the positioning device 1 and the positioning device 3, and it should be ensured that there is no obstacle between the three.

[0199] Six rules for installing positioning devices are introduced above. It should be understood that the embodiment of the present application may include more or fewer rules while ensuring that three positioning devices can locate any device in a home scene. The embodiment of the present application is not limited to this.

[0200] After installing the three positioning devices, it is also necessary to establish a connection between the handheld device and the three positioning devices so that the handheld device can interact with the three positioning devices to further determine the location of each device in the home scene and the distance between the handheld device and each device.

[0201] In a possible implementation, a connection can be established between the handheld device and the three positioning devices through a smart life application. Figure 7 This is a schematic diagram of an example of establishing a connection between a handheld device and a positioning device provided in an embodiment of the present application. Figure 7 Figure (a) shows the main interface 701 currently displayed on the mobile phone in unlock mode. The user clicks the icon of the smart life application on the main interface 701. In response to the user's click operation, the mobile phone displays the following Figure 7 The main interface 702 of the smart life application shown in FIG. (b) is as follows. Figure 7 As shown in Figure (b), the main interface 702 of the smart life application may include a virtual experience menu and an add device menu, as well as functional areas including "Home", "Mall", "Smart" and "My".

[0202] like Figure 7 As shown in Figure (a) of FIG. 1 , the user clicks the “Add Device” menu on the main interface 702 of the smart life application. In response to the user’s click operation, the mobile phone displays the following Figure 7 The automatic scanning interface 703 shown in FIG. (c) in FIG. Specifically, the mobile phone can automatically scan the surrounding devices or components that can be connected. It should be understood that during the process, it must be ensured that the three positioning components installed in the home scene are powered on and can be searched and found by the mobile phone.

[0203] Optionally, the user can also discover the three positioning devices through methods such as "manual addition" or "scan code to add" on the main interface 702 of the smart life application. The embodiment of the present application does not limit the method of discovering the positioning device by the mobile phone, which will not be repeated here.

[0204] When the mobile phone scans 3 positioning devices, it can display the following Figure 7 The interface 704 is shown in (d) of FIG. On the interface 704, the names of three positioning devices, such as Tag 1, Tag 2 and Tag 3, can be displayed. The user clicks on the positioning device Tag 1, and in response to the user's click operation, the Tag 1 can be directly connected to the mobile phone. Or, optionally, the mobile phone can display the network configuration interface 705 shown in (e) of FIG. 7. In the network configuration interface 705, the user can enter the current network name and password, and click the "Next" button, and the mobile phone jumps to the connection success interface 706 shown in (f) of FIG. 7.

[0205] Through the above steps, the connection between the mobile phone and the positioning device 1 is achieved. Similarly, repeat Figure 7 The processes shown in (d), (e) and (f) can realize the connection between the mobile phone and the positioning device 2 and the positioning device 3. Alternatively, the mobile phone and the positioning device can also be connected through shortcuts such as "touch", which will not be described in detail in the embodiment of the present application.

[0206] After three positioning devices are deployed in a home scene, the handheld device can determine the position of each device in the home scene and the distance between the handheld device and each device based on the three positioning devices.

[0207] Figure 8 This is a schematic diagram of an example of determining the position of a device in a home scene provided by an embodiment of the present application. Among them, a spatial coordinate system is established in the space where the room is located, for example, the spatial coordinate system is established with the length, width and height of the room as the coordinate axes. The embodiment of the present application does not limit the method of establishing the spatial coordinate system. It should be understood that as long as the same spatial coordinate system is used when calculating the position of each device in the home scene through three positioning devices.

[0208] The spatial coordinates (x1, y1, z1) of positioning device 1, the spatial coordinates (x2, y2, z2) of positioning device 2, and the spatial coordinates (x3, y3, z3) of positioning device 3 are known, and it is known that the distance d1 between device A and positioning device 1, the distance d2 between device A and positioning device 2, and the distance d3 between device A and positioning device 3 can be measured. After obtaining the above known parameters, assume the spatial coordinates (x0, y0, z0) of any device A in the home scene, and determine the values ​​of x0, y0, z0 according to the following calculation process.

[0209] (x1-x0) 2 +(y1-y0) 2 +(z1-z0) 2 =d1 2

[0210] (x2-x0) 2 +(y2-y0) 2 +(z2-z0) 2 =d2 2

[0211] (x3-x0) 2 +(y3-y0) 2 +(z3-z0) 2 =d3 2

[0212] After obtaining d1, d2, and d3, draw a circle with the position of positioning device 1 as the center and d1 as the radius, draw a circle with the position of positioning device 2 as the center and d2 as the radius, and draw a circle with the position of positioning device 3 as the center and d3 as the radius. The position coordinates of the intersection of the three circles are the position coordinates of device A, that is, the position of any device in the home scene can be determined by the three positioning devices 3. As a handheld device, the mobile phone can obtain the above known parameters and determine the position of any device according to the above calculation process.

[0213] In summary, as a handheld device, the mobile phone can already communicate with the three positioning devices deployed in the home scene. To achieve further quick gestures based on the user's finger, the user's handheld device can control other devices in the home scene. It is also necessary to give other devices in the home scene the ability to be controlled. Fig. 9 An implementation method is introduced, which can enable other devices in a home scene to be controlled by a user's handheld device after detecting a quick gesture of the user pointing.

[0214] Fig. 9 1 is a schematic diagram of an example of setting devices in a home scene provided by an embodiment of the present application. Fig. 9 Figure (a) shows the main interface 901 currently displayed on the mobile phone in the unlock mode. The user clicks the icon of the smart life application on the main interface 901. In response to the user's click operation, the mobile phone displays the following Fig. 9 The main interface 902 of the smart life application is shown in Figure (b).

[0215] Optionally, the main interface 902 of the smart life application has added the infrared remote control, my speaker, Huawei router, my air conditioner and Huawei Smart Screen in the home scene. In other words, the main interface 902 of the smart life application can display multiple cards of devices that have been paired with the handheld device.

[0216] It should be understood that the method of adding various electronic devices in home scenes in life applications can refer to the methods in the prior art, such as scanning to add, manual adding, touch to add and other possible methods, which will not be repeated here.

[0217] In one possible implementation, on the main interface 902 of the smart life application, the user selects the area where the card of the device in a home scene is located through a preset operation, and then points the handheld device to the device corresponding to the area where the card is located, thereby activating the device, and agreeing that the device can be controlled by the handheld device when a user's pointing gesture is detected.

[0218] For example, Fig. 9As shown in Figure (b), on the main interface 902 of the smart life application, the user long presses the area where the "My Speaker" card is located, and at the same time or within a certain period of time, the user uses the mobile phone to point to the location where the speaker is placed.

[0219] It should be understood that the long press operation can trigger the activation of the speaker, and then by pointing the mobile phone to the location where the speaker is placed, information interaction can be carried out between the mobile phone and the speaker, so that when the mobile phone detects the user's pointing gesture, the speaker can be controlled by the mobile phone.

[0220] It should also be understood that during this process, the user can interact with the mobile phone and the speaker by pointing the mobile phone to the location where the speaker is placed. It can also be understood that the mobile phone locates the speaker through the three positioning devices connected above and determines the specific location of the speaker in the home scene.

[0221] Optionally, the method of triggering the activation of the speaker is not limited to Fig. 9 The operation of long pressing the area where the "My Speaker" card is located and pointing the mobile phone to the position where the speaker is placed as shown in (b) of the figure can also be achieved through other preset operations or gestures such as swiping operations, as well as the operation of pointing the mobile phone to the position where the speaker is placed, which is not limited to the embodiments of the present application.

[0222] In another possible implementation, the user can select the area where the card of a device in a certain home scene is located, enter the detailed interface of the device, turn on the point-and-click control function switch of the device, and when the user's point-and-click gesture is detected, the device can be controlled by the handheld device. It should be understood that in this implementation, the user can also point the handheld device to the device corresponding to the area where the card is located, thereby activating the device. At this time, the mobile phone locates the device through the three positioning devices connected above, determines the specific location of the device in the home scene, and stipulates that when the handheld device detects the user's point-and-click gesture, the device can be controlled by the handheld device.

[0223] Optionally, the user can select the area where the card of the device in a certain home scene is located on the main interface 902 of the smart life application, or select the area where the card of the device in a certain home scene is located through a quick entry.

[0224] For example, Fig. 9 As shown in FIG. 1(c), on the main interface 903 of the mobile phone, the user performs a pull-down operation as indicated by the black arrow on the interface 903. In response to the pull-down operation of the user, the mobile phone displays Fig. 9 The control center interface 904 shown in Figure (d) can display my speaker and Huawei Smart Screen cards in the added home scene.

[0225] When the user clicks on the area where the Huawei Smart Screen card is located, the phone can enter Fig. 9 The details interface 905 of the Huawei Smart Screen shown in Figure (e) in FIG. A "one-finger control function" switch can be added to the details interface 905 of the Huawei Smart Screen. The user clicks to turn on the "one-finger control function" switch, so that the switch is in the on state, and the Huawei Smart Screen can be activated. At this time, the mobile phone locates the Huawei Smart Screen through the three positioning devices connected above, determines the specific position of the Huawei Smart Screen in the home scene, and stipulates that when the mobile phone detects the user's one-finger gesture, the Huawei Smart Screen can be controlled by the mobile phone.

[0226] In another possible implementation, after the user activates a device in a home scene through a handheld device, a prompt message may be displayed to the user, where the prompt message is used to indicate that the device has been successfully activated.

[0227] Optionally, when the mobile phone completes the positioning of a device in the home scene, the prompt information can be displayed in a pop-up window or other manner on a specific application or interface to inform the user that the device has been positioned and successfully activated.

[0228] Optionally, if during the process of activating a device by a mobile phone, it is found that the distance between the device and the mobile phone is very close, the user may be prompted that "the distance between the mobile phone and the located XX device is very close, it is recommended to move the mobile phone or XX device away", or multiple possible devices may be displayed, and the user may further select the currently located device in actual operation. This embodiment of the present application is not limited to this.

[0229] Similarly, by repeating the above operations, the mobile phone can locate and activate any other device such as the smart screen in the home scene. I will not go into details here.

[0230] Combination Fig. 9 A variety of possible methods are introduced in this article. This method can be used in a specific application or interface (such as smart life), or the control center interface of a handheld device, or the device details interface of a device, etc., through preset operations (long press, swipe, etc.) or clicking a preset switch, so that other devices that have been paired with the handheld device and are being pointed at by the user in the home scene with the handheld device enter an activated state, so as to stipulate that when the handheld device detects the user's pointing gesture, the device can be controlled by the handheld device.

[0231] In summary, through Figures 5 to 9 According to the introduction, at least three positioning devices have been deployed in the current home scene, and the user has turned on the function of pointing to control other devices on the mobile phone and activated multiple devices in the home scene. It is agreed that when the mobile phone detects the user's pointing gesture, the device pointed by the user can be controlled by the mobile phone. Figures 10 to 19 This chapter describes in detail the process of users controlling other devices in the home scene by pointing.

[0232] Fig.10 is a schematic diagram of an example of a user pointing to a speaker provided in an embodiment of the present application, wherein: Fig.10 The process from (a) to (c) in the figure shows a possible pointing operation process of the user.

[0233] For example, the user can Fig.10 The mobile phone is held in the posture shown in Figure (a) and the Fig.10 The posture shown in Figure (b) is finally as follows Fig.10 The posture shown in Figure (c) points to the speaker in the home scene. The mobile phone can detect that the user holds the mobile phone and undergoes a displacement change process as shown in the figure, and finally points to the location of the speaker.

[0234] In a possible implementation, a mobile phone as a handheld device of a user can identify the user's pointing operation by means of one or more sensors of the mobile phone or a camera of the mobile phone, etc. In the description of the subsequent embodiments, the user's pointing operation can be referred to as "pointing operation".

[0235] Optionally, the sensor of the mobile phone may include an accelerometer, a gyroscope, a magnetometer sensor (M-sensor), an inertial measurement unit (IMU), etc. The camera of the mobile phone is not limited to the front camera or the rear camera of the mobile phone; or the camera of the mobile phone is not limited to one or more of the main camera, the wide-angle camera, and the telephoto camera of the mobile phone. The pointing operation of the user is recognized according to the picture obtained by the camera, and this embodiment of the application does not limit this.

[0236] In the process of determining whether the user has directed the speaker, establish Fig.10 The coordinate system O-XYZ is shown in Figure (d) of FIG. Specifically, the coordinate system O-XYZ takes the center of gravity of the mobile phone as the coordinate origin O, the straight line passing through the coordinate origin O where the long frame of the mobile phone is located as the Y axis, the straight line where the short frame of the mobile phone is located as the X axis, and the straight line perpendicular to the ground as the Z axis. Then, when the user performs a pointing operation, the movement process of the coordinate system O-XYZ can be as follows: Fig.10 As shown in Figures (b) and (c).

[0237] Exemplarily, during the user's operation of pointing to the speaker, the following action change processes may be included:

[0238] (1) The user's starting position can be Fig.10The user's hand is naturally stretched out holding the mobile phone as shown in (a) of FIG. Fig.10 As shown in Figure (b), the user holds the mobile phone in his hand and places it naturally in front of his chest.

[0239] In other words, the user's pointing operation can be from Figure (a) directly to Figure (c), including different trajectories; or from Figure (b) directly to Figure (c); or via the trajectory of Figure (a)-Figure (b)-Figure (c), which is not limited in this embodiment of the present application.

[0240] (2) The user's arm extends forward, and the mobile phone accelerates along the straight line where the Y axis is located. The acceleration sensor of the mobile phone detects that the acceleration of the mobile phone along the straight line where the Y axis is located is greater than or equal to a first preset threshold value a. A The acceleration curve can be shown as Fig.10 As shown in Figure (e), at time T2, the acceleration of the mobile phone along the straight line where the Y axis is located is greater than or equal to the first preset threshold a A .

[0241] At the same time, it is detected that the acceleration of the mobile phone on the straight line where the X axis is located and the straight line where the Z axis is located is less than or equal to the second preset threshold a B .

[0242] Optionally, the acceleration of the mobile phone in different directions can be detected by the IMU of the mobile phone. Alternatively, the acceleration of the mobile phone in different directions can be detected based on the picture captured by the front camera of the mobile phone and the normally-on optical flow algorithm. Specifically, the front camera of the mobile phone is used to obtain adjacent frames in the picture, wherein the two-dimensional vector field of the feature points in the adjacent frames during the translation process is a velocity field that represents the three-dimensional motion of the object point through a two-dimensional image. That is, the image changes formed by the movement of the object point within a certain time interval can be reflected according to the velocity field of the three-dimensional motion of the object point, so as to determine the movement direction and movement rate of the object point on the image, and then determine whether the motion trajectory of the mobile phone is a straight line or a quasi-straight line.

[0243] In a possible implementation, when the handheld device detects the movement of the mobile phone along the straight line where the Y axis is located, and the acceleration value is greater than or equal to the first preset threshold value a A When Fig.10 As shown in Figure (e), at time T2, a first prompt message may appear on the handheld device to guide the user to continue accelerating along the Y-axis direction.

[0244] (3) The acceleration time of the acceleration of the mobile phone along the straight line where the Y axis is located detected by the IMU of the mobile phone reaches a first preset time length t a .

[0245] In a possible implementation, when the handheld device detects the movement of the mobile phone along the straight line where the Y axis is located, the acceleration value is greater than or equal to the first preset threshold value aA And the acceleration time reaches the first preset time t a When the wireless positioning function is enabled, a second prompt message may appear on the handheld device, prompting the user that the handheld device has enabled the wireless positioning function.

[0246] (4) Detect that the mobile phone is away from the user's body.

[0247] Optionally, the mobile phone can detect the body of the user far away or close to the user through a magnetometer. Since the human body itself is a magnetic field, when the mobile phone is far away from the human body or close to the human body, the magnetic field strength detected by the magnetometer will change significantly. Therefore, it can be determined that the mobile phone is far away from the user's body or close to the user's body based on the magnetic field strength detected by the magnetometer.

[0248] Alternatively, the mobile phone can detect the state of being away from the user's body or away from the user's body through the image acquired by the camera. Specifically, the camera of the mobile phone can acquire images in real time and perform facial feature detection based on the acquired images. Within a certain period of time, the mobile phone first detects facial information in the acquired images and determines it as a state close to the user's body; then, no facial information is detected in the acquired images and it is determined to be a state far from the user's body. When the above-mentioned change process occurs, it can be determined that the mobile phone gradually changes from being close to the user's body to being away from the user's body.

[0249] (5) When the IMU of the mobile phone detects that the mobile phone stops accelerating and the hovering time of the mobile phone reaches the second preset time t dwell .

[0250] When the mobile phone detects the above action change process (1)(2)(3)(4)(5), it can be determined that the current user has performed a pointing operation, thereby triggering the mobile phone to determine the location of the device pointed to by the user through wireless positioning technology. This process can be combined with data collected by multiple sensors to more accurately determine whether the user has performed a pointing operation. It should be understood that the mobile phone can also trigger the mobile phone to determine the location of the device pointed to by the user through wireless positioning technology when one or more of the above action change processes (1)(2)(3)(4)(5) are detected, and the embodiments of the present application are not limited to this.

[0251] In another possible implementation, in order to accurately identify the user's pointing operation and reduce the false touch rate, the embodiment of the present application can trigger the handheld device to detect the user's pointing operation only under preset scenarios, and the preset scenarios include: the handheld device is in the screen-on state; and / or, the handheld device is in the unlocked state; and / or, the handheld device displays the main interface; and / or, it is detected that the normal direction of the display screen perpendicular to the handheld device has a certain angle with the ground. Among them, the detection direction of the display screen of the handheld device has a certain angle with the ground, which can be understood as the normal direction of the display screen is not perpendicular to the ground.

[0252] In another possible implementation, when the IMU of the mobile phone detects that the mobile phone stops accelerating and the hovering time of the mobile phone reaches a second preset time length t dwell When the user points to a target device, a third prompt message may appear on the handheld device, and the third prompt message may prompt the user of the target device being pointed to. When at least two devices are detected in the direction pointed to by the user, the third prompt message may display information of the at least two devices to the user. The user may select a target device from them.

[0253] Alternatively, a fourth prompt message may also appear on the pointed device at this time, prompting the user to perform the correct pointing operation to point to the device. For example, if the user points to the smart screen through a mobile phone, since the user will be looking at the smart screen during the pointing process, a pop-up window may appear on the smart screen, and the pop-up window includes the fourth prompt message, indicating the correct pointing operation that the user needs to perform to control the smart screen.

[0254] In the above, by using visual and non-visual prompting methods, the pointing operation is guided and fed back on the handheld device and the pointed target device, thereby improving the success rate of the pointing operation and enhancing the user experience.

[0255] The above embodiments introduce possible situations in which prompt information is displayed on a handheld device or a target device. It should be understood that due to the phenomenon of hand-eye separation when the user points to the target device, the eyes cannot see the interface of the mobile phone. Therefore, the embodiments of the present application do not limit the prompt method of any of the above prompt information (first prompt information, second prompt information, third prompt information, fourth prompt information), for example, it can be a visual or non-visual prompt that is recognizable to the user, such as different interface prompts, vibrations, indicator lights, voice and other prompt methods.

[0256] Optionally, the prompt effect of the prompt information on the handheld device may be matched with the direction of the user's pointing action, the acceleration of the handheld device, and the like.

[0257] Through the setting of the above-mentioned preset scenarios, the mobile phone can be limited to one preset scenario or multiple scenarios are met at the same time to identify whether the user has performed a pointing operation. For example, the mobile phone can only identify whether the user has performed a pointing operation when the screen is on and / or unlocked, or it can be limited to identify whether the user has performed a pointing operation when the mobile phone displays the main interface, or it can be limited to identify whether the user has performed a pointing operation when the mobile phone is in the screen-on state and unlocked state and displays the main interface of the mobile phone. The setting of the preset scenarios can increase the accuracy of identifying the user's pointing operation and avoid detection in possible scenarios such as the user just handing the mobile phone to other people, thereby reducing the false touch rate and improving the user experience.

[0258] In another possible implementation, when the mobile phone determines that the user is currently performing a pointing operation on the speaker, the user may Fig.10 The operation shown in Figure (c) and the mobile phone hovering time reaches the second preset time length t dwell When the user presses a physical button, or performs a preset screen gesture, or an air gesture, etc., to send a signal to the mobile phone to determine the detection result of the current mobile phone detection.

[0259] For example, Fig.10 The operation shown in Figure (c) and the mobile phone hovering time reaches the second preset time length t dwell When the user presses the power button or the volume button, the mobile phone can receive the user's pressing operation and determine that the currently detected operation is the user's pointing operation. It should be understood that the embodiments of the present application do not limit the number of times the user presses the physical button, the type of physical button pressed, or the preset screen gesture or air gesture.

[0260] Through the above process, the accuracy of the mobile phone in identifying the user's pointing operation can be increased. This process can be understood as the user sending a confirmation message to the mobile phone through the above pressing operation or preset gesture to confirm that the action currently detected by the mobile phone is the user's pointing operation to the speaker, which can more accurately trigger the mobile phone's positioning process for the speaker. On the contrary, if the mobile phone detects that the user has performed the following Fig.10 The operation shown in Figure (c) and the mobile phone hovering time reaches the second preset time length t dwell When the user does not press the physical button or the preset button, the mobile phone does not receive the user's confirmation signal, and the positioning process of the speaker may not be triggered.

[0261] According to the above steps, after the mobile phone has recognized the user's current operation of pointing to the speaker, it further activates the wireless positioning function of the mobile phone, searches for peripheral devices of the mobile phone, and determines the location of the peripheral devices, as well as the distance between the peripheral devices and the mobile phone, to determine whether there is a device in the direction of the user's pointing. For example, Fig.10 As shown in Figure (c), when the mobile phone detects the user's pointing operation as shown in the figure, the mobile phone's infinite positioning technology can be triggered.

[0262] In a possible implementation, when the target device pointed to by the user is equipped with a positioning structure, the mobile phone can perform wireless positioning based on the two-way connection between itself and the target device. Exemplarily, the target device pointed to by the user can be equipped with a Bluetooth positioning chip, a UWB positioning chip, or a GPS positioning structure, a WIFI positioning structure, and one or more structures such as a laser sensor and an infrared sensor with positioning function, and the target device can be positioned based on the two-way connection between the target device and the mobile phone.

[0263] For example, if smart screens, air conditioners, speakers and other devices themselves have UWB chips, the positioning of any device in the home scene can be achieved through the interaction between mobile phones and smart screens, air conditioners, speakers and other devices.

[0264] It should be understood that both the mobile phone and the pointed target device have hardware capable of transmitting or receiving wireless positioning signals, and based on the transmission or reception of wireless positioning signals between the mobile phone and the pointed target device, the distance and angle between the mobile phone and the pointed target device are calculated and determined.

[0265] In another possible implementation, when three positioning devices are deployed in a home scene, the mobile phone can combine the three-point positioning capabilities of the three positioning devices to determine the distance and direction between the mobile phone and the target device it is pointing to.

[0266] For example, according to Figure 6 The method described in this article is to install three positioning devices in the home scene and then follow the Figure 7 The method shown establishes a connection between the handheld device and the three positioning devices so that the handheld device can interact with the three positioning devices, and further Figure 8 The three-point positioning method introduced determines the position of each device in the home scene and the distance between the handheld device and each device, etc., which will not be repeated here.

[0267] In summary, the mobile phone can determine the distance and angle between the mobile phone and the target device by triangulating between multiple devices. It should be understood that the mobile phone can transmit or receive wireless positioning signals, as long as the space of the home scene includes 3 or more fixed positioning devices that can transmit or receive wireless positioning signals, such as Fig.10 The mobile phone calculates the absolute position (x, y, z, ). That is, the position of each device in the home scene and the distance between the handheld device and each device can be determined by the three-point positioning method.

[0268] Optionally, when there are enough positioning devices capable of transmitting or receiving wireless positioning signals in a home scene, the six degrees of freedom (6DoF) positioning of the mobile phone can be determined, which will not be repeated here.

[0269] It should be understood that the positioning device can exist independently or be installed on other devices in a home scene. It is a component of smart screens, air conditioners, speakers and other devices. The embodiments of the present application are not limited to this.

[0270] Through the above process, the mobile phone has recognized the user's current operation of pointing to the speaker, and further activated the wireless positioning function of the mobile phone, and determined the position of the speaker pointed by the user. After responding to the above-mentioned user's pointing operation, the speaker pointed by the user can be controlled on the mobile phone.

[0271] Fig.11 is a schematic diagram of an example of a user pointing to a speaker provided in an embodiment of the present application, Fig.12 This is a schematic diagram of an example of a user controlling a speaker through a handheld device provided in an embodiment of the present application.

[0272] For example, Fig.11 The processes in (a), (b) and (c) show a possible pointing operation process of the user. Specifically, Fig.12 FIG. (a) in FIG. 1 shows the main interface 1201 of the mobile phone. When the mobile phone is in the bright screen state and displays the main interface of the mobile phone, if the user uses the mobile phone to perform the above-mentioned pointing to the speaker operation, the mobile phone can display the following Fig.12 The interface 1202 shown in FIG. (b) in FIG. 1202 , a speaker control window 20 is automatically popped up, and the speaker control window 20 can be suspended on the main interface of the mobile phone.

[0273] Optionally, the speaker control window 20 may include a window zoom option 21, a window close option 22, a speaker icon and name display area 23, and a song playback control area 24. The user can click the window zoom option 21 to enlarge the speaker control window 20 to full screen display or reduce it to a floating display. The song playback control area 24 can display buttons such as singers, singer photos, album pictures, song names, play switches, play progress bars, previous songs, and next songs. Users can click any button according to their needs to control the playback of the speaker. The song playback control area 24 also includes a sound projection button for controlling the playback of mobile phone music through the speaker, which will not be repeated here.

[0274] It should be understood that the speaker control window 20 may include the control options, menus, buttons or other functional areas listed above, or the speaker control window 20 may also include other display contents not listed above, such as an area for displaying lyrics, etc. The speaker control window 20 may display different styles and contents according to the type and screen size of the handheld device, etc., which will not be described in detail here.

[0275] Fig.13 is a schematic diagram of an example of a user pointing to a smart screen provided in an embodiment of the present application. Fig.14 This is a schematic diagram of an example of a user controlling a smart screen through a handheld device provided in an embodiment of the present application.

[0276] For example, Fig.13 The processes in (a), (b) and (c) show a possible pointing operation process of the user. Specifically, Fig.14 Figure (a) in the figure shows the main interface 1401 of the mobile phone. When the mobile phone is in the screen-on state and displays the main interface of the mobile phone, if the user uses the mobile phone to perform the above-mentioned pointing to the smart screen operation, the mobile phone can display the following Fig.14 The interface 1402 shown in (b) of FIG. In the interface 1402, the smart screen control window 30 is automatically popped up, and the smart screen control window 30 can be suspended on the main interface of the mobile phone.

[0277] Optionally, in this embodiment Fig.13 The trajectory of a user pointing to a smart screen can be different from Fig.11 The trajectory of the user pointing to the speaker may also be the same, which is not limited in the embodiments of the present application.

[0278] Optionally, the smart screen control window 30 may include a window zoom option 31, a window close option 32, a smart screen icon and name display area, and a smart screen playback control area 33. The user can enlarge the smart screen control window 30 to full screen display or reduce it to floating display by clicking the window zoom option 31. The smart screen playback control area 33 can display buttons such as the movie name, the selection list, the play switch, the playback progress bar, the previous episode, and the next episode. The user can click any button according to their needs to control the playback of the movie on the smart screen.

[0279] It should be understood that the smart screen control window 30 may include the control options, menus, buttons or other functional areas listed above, or the smart screen control window 30 may also include other display content not listed above, and the present application embodiment does not limit this. The smart screen control window 30 can display different styles and contents according to the type and screen size of the handheld device, etc., which will not be repeated here.

[0280] Fig.15 is a schematic diagram of an example of a user pointing to an air conditioner provided in an embodiment of the present application. Fig.16 This is a schematic diagram of an example of a user controlling an air conditioner through a handheld device provided in an embodiment of the present application.

[0281] For example, Fig.15 The processes in (a), (b) and (c) show a possible pointing operation process of the user. Specifically, Fig.16 Figure (a) shows the main interface 1601 of the mobile phone. When the mobile phone is in the bright screen state and displays the main interface of the mobile phone, if the user uses the mobile phone to perform the above-mentioned pointing to the air conditioner operation, the mobile phone can display the following Fig.16The interface 1602 shown in FIG. (b) in FIG. 1602 , the air conditioning control window 40 is automatically popped up, and the air conditioning control window 40 can be suspended on the main interface of the mobile phone.

[0282] Optionally, the air conditioning control window 40 may include a window zoom option 41, a window close option 42, an air conditioning icon and name display area 43, and an air conditioning control area 44. The user may enlarge the air conditioning control window 40 to display it in full screen or reduce it to a floating display by clicking the window zoom option 41. The air conditioning control area 44 may display whether the air conditioning is turned on, working modes such as cooling, indoor temperature, working power, temperature increase and decrease buttons, wind speed setting buttons, etc. The user may click any button according to his / her needs to control the turning on of the air conditioning, temperature setting, working mode setting, etc.

[0283] It should be understood that the air conditioning control window 40 may include the control options, menus, buttons or other functional areas listed above, or the air conditioning control window 40 may also include other display contents not listed above, which is not limited in the embodiment of the present application. The air conditioning control window 40 may display different styles and contents according to the type and screen size of the handheld device, which will not be described in detail here.

[0284] Through the above method, based on the recognition function and positioning function of the handheld device, when the handheld device detects the operation of the user pointing to the target device, a window for controlling the target device can be displayed on the handheld device, and the user can control the target device through the window. The method is simple to operate and reduces the steps for the user to operate the target device; in addition, the method selects the target device according to the user's pointing operation, which can help the user select a device that is farther away. The distance limit is reduced and the user experience is improved.

[0285] In another possible implementation, when the user uses a handheld device to point to a target device, a prompt message may be displayed on the handheld device, and the prompt message is used to guide the user to perform a correct pointing operation to implement the control of the target device through a pointing operation provided in an embodiment of the present application.

[0286] Optionally, the prompt information can be displayed through a window displayed on the handheld device, or displayed on the handheld device in the form of an icon, or in other visual guidance methods, to guide the user to perform the correct pointing operation, so as to achieve the control of the target device through pointing operation provided in the embodiment of the present application.

[0287] Fig.17 This is another schematic diagram of a user controlling a speaker through a handheld device provided in an embodiment of the present application.

[0288] For example, Fig.17The processes in (a), (b) and (c) show a possible pointing operation process of the user. Specifically, Fig.17 Figure (d) shows the main interface 1701 of the mobile phone. When the mobile phone is in the bright screen state and displays the main interface 1701 of the mobile phone, if the user uses the mobile phone to execute Fig.17 The operations shown in (a) and (b) above are as follows: the mobile phone does not detect Fig.17 As shown in Figure (c) of the figure, the operation of pointing to the speaker and hovering, etc., the main interface of the mobile phone can display the following Fig.17 The interface 1702 shown in (e) of FIG. In the interface 1702, the guide icon 50 is automatically displayed, and the guide icon 50 can be suspended on the main interface of the mobile phone in the form of an arrow.

[0289] When the user continues to execute the following steps according to the guide icon 50 Fig.17 When the pointing operation is performed as shown in FIG. 17 (c), the interface of the mobile phone may be displayed as the interface 1703 shown in FIG. 17 (f), and the icon 60 is displayed on the interface 1703 to indicate that the mobile phone is currently detecting the user's pointing operation again. Fig.17 When the user points to the speaker as shown in FIG. 7 (c), a speaker control window 20 may be displayed on the handheld device, which will not be described in detail here.

[0290] In another possible implementation, during the operation of the user using the handheld device to point to the target device, in addition to Fig.17 In addition to the guide icon shown in Figure (e), the user can be guided to perform the correct pointing operation through non-visual methods such as sound feedback and vibration feedback, so as to realize the control of the target device through pointing operation provided in the embodiment of the present application.

[0291] Optionally, the voice feedback may be issued by a handheld device, or by a target device being pointed at, for example, a mobile phone prompts the user in a voice message “please move right”.

[0292] Optionally, the vibration feedback may be emitted by a handheld device so as to be perceived by the user more quickly and sensitively. It should be understood that the embodiments of the present application are not limited to this.

[0293] Alternatively, after the user successfully points to the target device using the handheld device, feedback may be provided on the handheld device or the pointed target device, and the feedback is used to inform the user that the pointing operation is successful. The feedback information may include one or more of visual feedback, sound feedback, vibration feedback, etc. For example, Fig.17 As shown in Figure (f), when the user successfully points the mobile phone at the speaker, the mobile phone can vibrate to inform the user that the pointing operation is completed.

[0294] It should be understood that when the user successfully points to the target device using the handheld device, the target device can transmit the currently logged-in account information and device information to the handheld device, and then display the information on the handheld device. Fig.17 The target device control window shown in Figure (g) in the figure.

[0295] It should also be understood that when determining whether the handheld device is pointing to the target device, when the axis of the handheld device and the position of the target device are within a preset range, the target device pointed to by the user can be determined. Specifically, when the axis of the handheld device is identified to substantially intersect with the physical position of the target device with a certain predetermined accuracy, or when the handheld device is identified to be aligned with the target device, the target device pointed to by the user can be determined, that is, feedback can be provided on the handheld device or the pointed target device, and the login account information and device information of the target device can be transmitted to the handheld device, and the target device control window 20 or an interface related to the target device can be displayed on the handheld device.

[0296] In summary, the embodiment of the present application provides a method for realizing cross-device interaction based on pointing operation, in which a user can use a handheld device to point to a target device, and based on one or more sensors such as an accelerometer, a gyroscope, an IMU and / or a camera of the handheld device, detect the user's pointing operation, and trigger the wireless positioning function of the handheld device. The handheld device determines the distance between the handheld device and other devices, as well as the position and direction of other devices according to the wireless positioning function. When it is identified that the axis of the handheld device and the physical position of the target device are judged to be intersecting or substantially intersecting with a predetermined accuracy, at least one feedback such as visual feedback, sound feedback, and vibration can also be provided on the handheld device and / or the target device, and the login account information and device information of the target device are transmitted to the handheld device, and the control window of the target device is displayed on the handheld device, and the user can control the target device through the control window on the handheld device. When it is identified that the axis of the handheld device does not intersect with the physical position of the target device, at least one guidance method such as visual guidance, sound guidance, and vibration can also be provided on the handheld device and / or the target device to guide the user to perform the correct pointing operation, and further realize the display of the control window of the target device and the control function of the target device on the handheld device.

[0297] In another possible implementation, when the user performs the action of pointing at the target device, there may be two or more relatively close devices at the same position. In other words, the axis of the y-axis may be located between the two devices during the movement of the mobile phone, and the accuracy between the axis and the physical positions of the two devices is within a predetermined accuracy. When it is determined that the axis of the y-axis of the mobile phone intersects or substantially intersects with device 1 or device 2 at the same time, the pointing operation of the user's handheld device can be identified as pointing at device 1 or device 2. For this scenario, the embodiment of the present application also provides a method for providing a selection window for the user to select the target device in the process of detecting the user's pointing operation.

[0298] Fig.18 is another schematic diagram of a user controlling a speaker through a handheld device provided in an embodiment of the present application. For example, Fig.18 Figure (a) shows the main interface 1801 of the mobile phone. When the mobile phone is in the bright screen state and displays the main interface 1801 of the mobile phone, the mobile phone can display the following when detecting the user's pointing operation: Fig.18 The interface 1802 shown in FIG. 5 (b) displays a guide icon 50 on the interface 1802 to guide the user to perform a correct pointing operation.

[0299] If the user uses the mobile phone to point at the air conditioner in the living room, since the air conditioner and the speaker are located very close, when the mobile phone detects the user's pointing operation, it determines that the pointing operation can trigger both the air conditioner and the speaker to be controlled by the mobile phone.

[0300] In this scenario, the phone can further display Fig.18 The interface 1803 shown in FIG. (c) in FIG. 1803 simultaneously displays the speaker card 70 corresponding to the speaker currently pointed to by the user and the air conditioner card 80 corresponding to the air conditioner. Optionally, the speaker card 70 may only display part of the device name, icon, zoom option, and window close option; similarly, the air conditioner card 80 may only display part of the device name, icon, zoom option, and window close option, which is not limited in the embodiments of the present application.

[0301] Users can execute Fig.18 In the operation shown in FIG. (c), click on any area of ​​the speaker card 70 that is currently expected to be operated or controlled. In response to the user's click operation, the mobile phone displays the following Fig.18 The interface 1804 shown in FIG. 1804 displays a speaker control window 20, which will not be described in detail here.

[0302] The above describes the implementation process when the mobile phone recognizes that the user is pointing to two devices. When the mobile phone recognizes that the user is pointing to more devices, the method can also be used to display cards of multiple devices for the user, so that the user can select at least one target device according to his or her needs. In addition, the user can also click to select multiple devices as target devices at the same time, so that multiple device control windows can be displayed on the interface 1804, and the user can change different device control windows by sliding switching, which will not be repeated here.

[0303] Through the above process, for multiple devices in close proximity, the handheld device may identify multiple devices pointed to by the user, or when the user desires to control the multiple devices in close proximity, multiple device cards may be displayed for the user, allowing the user to select at least one target device according to their needs. This design is more user-friendly, can better meet the different needs of users, and improves the user experience.

[0304] The above embodiments are combined with Figures 5 to 18 , from the user interaction level, introduces the interaction method between pointing devices. Fig.19 , from the perspective of software implementation strategy, the interaction method between devices provided by the embodiment of the present application is introduced. It should be understood that the method can be used in Figure 1 , Figure 2 The structure shown is implemented in an electronic device (such as a mobile phone, tablet computer, etc.) with a touch screen, a positioning chip, a camera component, etc.

[0305] Fig.19 is a schematic flow chart of an example of an interaction method between pointing devices provided in an embodiment of the present application. Taking a mobile phone as a handheld device, such as Fig.19 As shown, the method may include the following steps:

[0306] 1901. The user turns on the finger control function of the handheld device.

[0307] For example, the user can follow Figure 5 By turning on the "One-finger control function" switch in the Settings app, you can further enable the phone's function of using quick gestures to interact between devices.

[0308] Alternatively, the "one-finger control function" switch can be added to other applications, for example, the "one-finger control function" switch can be added to the smart life application. Users can use the "one-finger control function" switch of smart life to turn on the mobile phone's function of interacting between devices through quick gestures.

[0309] Alternatively, in addition to enabling the quick interaction function between devices in the settings application and smart life application installed on the mobile phone, you can also add a "one-finger control function" switch in the notification bar of the mobile phone. For example, the user can perform a pull-down operation from the top of the mobile phone on any interface of the mobile phone, and in response to the user's pull-down operation, the mobile phone displays the notification interface. The user can click the "one-finger control function" switch in the notification interface to enable the mobile phone to interact with devices through quick gestures. The embodiment of the present application does not limit the method of enabling the quick interaction function between devices.

[0310] It should be understood that as a user's handheld device, the mobile phone also needs to locate smart TVs (smart screens), air conditioners, speakers, routers and other devices in the home scene. That is, the mobile phone needs to determine the location of each device, and then control it more accurately through quick gestures.

[0311] In a possible implementation, at least three fixed positioning devices are installed in a home scene, and it is ensured that the three positioning devices are powered on and in working state, and the at least three positioning devices can communicate with the user's handheld device.

[0312] Optionally, at least three positioning devices may be provided according to Figure 6 Install according to the methods and rules described in Figure 7 The method shown in is to establish a connection between the mobile phone and the positioning device, that is, the mobile phone can send signals to the positioning device or receive signals sent by the positioning device. The mobile phone can be combined with at least three positioning devices, according to Figure 8 The process described here determines the location of any device in a home scenario.

[0313] Optionally, the positioning device may be a sensor with a positioning function or a structure with a positioning function, for example, the sensor may be a laser sensor, an infrared sensor, etc.; the structure with a positioning function may be a chip. For example, the structure with a positioning function may be a positioning structure based on a Bluetooth module, a positioning chip based on ultra-wideband UWB wireless sensing capability, a positioning structure based on GPS, a positioning structure based on a WIFI module, etc.

[0314] Optionally, in addition to pre-arranging the above-listed positioning devices in the home scene, the positioning device can also be installed on smart TVs (smart screens), air conditioners, speakers, routers and other devices in the home scene. For example, if the smart screen, air conditioner, speaker and other devices themselves have UWB chips, there is no need to arrange at least 3 additional fixed positioning devices in the home scene, and the positioning function of any device in the home scene can be realized through the interaction between the mobile phone and the smart screen, air conditioner, speaker and other devices.

[0315] 1902, detecting a pointing operation of a user.

[0316] Exemplarily, in a possible implementation, a mobile phone, as a handheld device of a user, can identify the user's pointing operation with the help of one or more sensors of the mobile phone, or a camera of the mobile phone.

[0317] Optionally, the sensor of the mobile phone may include an accelerometer, a gyroscope, a magnetometer, an inertial detection unit (IMU), etc. The camera of the mobile phone is not limited to the front camera or the rear camera of the mobile phone; or, the camera of the mobile phone is not limited to one or more of the main camera, wide-angle camera, and telephoto camera of the mobile phone. The user's pointing operation is recognized based on the image obtained by the camera, and this embodiment of the application does not limit this. The specific detection process can be referred to the aforementioned Fig.10 The specific description of the operation process of the user pointing to the speaker will not be repeated here.

[0318] 1903, there are other devices in the direction pointed by the user.

[0319] 1904, when no other device is detected in the pointing direction, the handheld device provides guidance and feedback on the pointing operation to the user through visual and / or non-visual prompts on the handheld device and the pointed target device.

[0320] It should be understood that, when the user points to the target device, the hand-eye separation phenomenon may occur, that is, the eyes cannot see the interface of the mobile phone. Therefore, the embodiment of the present application does not limit the prompt method of any of the above prompt information (the first prompt information, the second prompt information, the third prompt information, the fourth prompt information, etc.), for example, it can be a visual or non-visual prompt that is recognizable to the user, such as different interface prompts, vibration, indicator light, voice and other prompt methods.

[0321] In a possible implementation, when the user points to the target device, Fig.17 As shown in Figure (e), the mobile phone can automatically display a guidance icon on the interface, and the guidance icon can be suspended on the interface of the mobile phone in the form of an arrow.

[0322] In another possible implementation, when the user uses a handheld device to point to a target device, in addition to the guide icon, the user can be guided to perform the correct pointing operation through non-visual methods such as sound feedback and vibration feedback, so as to achieve the control of the target device through pointing operation provided in the embodiment of the present application.

[0323] Optionally, the voice feedback may be issued by a handheld device, or by a target device being pointed at, for example, a mobile phone prompts the user in a voice message “please move right”.

[0324] Optionally, the vibration feedback may be emitted by a handheld device so as to be perceived by the user more quickly and sensitively. It should be understood that the embodiments of the present application are not limited to this.

[0325] 1905, when the presence of other devices is detected in the pointing direction, the handheld device determines the distance and direction between the handheld device and the other devices based on wireless positioning technology or three-point positioning technology between multiple devices.

[0326] For example, after the mobile phone recognizes the user's pointing operation at the target device, it further activates the wireless positioning function of the mobile phone to search for devices around the mobile phone, and determines the location of the surrounding devices, as well as the distance between the surrounding devices and the mobile phone, to determine whether there is a device in the direction pointed by the user. Fig.10 As shown in Figure (c), when the mobile phone detects the user's pointing operation as shown in the figure, the mobile phone's infinite positioning technology can be triggered.

[0327] In a possible implementation, when the target device pointed to by the user is equipped with a positioning structure, the mobile phone can perform wireless positioning based on the two-way connection between itself and the target device. Exemplarily, the target device pointed to by the user can be equipped with a Bluetooth positioning chip, a UWB positioning chip, or a GPS positioning structure, a WIFI positioning structure, and one or more structures such as a laser sensor and an infrared sensor with positioning function, and the target device can be positioned based on the two-way connection between the target device and the mobile phone.

[0328] For example, if smart screens, air conditioners, speakers and other devices themselves have UWB chips, the positioning of any device in the home scene can be achieved through the interaction between mobile phones and smart screens, air conditioners, speakers and other devices.

[0329] It should be understood that both the mobile phone and the pointed target device have hardware capable of transmitting or receiving wireless positioning signals, and based on the transmission or reception of wireless positioning signals between the mobile phone and the pointed target device, the distance and angle between the mobile phone and the pointed target device are calculated and determined.

[0330] In another possible implementation, when three positioning devices are deployed in a home scene, the mobile phone can combine the three-point positioning capabilities of the three positioning devices to determine the distance and direction between the mobile phone and the target device it is pointing to.

[0331] 1906 , determining whether the axis of the handheld device substantially intersects the physical position of the target device with a preset accuracy.

[0332] When determining whether the handheld device is pointing to the target device, if the axis of the handheld device and the position of the target device are within a preset range, the target device pointed to by the user can be determined. Specifically, if the axis of the handheld device is identified to substantially intersect the physical position of the target device with a certain predetermined accuracy, or if the handheld device is identified to be aligned with the target device, the target device pointed to by the user can be determined, that is, feedback can be provided on the handheld device or the pointed target device, and the login account information and device information of the target device can be transmitted to the handheld device, and the target device control window 20 or an interface related to the target device can be displayed on the handheld device.

[0333] Optionally, the user's pointing operation can be described as a pointing operation in which the user holds the mobile phone and moves from the first position as the starting position to the second position, and at the second position, the axis of the long side of the mobile phone intersects or approximately intersects with one or more other devices. Here, intersection can be understood as other devices just existing in the direction pointed by the user; approximately intersection can be understood as other devices existing within a preset accuracy range in the direction pointed by the user, and both situations can be understood as the user's pointing operation being detected.

[0334] It should be understood that the axis of the mobile phone is related to the antenna layout of the mobile phone. When the antenna is arranged at the front frame of the mobile phone, the user can point the axis where the long side is located to the target device, or when the antenna is arranged inside the mobile phone case, the user can also stand the mobile phone in a similar posture to taking a photo, with the normal line perpendicular to the mobile phone display as the axis line, so that the axis line where the normal line is located points to the target device, and the embodiments of the present application are not limited to this.

[0335] 1907, providing feedback on handheld devices, such as visual feedback, sound feedback, vibration feedback, etc.

[0336] It should be understood that the target device pointed to by the user is determined according to step 1906, that is, the user successfully points to the target device using the handheld device. At this time, feedback can also be provided on the handheld device or the pointed target device, and the feedback is used to inform the user that the pointing operation is successful. The feedback information may include one or more of visual feedback, sound feedback, vibration feedback, etc.

[0337] For example, Fig.17 As shown in Figure (f), when the user successfully points the mobile phone at the speaker, the mobile phone can vibrate to inform the user that the pointing operation is completed.

[0338] 1908, transmit the login account information and device information of the target device to the handheld device, and display the interface related to the target device on the handheld device.

[0339] When the user successfully points the handheld device to the target device, the target device can transmit the currently logged-in account information and device information to the handheld device, and then display the information on the handheld device. Fig.17 The target device control window shown in Figure (g) in the figure.

[0340] 1909, the end.

[0341] In summary, the embodiment of the present application provides a method for realizing cross-device interaction based on pointing operation, in which a user can use a handheld device to point to a target device, and based on one or more sensors such as an accelerometer, a gyroscope, an IMU and / or a camera of the handheld device, detect the user's pointing operation, and trigger the wireless positioning function of the handheld device. The handheld device determines the distance between the handheld device and other devices, as well as the position and direction of other devices according to the wireless positioning function. When it is identified that the axis of the handheld device and the physical position of the target device are judged to be intersecting or substantially intersecting with a predetermined accuracy, at least one feedback such as visual feedback, sound feedback, and vibration can also be provided on the handheld device and / or the target device, and the login account information and device information of the target device are transmitted to the handheld device, and the control window of the target device is displayed on the handheld device, and the user can control the target device through the control window on the handheld device. When it is identified that the axis of the handheld device does not intersect with the physical position of the target device, at least one guidance method such as visual guidance, sound guidance, and vibration can also be provided on the handheld device and / or the target device to guide the user to perform the correct pointing operation, and further realize the display of the control window of the target device and the control function of the target device on the handheld device.

[0342] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0343] In this embodiment, the electronic device can be divided into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0344] In the case of dividing each functional module into corresponding functional modules, Fig.13 A possible schematic diagram of the composition of the electronic device 1300 involved in the above embodiment is shown. Fig.13 As shown, the electronic device 1300 may include: a display unit 1301 , a detection unit 1302 and a processing unit 1303 .

[0345] The display unit 1301 may be used to support the electronic device 1300 in executing the above-mentioned steps 1201 and 1204, etc., and / or other processes of the technology described herein.

[0346] The detection unit 1302 may be used to support the electronic device 1300 in executing the above-mentioned step 1203 and the like, and / or other processes for the technology described herein.

[0347] The processing unit 1303 may be configured to support the electronic device 1300 in executing the above-mentioned step 1202 and / or other processes for the technology described herein.

[0348] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0349] The electronic device provided in this embodiment is used to execute the above-mentioned video playback method, and thus can achieve the same effect as the above-mentioned implementation method.

[0350] In the case of using an integrated unit, the electronic device may include a processing module, a storage module and a communication module. Among them, the processing module can be used to control and manage the actions of the electronic device, for example, it can be used to support the electronic device to execute the steps performed by the above-mentioned display unit 1301, the detection unit 1302 and the processing unit 1303. The storage module can be used to support the electronic device to execute stored program codes and data, etc. The communication module can be used to support the communication between the electronic device and other devices.

[0351] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0352] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 1 Device of the structure shown.

[0353] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the pointing-based interaction method between devices in the above-mentioned embodiment.

[0354] This embodiment also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the interaction method between pointing-based devices in the above-mentioned embodiment.

[0355] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the interaction method between pointing-based devices in the above-mentioned method embodiments.

[0356] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0357] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0358] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0359] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0360] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0361] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0362] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for interaction between devices, characterized in that: Applied to a first electronic device, the method includes: Acquire the acceleration of the first electronic device in the direction of a first axis, the acceleration of a second axis, and the acceleration of a third axis, wherein the first axis is parallel to the long side of the first electronic device, the second axis is parallel to the short side of the first electronic device, and the third axis is perpendicular to a plane defined by the first axis and the second axis; Determine that the first electronic device has a pointing movement from the first position as the starting position to the second position, wherein the determination that the first electronic device has a pointing movement includes one or more of the following: the acceleration in the direction of the first axis is greater than or equal to a first threshold value or the duration of the acceleration in the direction of the first axis being greater than or equal to the first threshold value is greater than or equal to a first preset duration; the acceleration in the direction of the second axis is less than or equal to a second threshold value; the acceleration in the direction of the third axis is less than or equal to a third threshold value; the first electronic device is away from the user's body; the hovering time of the first electronic device when accelerating to stop reaches a second preset duration; detecting that, at the second position, the first axis of the first electronic device intersects with at least one second electronic device, or that the first axis of the first electronic device approximately intersects with the at least one second electronic device within a preset accuracy range; Receiving account information and device information for logging into the at least one second electronic device sent by the at least one second electronic device; displaying a second window, wherein the second window includes information of the at least one second electronic device; In response to a first operation in the second window, a first window is displayed, wherein the first window displays an interface for controlling the second electronic device.

2. The method according to claim 1, characterized in that The obtaining of the acceleration of the first electronic device in the direction of the first axis, the acceleration on the second axis, and the acceleration on the third axis includes: Based on one or more of an acceleration sensor, an inertial measurement unit IMU, and a gyroscope, the acceleration in the direction of the first axis, the acceleration on the second axis, and the acceleration on the third axis are acquired.

3. The method according to claim 1 or 2, characterized in that: Before detecting that the first axis of the first electronic device intersects with at least one second electronic device at the second position, or that the first axis of the first electronic device approximately intersects with the at least one second electronic device within a preset accuracy range, the method further includes: Collecting images during the movement by using a camera of the first electronic device; It is determined that the picture collected by the first electronic device at the first position includes facial feature information of the user, and the picture collected at the second position does not include facial feature information of the user.

4. The method according to claim 1 or 2, characterized in that: Before detecting that the first axis of the first electronic device intersects with at least one second electronic device at the second position, or that the first axis of the first electronic device approximately intersects with the at least one second electronic device within a preset accuracy range, the method further includes: Collecting the magnetic induction intensity during the movement by using the magnetometer of the first electronic device; Determine that the magnetic induction intensity collected by the first electronic device at the first position is greater than or equal to a fourth threshold, the magnetic induction intensity collected at the second position is less than or equal to a fifth threshold, and the fourth threshold is greater than the fifth threshold.

5. The method according to claim 1 or 2, characterized in that: When the acceleration in the direction of the first axis is greater than or equal to the first threshold, the method further includes: The first prompt information is displayed, where the first prompt information is used to guide the user to continue accelerating along the direction of the first axis, and the first prompt information includes one or more of text, icon, sound and vibration.

6. The method according to claim 1 or 2, characterized in that: When the acceleration in the direction of the first axis is greater than or equal to the first threshold and the duration is greater than or equal to the first preset duration, the method further includes: A second prompt message is displayed, where the second prompt message is used to prompt the user that the acceleration time along the direction of the first axis reaches the first preset time, and the second prompt message includes one or more of text, icon, sound and vibration.

7. The method according to claim 1 or 2, characterized in that: When the hovering time of the first electronic device when accelerating to stop reaches a second preset time length, the method further includes: Display third prompt information, where the third prompt information is used to indicate that the first electronic device detects whether the first axis intersects with the second electronic device at the second position, and the third prompt information includes one or more of text, icon, sound and vibration.

8. The method according to claim 1 or 2, characterized in that: After detecting that the first axis of the first electronic device intersects with the second electronic device at the second position, or that the first axis of the first electronic device approximately intersects with the second electronic device within a preset accuracy range, the method further includes: Display fourth prompt information, where the fourth prompt information is used to indicate that the first electronic device is pointing to the second electronic device, and the fourth prompt information includes one or more of text, icon, sound, and vibration.

9. The method according to claim 1 or 2, characterized in that: The detecting that at the second position, the first axis of the first electronic device intersects with the second electronic device, or the first axis of the first electronic device approximately intersects with the second electronic device within a preset accuracy range, comprises: According to the positioning device, it is detected that at the second position, the first axis of the first electronic device intersects with the second electronic device, or the first axis of the first electronic device approximately intersects with the second electronic device within a preset accuracy range, The positioning device is an independently installed positioning sensor, or the positioning device is a positioning chip installed on the first electronic device and the second electronic device, and the positioning chip includes any one of a Bluetooth positioning chip, an ultra-wideband UWB positioning chip, and a wireless fidelity WIFI positioning chip.

10. The method according to claim 1 or 2, characterized in that: The method further comprises: In the first window, detecting a second operation; In response to the second operation, the first electronic device sends a control instruction to the second electronic device, where the control instruction is used to control a behavior of the second electronic device.

11. The method according to claim 1 or 2, characterized in that: The second threshold is equal to the third threshold.

12. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores one or more instructions, and when the one or more instructions are executed by the processor, the electronic device executes the method as claimed in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 11.

14. A computer program product comprising instructions, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 11.

Images