Interaction method between devices based on pointing operation and electronic device

Abstract

The application provides a pointing operation-based interaction method between devices and an electronic device, which can be a mobile phone or the like. The method is based on an acceleration sensor, a gyroscope, an IMU, a camera or the like of a handheld device to detect a pointing operation of a user using the handheld device to a target device, and triggers a wireless positioning function of the handheld device. When it is identified that an axis of the handheld device intersects or approximately intersects the target device, at least one feedback such as visual feedback, sound feedback, vibration or the like can be provided on the handheld device and / or the target device, and account information and device information of the target device logged in are transmitted to the handheld device. A control window of the target device is displayed on the handheld device, and the user can control the target device through the control window on the handheld device. According to the pointing operation of the user, the method helps the user to select a device far away, simplifies the process of interaction between devices, and improves user experience.

Description

[0001] This application is a divisional application, the original application number is 202010781046.7, the original application date is August 5, 2020, and the entire contents of the original application are incorporated into the present application by reference. TECHNICAL FIELD

[0002] The present application relates to the field of electronic technology, and in particular to a device interaction method based on pointing operation and an electronic device. BACKGROUND

[0003] In the environment of a large number of terminals, each user may have different types of electronic devices such as smart phones, personal computers, smart televisions, tablets and sound boxes. In a home scenario, a user may also have a variety of home devices such as smart audio and video devices, routers, wireless fidelity (WIFI) boxes, smart cleaning devices, smart kitchen electrical devices, smart lighting systems, etc. With the development of technology, in the Internet of Everything scenario, the user can control more and more devices, and at the same time, the demand for interconnection between multiple devices is also increasing.

[0004] In the process of interconnection between multiple devices, the user often needs to select a specific target device or multiple target devices, and then perform different operations such as device discovery, pairing, data transmission, screen projection, etc. on the selected target device. The process is cumbersome, the interaction process is complex, and the user experience is poor. SUMMARY

[0005] The present application provides a device interaction method based on pointing operation and an electronic device, which helps the user to select and control the remote device according to the user's pointing operation, simplifies the process of interaction between devices, and improves the user experience.

[0006] In a first aspect, a method for interaction between devices is provided. The method is applied to a first electronic device and includes: obtaining acceleration of the first electronic device in a first axis direction, acceleration in a second axis direction, and acceleration in a third axis direction, wherein the first axis is parallel to a long side of the first electronic device, the second axis is parallel to a short side of the first electronic device, and the third axis is perpendicular to a plane determined by the first axis and the second axis; determining that a user holds the first electronic device, and during movement of the first electronic device from a first position to a second position, the acceleration in the first axis direction is greater than or equal to a first threshold value or the acceleration in the first axis direction is greater than or equal to the first threshold value for a duration greater than or equal to a first preset time length, the acceleration in the second axis direction is less than or equal to a second threshold value, and the acceleration in the third axis direction is less than or equal to a third threshold value; detecting that, at the second position, the first axis of the first electronic device intersects a second electronic device or the first axis of the first electronic device approximately intersects the second electronic device within a preset accuracy range; receiving account information and device information of the second electronic device logged in by the second electronic device; and displaying a first window based on the account information and the device information of the second electronic device logged in by the second electronic device, wherein the first window displays an interface for controlling the second electronic device.

[0007] Optionally, the pointing operation of the user can be described as a pointing operation of the user holding the first electronic device and moving from a first position to a second position, and at the second position, an axis along the long side of the first electronic device intersects or approximately intersects one or more other devices. Here, intersecting can be understood as the existence of other devices in the direction pointed by the user, and approximately intersecting can be understood as the existence of other devices within a preset accuracy range in the direction pointed by the user. Both cases can be understood as detecting the pointing operation of the user.

[0008] In the process of determining the pointing operation of the user holding the first electronic device to point to the second electronic device, a coordinate system O-XYZ is established on the first electronic device. Specifically, the coordinate system O-XYZ takes the center of gravity of the first electronic device as the coordinate origin O, takes a straight line along the long side frame of the first electronic device passing through the coordinate origin O as the Y axis, i.e., the first axis, takes a straight line along the short side frame of the first electronic device as the X axis, i.e., the second axis, and takes a straight line perpendicular to the ground or the XOY plane as the Z axis, i.e., the third axis.

[0009] It should be understood that the axis of the first electronic device is related to the antenna layout of the first electronic device. When the antenna layout is at the front bezel of the first electronic device, the user can point the axis along which the long side is located to the second electronic device, or when the antenna layout is inside the shell of the first electronic device, the first electronic device can also be stood up in a similar posture of taking a photo, and the normal line perpendicular to the display screen of the first electronic device is taken as the axis, so that the axis along which the normal line is located is pointed to the second electronic device, and the embodiments of the present application are not limited in this regard.

[0010] In a possible implementation manner, the pointing operation of the user can be recognized by means of one or more sensors of the first electronic device or a camera of the first electronic device. In the description of subsequent embodiments, the pointing operation of the user can be referred to as a "pointing operation".

[0011] Optionally, the sensors of the first electronic device can include an acceleration sensor, a gyroscope, a magnetometer sensor (M-sensor), an inertial measurement unit (IMU), etc. The camera of the first electronic device is not limited to the front camera or the rear camera of the first electronic device; or the camera of the first electronic device is not limited to one or more of the main camera, the wide-angle camera, and the long-focus camera of the first electronic device, and the pointing operation of the user is recognized according to the picture obtained by the camera, and the embodiments of the present application are not limited in this regard.

[0012] With reference to the first aspect, in some implementation manners of the first aspect, the acceleration of the first electronic device in the first axis direction, the acceleration in the second axis direction, and the acceleration in the third axis direction are obtained by: obtaining the acceleration of the first electronic device in the first axis direction, the acceleration in the second axis direction, and the acceleration in the third axis direction based on one or more of an acceleration sensor, an inertial measurement unit (IMU), and a gyroscope.

[0013] It should be understood that when the user's arm is stretched forward during the movement, the first electronic device accelerates along the straight line along which the Y axis is located, and the acceleration sensor of the first electronic device detects that the acceleration of the first electronic device along the straight line along which the Y axis is located is greater than or equal to a first preset threshold value a A . At the same time, it is detected that the acceleration of the first electronic device along the straight line along which the X axis is located and the straight line along which the Z axis is located is less than or equal to a second preset threshold value a B .

[0014] Optionally, the acceleration of the first electronic device in different directions can be detected by the IMU of the first electronic device. Alternatively, the acceleration of the first electronic device in different directions can be detected based on a picture captured by the front camera of the first electronic device and an always-on optical flow algorithm. Specifically, adjacent frames in the picture are obtained by using the front camera of the first electronic device, wherein a two-dimensional vector field of feature points in the adjacent frames in the translation process is a velocity field of three-dimensional motion of an object point represented by a two-dimensional image. That is, the velocity field of the three-dimensional motion of the object point can reflect the image change within a certain time interval due to the motion of the object point to determine the motion direction and motion rate of the object point on the image, and further determine whether the motion trajectory of the first electronic device is a straight line or a quasi-straight line.

[0015] In one possible implementation, during the operation of the user using the first electronic device to point to the second electronic device, the first electronic device can guide the user to perform the correct pointing operation in different ways such as a guide icon, sound feedback, vibration feedback, and the like, to achieve the pointing operation control of the second electronic device provided by the embodiments of the present application.

[0016] Optionally, the sound feedback can be emitted by the first electronic device or by the pointed second electronic device, for example, the first electronic device prompts the user in the form of voice "please move to the right".

[0017] Optionally, the vibration feedback can be emitted by the first electronic device to be perceived more quickly and sensitively by the user. It should be understood that the embodiments of the present application do not limit this.

[0018] Alternatively, after the user successfully points to the second electronic device using the first electronic device, feedback can also be provided on the first electronic device or the pointed second electronic device, and the feedback is used to inform the user that the pointing operation is successful. The feedback information can include one or more of visual feedback, sound feedback, vibration feedback, and the like. For example, after the user successfully points to the sound box using the first electronic device, the first electronic device can vibrate to inform the user that the pointing operation is completed.

[0019] It should be understood that after the user successfully points to the second electronic device using the first electronic device, the second electronic device can transmit the currently logged account information and device information to the first electronic device, and then display the second electronic device control window on the first electronic device.

[0020] It should also be understood that when determining whether the first electronic device is pointing at the second electronic device, if the axis of the first electronic device and the position of the second electronic device are identified to be within a preset range, it can be determined that the second electronic device is pointed at by the user. Specifically, when the axis of the first electronic device is identified to substantially intersect with the physical position of the second electronic device with a certain predetermined accuracy, or when the first electronic device is identified to be aligned with the second electronic device, it can be determined that the second electronic device is pointed at by the user, that is, feedback can be provided on the first electronic device or the pointed second electronic device, and the logged account information and device information of the second electronic device can be transmitted to the first electronic device, and a second electronic device control window or an interface related to the second electronic device can be displayed on the first electronic device.

[0021] In summary, the method for implementing cross-device interaction based on pointing operation provided by the embodiments of the present application can be used for a user to point at a second electronic device using a first electronic device, and the pointing operation of the user can be detected based on one or more sensors such as an acceleration sensor, a gyroscope, an IMU, and / or a camera of the first electronic device, and the wireless positioning function of the first electronic device can be triggered. The first electronic device can determine the distance between the first electronic device and other devices and the position and direction of the other devices according to the wireless positioning function. When the axis of the first electronic device is identified to intersect or substantially intersect with the physical position of the second electronic device with a predetermined accuracy, visual feedback, sound feedback, vibration, or at least one of the feedback can be provided on the first electronic device and / or the second electronic device, the logged account information and device information of the second electronic device can be transmitted to the first electronic device, and a control window of the second electronic device can be displayed on the first electronic device, and the user can control the second electronic device through the control window on the first electronic device. When the axis of the first electronic device is identified to not intersect with the physical position of the second electronic device, visual guidance, sound guidance, vibration, or at least one of the guidance modes can be provided on the first electronic device and / or the second electronic device to guide the user to perform a correct pointing operation, and the control window of the second electronic device can be further displayed on the first electronic device and the control function of the second electronic device can be implemented.

[0022] In combination with the first aspect and the above implementation manners, in some implementation manners of the first aspect, before the first axis of the first electronic device and the second electronic device are determined to intersect or approximately intersect within a preset accuracy range at the second position, the method further includes: capturing a picture during the movement by a camera of the first electronic device; and determining that the picture captured by the first electronic device at the first position includes facial feature information of a user, and the picture captured at the second position does not include the facial feature information of the user.

[0023] Optionally, the first electronic device can detect the state of being away from or close to the user's body through images acquired by the camera. Specifically, the camera of the first electronic device can acquire images in real time, and perform face feature detection according to the acquired images. Within a certain period of time, the first electronic device first detects face information in the acquired images, and determines that the state is close to the user's body; then, no face information is detected in the acquired images, and it is determined that the state is away from the user's body. When the above change process occurs, it can be determined that the first electronic device gradually changes from being close to the user's body to being away from the user's body.

[0024] With reference to the first aspect and the above implementation manners, in some implementation manners of the first aspect, before detecting that the first axis of the first electronic device intersects with the second electronic device or the first axis of the first electronic device approximately intersects with the second electronic device within a preset accuracy range at the second position, the method further includes: collecting, by the magnetometer of the first electronic device, a magnetic induction intensity in the motion process; determining that the magnetic induction intensity collected by the first electronic device at the first position is greater than or equal to a fourth threshold value, and the magnetic induction intensity collected at the second position is less than or equal to a fifth threshold value, and the fourth threshold value is greater than the fifth threshold value.

[0025] Optionally, the first electronic device can detect the state of being away from or close to the user's body through the magnetometer. Since the human body itself is a magnetic field, when the first electronic device is away from or close to the human body, the magnetic field intensity detected by the magnetometer will change obviously. Therefore, the state of the first electronic device being away from or close to the user's body can be determined according to the magnetic field intensity detected by the magnetometer.

[0026] With reference to the first aspect and the above implementation manners, in some implementation manners of the first aspect, before detecting that the first axis of the first electronic device intersects with the second electronic device or the first axis of the first electronic device approximately intersects with the second electronic device within a preset accuracy range at the second position, the method further includes: determining that a duration of hovering of the first electronic device at the second position is greater than or equal to a second preset duration.

[0027] Specifically, when the IMU of the first electronic device detects that the first electronic device stops accelerating, and the hovering time of the first electronic device reaches the second preset duration t dwell , it can be determined that the user points to the second electronic device.

[0028] When the pointing operation is detected, the first electronic device further triggers the position of the device pointed by the user to be determined through the wireless positioning technology. This process can combine the data collected by multiple sensors to more accurately determine that the user performs the pointing operation.

[0029] With reference to the first aspect and the above implementation manners, in some implementations of the first aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold, the method further includes: displaying first prompt information, the first prompt information being used to guide the user to continue to accelerate in the direction of the first axis, the first prompt information including one or more of text, an icon, a sound, and a vibration.

[0030] It should be understood that, due to hand-eye separation in the process of pointing to the second electronic device by the user, the interface of the first electronic device cannot be seen. Therefore, the application does not limit the prompting manner of the prompt information (the first prompt information, the second prompt information, the third prompt information, the fourth prompt information, and the like) described above, for example, the prompt information can be visual or non-visual prompt information that can be recognized by the user, such as different interface prompts, vibrations, indicator lights, voices, and the like.

[0031] In a possible implementation, in the process of pointing to the second electronic device by the user using the first electronic device, prompt information can be displayed on the first electronic device, the prompt information being used to guide the user to perform correct pointing operations to control the second electronic device through the pointing operations.

[0032] Optionally, the prompt information can be displayed through a window displayed on the first electronic device, or in the form of an icon on the first electronic device, or in the form of an arrow or other visual guidance to guide the user to perform correct pointing operations to control the second electronic device through the pointing operations. Optionally, the display position of the arrow or the icon can be changed according to different positions of the user.

[0033] With reference to the first aspect and the above implementation manners, in some implementations of the first aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold for a duration greater than or equal to the first preset duration, the method further includes: displaying second prompt information, the second prompt information being used to prompt the user that the acceleration in the direction of the first axis has reached the first preset duration, the second prompt information including one or more of text, an icon, a sound, and a vibration.

[0034] With reference to the first aspect and the above implementation manners, in some implementations of the first aspect, when the first electronic device hovers at the second position for a duration greater than or equal to a second preset duration, the method further includes: displaying third prompt information, the third prompt information being used to indicate whether the first axis intersects the second electronic device at the second position, the third prompt information including one or more of text, an icon, a sound, and a vibration.

[0035] With reference to the first aspect and the above implementation manners, in some implementation manners of the first aspect, after detecting that the first axis of the first electronic device intersects with the second electronic device or the first axis of the first electronic device approximately intersects with the second electronic device within a preset accuracy range at the second position, the method further includes: displaying fourth prompt information, the fourth prompt information being used to indicate that the first electronic device is pointed at the second electronic device, the fourth prompt information including one or more of text, an icon, a sound, and vibration.

[0036] In a possible implementation manner, in order to accurately identify the pointing operation of the user and reduce the false touch rate, the embodiment of the present application can trigger the first electronic device to detect the pointing operation of the user only in a preset scenario. The preset scenario includes: the first electronic device is in a bright screen state; and / or, the first electronic device is in an unlocked state; and / or, the first electronic device displays a main interface; and / or, a normal direction of a display screen of the first electronic device and the ground have a certain included angle. The certain included angle between the normal direction of the display screen of the first electronic device and the ground can be understood as that the normal direction of the display screen is not perpendicular to the ground.

[0037] In another possible implementation manner, when the IMU of the first electronic device detects that the first electronic device stops accelerating, and the hovering time of the first electronic device reaches a second preset time t dwell When the IMU of the first electronic device detects that the first electronic device stops accelerating, and the hovering time of the first electronic device reaches a second preset time t

[0038] Alternatively, optionally, fourth prompt information can also appear on the pointed device at this time, for prompting the user to perform a correct pointing operation to point at the device. For example, if the user points at the smart screen through the first electronic device, since the user gazes at the smart screen during the pointing process, a pop-up window including the fourth prompt information can appear on the smart screen, for indicating the correct pointing operation that the user needs to perform to control the smart screen.

[0039] The above guides and feeds the pointing operation on the first electronic device and the pointed second electronic device through visual and non-visual prompt manners, improves the success rate of the pointing operation, and improves the user experience.

[0040] Through the above preset scene setting, it is limited that the first electronic device can recognize whether the user has performed the pointing operation only in a preset scene or simultaneously meet multiple scenes, for example, the first electronic device only recognizes whether the user has performed the pointing operation in the screen-on state and / or the unlocked state, or the first electronic device only recognizes whether the user has performed the pointing operation when the first electronic device displays the main interface, or the first electronic device only recognizes whether the user has performed the pointing operation when the first electronic device simultaneously meets the screen-on state, the unlocked state, and the main interface of the first electronic device is displayed. The preset scene setting can increase the accuracy of recognizing the user's pointing operation, avoid detecting in possible scenarios such as the user only handing the first electronic device to others, thereby reducing the false touch rate and improving the user experience.

[0041] With reference to the first aspect and the above implementation manners, in some implementation manners of the first aspect, the detecting that the first axis of the first electronic device intersects the second electronic device or the first axis of the first electronic device approximately intersects the second electronic device within a preset accuracy range at the second position comprises: detecting that the first axis of the first electronic device intersects the second electronic device or the first axis of the first electronic device approximately intersects the second electronic device within a preset accuracy range at the second position according to a positioning device, wherein the positioning device is a positioning sensor independently installed, or the positioning device is a positioning chip installed on the first electronic device and the second electronic device, and the positioning chip comprises any one of a Bluetooth positioning chip, an ultra-wideband (UWB) positioning chip, and a wireless fidelity (WIFI) positioning chip.

[0042] In a possible implementation manner, in a home scenario, at least three fixed positioning devices can be installed, and it is ensured that the three positioning devices are in a powered-on working state, and the at least three positioning devices can communicate with the first electronic device of the user.

[0043] Optionally, the positioning device can be a sensor with a positioning function or a structure with a positioning function, for example, the sensor can be a laser sensor, an infrared sensor, or the like; and the structure with a positioning function can be a positioning chip. For example, the structure with a positioning function can be a Bluetooth positioning chip, an ultra-wideband (UWB) positioning chip, a wireless fidelity (WIFI) positioning chip, or the like.

[0044] Optionally, in addition to pre-laying the above-mentioned positioning devices in the home scene, the positioning devices can also be installed on smart TVs (smart screens), air conditioners, sound boxes, routers and other devices in the home scene. For example, if the smart screen, air conditioner, sound box and other devices themselves have UWB chips, at least three fixed positioning devices do not need to be additionally laid out in the home scene, and the positioning function of any device in the home scene can be realized through the interaction between the first electronic device and the smart screen, air conditioner, sound box and other devices. The specific positioning algorithm is described in the subsequent process. It should be understood that the number and implementation form of the positioning device are not limited by the embodiments of the present application.

[0045] Through the above scheme, the first electronic device can detect the second electronic device pointed by the user through the three positioning devices laid out in the home scene, or the first electronic device detects the second electronic device pointed by the user through the positioning chip, and further after detecting the quick gesture pointed by the user, the second electronic device can be controlled by the first electronic device of the user.

[0046] In combination with the first aspect and the above implementation manners, in some implementation manners of the first aspect, when the first axis of the first electronic device intersects or approximately intersects one or more second electronic devices, the method further includes: displaying a second window, the second window including information of the one or more second electronic devices; detecting a first operation in the second window; and in response to the first operation, the first electronic device displays the first window.

[0047] Through the above process, for multiple second electronic devices in close proximity, the first electronic device can identify multiple devices pointed by the user, or when the user expects to control the multiple devices in close proximity, the first electronic device can display cards of the multiple devices for the user to select at least one second electronic device according to the user's needs. The design is more humanized and can better meet the different needs of users, improving the user experience.

[0048] In combination with the first aspect and the above implementation manners, in some implementation manners of the first aspect, the method further includes: detecting a second operation in the first window; and in response to the second operation, the first electronic device sends a control instruction to the second electronic device, the control instruction being used to control the behavior of the second electronic device.

[0049] In a possible case, the second threshold value is equal to the third threshold value.

[0050] In a second aspect, a first electronic device is provided, including a processor and a memory storing one or more instructions that, when executed by the processor, cause the electronic device to perform the following steps: obtaining an acceleration in a first axis direction, an acceleration in a second axis direction, and an acceleration in a third axis direction, wherein the first axis is parallel to a long side of the first electronic device, the second axis is parallel to a short side of the first electronic device, and the third axis is perpendicular to a plane determined by the first axis and the second axis; determining that a user holds the first electronic device, and during movement of the first electronic device from a first position to a second position, the acceleration in the first axis direction is greater than or equal to a first threshold value or the acceleration in the first axis direction is greater than or equal to the first threshold value for a duration greater than or equal to a first preset time duration, the acceleration in the second axis direction is less than or equal to a second threshold value, and the acceleration in the third axis direction is less than or equal to a third threshold value; detecting that, at the second position, the first axis of the first electronic device intersects a second electronic device or the first axis of the first electronic device approximately intersects the second electronic device within a preset accuracy range; receiving account information and device information of the second electronic device logged in by the second electronic device; and displaying a first window based on the account information and the device information of the second electronic device logged in by the second electronic device, the first window displaying an interface for controlling the second electronic device.

[0051] With reference to the second aspect, in some implementations of the second aspect, the one or more instructions, when executed by the processor, cause the electronic device to perform the following steps: obtaining the acceleration in the first axis direction, the acceleration in the second axis direction, and the acceleration in the third axis direction based on one or more of an acceleration sensor, an inertial measurement unit (IMU), or a gyroscope.

[0052] With reference to the second aspect and the above implementation, in some implementations of the second aspect, the one or more instructions, when executed by the processor, cause the electronic device to perform the following steps: capturing a picture during the movement by a camera of the first electronic device; and determining that the picture captured at the first position includes facial feature information of a user and the picture captured at the second position does not include the facial feature information of the user.

[0053] With reference to the second aspect and the above implementation, in some implementations of the second aspect, the one or more instructions, when executed by the processor, cause the electronic device to perform the following steps: capturing a magnetic induction intensity during the movement by a magnetometer of the first electronic device; and determining that the magnetic induction intensity captured at the first position is greater than or equal to a fourth threshold value and the magnetic induction intensity captured at the second position is less than or equal to a fifth threshold value, and the fourth threshold value is greater than the fifth threshold value.

[0054] With reference to the second aspect and the foregoing implementation manners, in some implementations of the second aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold value, the one or more instructions, when executed by the processor, cause the electronic device to further perform the following step: displaying first prompt information for guiding the user to continue accelerating in the direction of the first axis, the first prompt information including one or more of text, an icon, a sound, and a vibration.

[0055] With reference to the second aspect and the foregoing implementation manners, in some implementations of the second aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold value, the one or more instructions, when executed by the processor, cause the electronic device to further perform the following step: displaying first prompt information for guiding the user to continue accelerating in the direction of the first axis, the first prompt information including one or more of text, an icon, a sound, and a vibration.

[0056] With reference to the second aspect and the foregoing implementation manners, in some implementations of the second aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold value, the one or more instructions, when executed by the processor, cause the electronic device to further perform the following step: displaying first prompt information for guiding the user to continue accelerating in the direction of the first axis, the first prompt information including one or more of text, an icon, a sound, and a vibration.

[0057] With reference to the second aspect and the foregoing implementation manners, in some implementations of the second aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold value, the one or more instructions, when executed by the processor, cause the electronic device to further perform the following step: displaying first prompt information for guiding the user to continue accelerating in the direction of the first axis, the first prompt information including one or more of text, an icon, a sound, and a vibration.

[0058] With reference to the second aspect and the foregoing implementation manners, in some implementations of the second aspect, when the acceleration in the direction of the first axis is greater than or equal to the first threshold value, the one or more instructions, when executed by the processor, cause the electronic device to further perform the following step: displaying first prompt information for guiding the user to continue accelerating in the direction of the first axis, the first prompt information including one or more of text, an icon, a sound, and a vibration.

[0059] With reference to the second aspect and the above implementation manners, in some implementations of the second aspect, when the one or more instructions are executed by the processor, the electronic device further performs the following steps: detecting, according to a positioning device, that the first axis of the first electronic device intersects the second electronic device or that the first axis of the first electronic device approximately intersects the second electronic device at the second position, wherein the positioning device is a positioning sensor independently installed, or the positioning device is a positioning chip installed on the first electronic device and the second electronic device, and the positioning chip includes any one of a Bluetooth positioning chip, an ultra-wideband (UWB) positioning chip, and a wireless fidelity (WIFI) positioning chip.

[0060] With reference to the second aspect and the above implementation manners, in some implementations of the second aspect, when the first axis of the first electronic device intersects or approximately intersects the one or more second electronic devices, when the one or more instructions are executed by the processor, the electronic device further performs the following steps: displaying a second window, the second window including information of the one or more second electronic devices; detecting a first operation in the second window; and in response to the first operation, the first electronic device displays the first window.

[0061] With reference to the second aspect and the above implementation manners, in some implementations of the second aspect, when the one or more instructions are executed by the processor, the electronic device further performs the following steps: detecting a second operation in the first window; and in response to the second operation, the first electronic device sends a control instruction to the second electronic device, the control instruction being used to control a behavior of the second electronic device.

[0062] With reference to the second aspect and the above implementation manners, in some implementations of the second aspect, the second threshold value is equal to the third threshold value.

[0063] In a third aspect, the present application provides a device, which is included in an electronic device, and the device has a function of implementing behaviors of the electronic device in the above aspects and possible implementation manners of the above aspects. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a display module or unit, a detection module or unit, a processing module or unit, and the like.

[0064] In a fourth aspect, the present application provides an electronic device, comprising: a touch display screen, wherein the touch display screen comprises a touch-sensitive surface and a display; a positioning chip; one or more cameras; one or more processors; one or more memories; a plurality of application programs; and one or more computer programs. The one or more computer programs are stored in the memory, and the one or more computer programs comprise instructions. When the instructions are executed by the one or more processors, the electronic device performs the interaction method between devices based on the pointing operation in any possible implementation of any of the above aspects.

[0065] In a fifth aspect, the present application provides an electronic device, comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are configured to store computer program codes, the computer program codes comprising computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the interaction method between devices based on the pointing operation in any possible implementation of any of the above aspects.

[0066] In a sixth aspect, the present application provides a computer storage medium, comprising computer instructions. When the computer instructions are run on an electronic device, the electronic device performs the method for starting the shortcut function in any possible implementation of any of the above aspects.

[0067] In a seventh aspect, the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device performs the interaction method between devices based on the pointing operation in any of the above aspects. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0069] Figure 2 FIG. 2 is a software structure block diagram of an electronic device provided by an embodiment of the present application.

[0070] Figure 3 FIG. 3 is a graphical user interface diagram of device interconnection provided by an embodiment of the present application.

[0071] Figure 4 FIG. 4 is another graphical user interface diagram of device interconnection provided by an embodiment of the present application.

[0072] Figure 5 FIG. 5 is a graphical user interface diagram of starting the shortcut function between devices provided by an embodiment of the present application.

[0073] Figure 6 FIG. 6 is a schematic diagram of installing a positioning device provided by an embodiment of the present application.

[0074] Figure 7 is a schematic diagram provided by an embodiment of the present application for establishing a connection between a handheld device and a positioning device.

[0075] Figure 8 is a schematic diagram provided by an embodiment of the present application for determining the position of a device in a home scenario.

[0076] Figure 9 is a schematic diagram provided by an embodiment of the present application for setting a device in a home scenario.

[0077] Figure 10 is a schematic diagram provided by an embodiment of the present application for a user pointing to a sound box.

[0078] Figure 11 is a schematic diagram provided by an embodiment of the present application for a user pointing to a sound box.

[0079] Figure 12 is a schematic diagram provided by an embodiment of the present application for a user controlling a sound box through a handheld device.

[0080] Figure 13 is a schematic diagram provided by an embodiment of the present application for a user pointing to a smart screen.

[0081] Figure 14 is a schematic diagram provided by an embodiment of the present application for a user controlling a smart screen through a handheld device.

[0082] Figure 15 is a schematic diagram provided by an embodiment of the present application for a user pointing to an air conditioner.

[0083] Figure 16 is a schematic diagram provided by an embodiment of the present application for a user controlling an air conditioner through a handheld device.

[0084] Figure 17 is a schematic diagram provided by an embodiment of the present application for a user controlling a sound box through a handheld device.

[0085] Figure 18 is a schematic diagram provided by an embodiment of the present application for a user controlling a sound box through a handheld device.

[0086] Figure 19 is a schematic diagram provided by an embodiment of the present application for a user controlling a sound box through a handheld device. DETAILED DESCRIPTION

[0087] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in this document only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0088] Hereinafter, the terms "first" and "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features.

[0089] The method provided by the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not make any limitation on the specific type of electronic device.

[0090] Exemplary, Figure 1A structural diagram of the electronic device 100 is shown. The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0091] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0092] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0093] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0094] The processor 110 can also have a memory that stores instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or is using repeatedly. If the processor 110 needs to use the instructions or data again, it can call them directly from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0095] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0096] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can include multiple sets of I2C buses. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the electronic device 100.

[0097] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple sets of I2S buses. The processor 110 can be coupled with the audio module 170 through the I2S buses to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the I2S interface to enable the function of answering a phone call through a Bluetooth earphone.

[0098] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In some embodiments, the audio module 170 can be coupled with the wireless communication module 160 through a PCM bus interface. In some embodiments, the audio module 170 can also deliver audio signals to the wireless communication module 160 through the PCM interface to enable the function of playing music through a Bluetooth earphone. Both the I2S interface and the PCM interface can be used for audio communication.

[0099] The UART interface is a universal serial bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to enable Bluetooth functionality. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the UART interface to enable the function of playing music through a Bluetooth earphone.

[0100] The MIPI interface can be used to connect the processor 110 and peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to enable the camera function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to enable the display function of the electronic device 100.

[0101] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

[0102] The USB interface 130 is an interface conforming to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0103] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection modes or combinations of multiple interface connection modes in the above embodiments.

[0104] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through the wireless charging coil of the electronic device 100. The charging management module 140 can charge the battery 142 while also supplying power to the electronic device through the power management module 141.

[0105] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0106] The wireless communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0107] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0108] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transfer the same to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor, and radiate the same as electromagnetic waves through the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the same device as at least part of the modules of the processor 110.

[0109] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the microphone 170B, etc.), or displays an image or a video through the display screen 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110, and disposed in the same device as the mobile communication module 150 or other functional modules.

[0110] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives an electromagnetic wave via the antenna 2, frequency-modulates and filters the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, frequency-modulate it, amplify it, and radiate it as an electromagnetic wave via the antenna 2.

[0111] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidu navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0112] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0113] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0114] The electronic device 100 can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0115] The ISP is configured to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be disposed in the camera 193.

[0116] The camera 193 is configured to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0117] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0118] The video codec is used to compress or decompress digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0119] The NPU is a neural-network (NN) calculation processor, which can quickly process input information by drawing on the structure of a biological neural network, such as drawing on the transmission mode between human brain neurons, and can also constantly self-learn. Through the NPU, the electronic device 100 can realize intelligent cognition applications such as image recognition, face recognition, voice recognition, text understanding, etc.

[0120] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize data storage functions. For example, music, video, etc. Files are saved in the external memory card.

[0121] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various function applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phonebook, etc.), etc. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0122] The electronic device 100 can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0123] The audio module 170 is configured to convert digital audio information into an analog audio signal output, and to convert an analog audio input into a digital audio signal. The audio module 170 can also be configured to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some of the functional modules of the audio module 170 can be disposed in the processor 110.

[0124] The speaker 170A, also referred to as a "loudspeaker", is configured to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0125] The receiver 170B, also referred to as a "earpiece", is configured to convert an audio electrical signal into a sound signal. When the electronic device 100 receives a call or a voice message, the user can listen to the voice by holding the receiver 170B close to the ear.

[0126] The microphone 170C, also referred to as a "microphone", "sound collector", is configured to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can make a sound by holding the mouth close to the microphone 170C, and input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, in addition to collecting sound signals, noise reduction functions can also be realized. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, to realize the collection of sound signals, noise reduction, and also to identify the source of the sound, to realize the function of directional recording, etc.

[0127] The earphone interface 170D is configured to connect a wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0128] The pressure sensor 180A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The pressure sensor 180A can be of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

[0129] The gyroscope sensor 180B can be configured to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake photography. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of shaking of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and lets the lens offset the shaking of the electronic device 100 by reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and motion sensing game scenarios.

[0130] The barometric pressure sensor 180C is configured to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude, assists positioning and navigation by using the air pressure value measured by the barometric pressure sensor 180C.

[0131] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can detect the opening and closing of a flip cover by using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Further, according to the detected opening and closing state of the cover or the flip cover, the electronic device 100 can set a feature such as automatic unlocking of the flip cover.

[0132] The acceleration sensor 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the acceleration sensor 180E can detect the magnitude and direction of gravity. The acceleration sensor 180E can also be used to identify the attitude of the electronic device, and can be applied to landscape / portrait screen switching and pedometer applications.

[0133] Distance sensor 180F is configured to measure distance. Electronic device 100 can measure distance by infrared or laser. In some embodiments, electronic device 100 can utilize distance sensor 180F to measure distance for fast focusing when taking a picture.

[0134] Proximity light sensor 180G can include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode can be an infrared light emitting diode. Electronic device 100 emits infrared light outwardly through the light emitting diode. Electronic device 100 detects infrared reflected light from nearby objects using the photodiode. When sufficient reflected light is detected, electronic device 100 can determine that there is an object near electronic device 100. When insufficient reflected light is detected, electronic device 100 can determine that there is no object near electronic device 100. Electronic device 100 can utilize proximity light sensor 180G to detect that a user is holding electronic device 100 close to the ear for a phone call, so as to automatically turn off the screen to save power. Proximity light sensor 180G can also be used for automatic unlocking and locking of the screen in a holster mode or a pocket mode.

[0135] Ambient light sensor 180L is configured to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 according to the sensed ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking a picture. Ambient light sensor 180L can also cooperate with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touch.

[0136] Fingerprint sensor 180H is configured to collect a fingerprint. Electronic device 100 can utilize the collected fingerprint characteristics to implement fingerprint unlocking, access application lock, fingerprint picture taking, fingerprint call answering, and the like.

[0137] Temperature sensor 180J is configured to detect temperature. In some embodiments, electronic device 100 utilizes the temperature detected by temperature sensor 180J to implement temperature processing strategies. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 implements performance reduction of a processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, electronic device 100 heats battery 142 to avoid abnormal shutdown of electronic device 100 caused by low temperature. In other embodiments, when the temperature is lower than yet another threshold, electronic device 100 implements voltage boosting of the output voltage of battery 142 to avoid abnormal shutdown caused by low temperature.

[0138] Touch sensor 180K, also referred to as "touch panel". Touch sensor 180K can be disposed on display screen 194, and touch sensor 180K and display screen 194 together form a touch screen, also referred to as "touch panel". Touch sensor 180K is configured to detect touch operations applied to or near the touch sensor 180K. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K can also be disposed on the surface of electronic device 100, which is different from the position of display screen 194.

[0139] Bone conduction sensor 180M can obtain vibration signals. In some embodiments, bone conduction sensor 180M can obtain vibration signals of the human body's vocal vibration bone block. Bone conduction sensor 180M can also contact the human body pulse to receive blood pressure pulsation signals. In some embodiments, bone conduction sensor 180M can also be disposed in a headset to form a bone conduction headset. Audio module 170 can analyze voice signals based on the vibration signals of the vocal vibration bone block obtained by the bone conduction sensor 180M to realize voice functions. The application processor can analyze heart rate information based on the blood pressure pulsation signals obtained by the bone conduction sensor 180M to realize heart rate detection functions.

[0140] Keys 190 include power on / off keys, volume keys, and the like. Keys 190 can be mechanical keys. They can also be touch keys. Electronic device 100 can receive key input and generate key signal input related to user settings and function control of electronic device 100.

[0141] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations applied to different regions of display screen 194 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminders, received messages, alarms, games, etc.) can also correspond to different vibration feedback effects. Touch vibration feedback effects can also be customizable.

[0142] Indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, and also to indicate messages, missed calls, notifications, and the like.

[0143] The SIM card interface 195 is configured to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external storage cards. The electronic device 100 interacts with a network through the SIM card to achieve functions such as call and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0144] The software system of the electronic device 100 can use a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. Embodiments of the present application exemplarily illustrate the software structure of the electronic device 100 by taking an Android system with a layered architecture as an example.

[0145] Figure 2 FIG. 1 is a software structure block diagram of an example of the electronic device 100 according to an embodiment of the present application. The layered architecture divides the software into several layers, each of which has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer, an application framework layer, an Android runtime and system library, and a kernel layer. The application layer can include a series of application packages.

[0146] As shown in FIG. 1, the application packages can include camera, music, settings, Bluetooth, find, smart life, and the like. Figure 2

[0147] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions. As shown in FIG. 1, the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like. Figure 2

[0148] The window manager is configured to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and take screenshots, and the like.

[0149] ​​The content provider stores and retrieves data and makes the data accessible to the application programs. The data can include videos, images, audio, dialed and received phone calls, browsing history and bookmarks, phone book, etc.

[0150] The view system includes visual controls, such as controls that display text, controls that display pictures, etc. The view system can be used to build an application program. A display interface can be composed of one or more views. For example, a display that includes an icon for a camera application on a phone interface can include a view that displays text and a view that displays a picture.

[0151] The telephony manager is used to provide the communication functions of the electronic device 100. For example, management of call status (including call connection, call hang-up, etc.).

[0152] The resource manager provides various resources for the application programs, such as localized strings, icons, pictures, layout files, video files, etc.

[0153] The notification manager enables the application programs to display notification information in the status bar, which can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of a download, a message reminder, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of a chart or a scrolling bar of text, such as a notification of an application program running in the background, and can also be a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, the electronic device vibrates, the indicator light flashes, etc.

[0154] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0155] The core library includes two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.

[0156] The application program layer and the application program framework layer run in the virtual machine. The virtual machine executes the java files of the application program layer and the application program framework layer into binary files. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection, etc.

[0157] The system library can include multiple functional modules. For example: a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL). The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media libraries support playback and recording of various common audio and video formats, as well as still image files. The media libraries support various audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D graphics processing libraries are used for 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D graphics. The image processing library provides analysis for various image data and offers various image processing algorithms, such as image cropping, image fusion, image blurring, and image sharpening, which will not be elaborated further here.

[0158] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0159] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 1 and Figure 2 Taking the mobile phone with the structure shown as an example, and in conjunction with the accompanying drawings and application scenarios, the interaction method between pointing devices provided in this application embodiment will be specifically described.

[0160] Figure 3 This is a schematic diagram of a graphical user interface (GUI) for device interconnection, which allows users to... Figure 3 The process shown involves detecting other electronic devices and establishing a connection with the target electronic device. Among these, Figure 3 Figure (a) shows the main interface 301 currently displayed on the phone in unlocked mode. This main interface 301 displays multiple applications (Apps), such as music, camera, settings, and smart living applications. It should be understood that this main interface 301 may also include other applications, and this embodiment of the application does not limit this.

[0161] like Figure 3 As shown in Figure (a), the user can perform a swipe operation from the bottom of the phone upwards on the main interface 301, such as the swipe trajectory indicated by the black arrow. In response to the user's swipe operation, the phone enters a state similar to... Figure 3The control center interface 302 can include different control menus or control options, such as "Airplay Mirroring", "Airdrop receiving off", "wireless local area networks (WLAN)", "mute", "auto-rotate", "camera", and the like, which are not limited in the embodiments of the present application.

[0162] For example, the user can perform the operation shown in FIG. 2(b) and click the "Airplay Mirroring" option on the control center interface 302. In response to the user's click operation, the phone enters the Airplay Mirroring details interface 303 as shown in FIG. 2(c). Figure 3 Figure 3 The control center interface 302 can include different control menus or control options, such as "Airplay Mirroring", "Airdrop receiving off", "wireless local area networks (WLAN)", "mute", "auto-rotate", "camera", and the like, which are not limited in the embodiments of the present application.

[0163] If the user currently expects the phone to connect to and control the smart TV in the home scenario, the user can click "userA's TV", and the phone enters the control center interface 403 of the smart TV. The control center interface 403 can include various menus or buttons for controlling the smart TV, such as a connection-off switch, a play progress bar, a pause button, a collection button, and the like. The user can control the smart TV in the home scenario through the menus or buttons of the control center interface 403.

[0164] The above method displays the devices that can be controlled by the phone in a list by performing device discovery and selection on the phone. The user can select a device in the list as needed and perform corresponding control. In this process, when the number of selectable devices in the list exceeds a certain number, the user can identify the target device from the multiple devices according to the device icons, device names, and the like, and combine a screen scrolling operation to select the target device. In one possible scenario, the arrangement order of the devices in the device list is related to the signal strength of the devices. When the signal is unstable or the user's location changes, the arrangement order of the devices in the device list can change, and the user needs to slide multiple times to find the target device, which increases the user's operation cost. In addition, since the phone system cannot perceive the target device that the user expects to select, it can only display all scanned devices to the user, which also increases the user's operation cost and the time to select the target device. Furthermore, when the user selects the target device from the device list, it is easy to cause misoperation such as accidental touch.​

[0165] Figure 4 This is yet another example of a graphical user interface diagram for device interconnection, where users can also... Figure 4 The process shown involves discovering other electronic devices and establishing a connection with the target electronic device. Among these, Figure 4 Figure (a) shows the main screen 401 currently displayed on the phone in unlock mode, which displays multiple applications (Apps), such as music, camera, settings, and search.

[0166] like Figure 4 As shown in Figure (a), when a user clicks to search for an application, the phone responds to the user's click action by entering the following... Figure 4 The search interface 402 is shown in Figure (b). On this search interface 402, the locations of multiple devices can be displayed in a map format, for example, as shown below. Figure 4 As shown in Figure (b), the mobile phone can locate Device 1, Device 2, Device 3, Device 4, and Device 5 and display the location tag for each device to mark its location. Users can establish a connection with the target device by clicking its location tag, or display the target device's control center interface to further control the target device.

[0167] When a search application displays multiple devices on a map, the map needs to occupy a large portion of the phone screen, resulting in a smaller display area for the device list and fewer electronic devices that can be shown. Furthermore, when devices are close together on the map, their positions may overlap, making it easy for users to accidentally tap other devices when trying to select a target device. Alternatively, the process may require users to perform complex interactive operations such as zooming and dragging to select the target device, increasing the user's operational burden and the time required to select the target device.

[0168] In another possible implementation, users can use near field communication (NFC) technology to discover and select devices by touching two devices together where NFC tags are affixed. This process requires the user to bring their phone close to the target device, physically touching a specific area to discover and select the device. However, not all devices support NFC, and both the phone and the target device must display NFC tags on their exteriors, which somewhat affects the device's appearance.

[0169] In summary, existing processes for device discovery, device connection, and inter-device control all require users to perform complex interactive operations, increasing user operating costs and the time spent selecting target devices. Therefore, embodiments of this application will provide a faster device interaction method that enables device discovery, connection, or inter-device control while reducing user operations.

[0170] It should be understood that in the description of the following embodiments, a mobile phone will be used as the user's handheld device, or "control device," to describe how to control or connect other electronic devices in a home setting via the mobile phone. These other electronic devices can be referred to as "controlled devices." In a home setting, the user's handheld device may have... Figure 1 and Figure 2 The devices shown in the diagram, such as mobile phones and tablets, can be controlled by devices including smart TVs (smart screens), air conditioners, speakers, personal computers, routers, etc., which will not be listed here. The following will use the example of a mobile phone controlling a smart TV (smart screen), air conditioner, and speaker in a home setting to detail how to achieve the device control process of this application embodiment through quick gestures.

[0171] Figure 5 This is a schematic diagram of a graphical user interface for enabling quick interaction between devices, provided in an embodiment of this application. Figure 5 Figure (a) shows the main interface 501 currently displayed on the phone in unlocked mode. When the user clicks the settings application icon on the main interface 501, the phone displays the following in response to the user's click: Figure 5 Figure (b) shows the main interface 502 of the settings application. Specifically, as shown in Figure (b), Figure 5 As shown in Figure (b), the main interface 502 of the settings application includes multiple menus for setting mobile network connection, Bluetooth connection, desktop and wallpaper, display and brightness, sound and smart assistance functions.

[0172] When a user taps the Smart Assistance and Gesture Control menu, the phone displays the following in response to the user's tap: Figure 5 Figure (c) shows the details interface 503 for intelligent assistance and gesture control. Interface 503 may further include various sub-menus for intelligent control of the mobile phone, such as travel assistant, one-handed operation, gesture control, smart screen recognition, case operation, stylus operation, etc. Interface 503 may also include switches for different functions such as anti-mistouch mode. It should be understood that this application embodiment does not limit the type or number of menus or switches displayed on the interface; the function of each menu or switch can be implemented with reference to existing technologies, and will not be elaborated here.

[0173] In a possible implementation, the embodiment of the present application can add a "one-finger control function" switch in the intelligent auxiliary and gesture control menu. The user can perform the operation shown in (c) of FIG. 1, click the "one-finger control function" switch, and turn on the device quick interaction function to be introduced in the embodiment of the present application, that is, control the interaction between devices through a quick mobile phone. Figure 5

[0174] It should be understood that the "one-finger control function" switch is introduced in the setting application, and the "one-finger control function" switch can also be added in other applications, for example, the "one-finger control function" switch can be added in the smart life application. The user can turn on the device quick interaction function through the "one-finger control function" switch of the smart life application, and the embodiment is not limited in this regard.

[0175] Alternatively, in addition to turning on the device quick interaction function in the setting application and the smart life application installed on the mobile phone, the "one-finger control function" switch can also be added in the notification bar of the mobile phone. For example, the user can perform a downward pull operation from the top of the mobile phone on any interface of the mobile phone. In response to the downward pull operation of the user, the mobile phone displays the notification interface 302 shown in (b) of FIG. 1, and adds the "one-finger control function" switch in the notification interface 302. The user can click the "one-finger control function" switch to turn on the device quick interaction function, and the embodiment of the present application is not limited in this regard. Figure 3

[0176] Through the method introduced in (c) of FIG. 1, the user turns on the device quick interaction function provided by the embodiment of the present application through a quick gesture. Before the quick gesture is introduced in detail, the mobile phone as a handheld device of the user also needs to realize positioning of the smart television (smart screen), air conditioner, sound box, router and other devices in the home scene, that is, the mobile phone needs to determine the position of each device, and then more accurately control through the quick gesture. Figure 5

[0177] In a possible implementation, at least three fixed positioning devices are installed in the home scene, and the three positioning devices are ensured to be in a powered-on working state, and the at least three positioning devices can communicate with the handheld device of the user.

[0178] ​​​Optionally, the positioning device can be a sensor or structure with positioning functionality. For example, the sensor can be a laser sensor, an infrared sensor, etc.; the structure with positioning functionality can be a chip. For example, the structure with positioning functionality can be a positioning structure based on a Bluetooth module, a positioning chip based on ultra-wideband (UWB) wireless sensing capabilities, a positioning structure based on a global positioning system (GPS), a positioning structure based on a wireless fidelity (WIFI) module, etc.

[0179] Optionally, in addition to pre-deploying the aforementioned positioning devices in a home environment, the positioning devices can also be installed on smart TVs (smart screens), air conditioners, speakers, routers, and other devices within the home environment. For example, if the smart screen, air conditioner, speaker, or other devices themselves have UWB chips, it is not necessary to deploy at least three additional fixed positioning devices in the home environment. The positioning function for any device in the home environment can be achieved through interaction between the mobile phone and the smart screen, air conditioner, speaker, or other devices. For the specific positioning algorithm, please refer to the process described later. It should be understood that this application embodiment does not limit the number or implementation form of the positioning devices.

[0180] Figure 6 This is a schematic diagram illustrating the installation of a positioning device according to an embodiment of this application. Specifically, taking the installation of three fixed positioning devices in a home setting as an example, the installation rules for the positioning devices are described. Specific installation rules may include:

[0181] (1) Barrier-free communication rules.

[0182] Specifically, during the installation of the positioning devices, it is ensured that each positioning device can communicate with any device in the home environment, so that the mobile phone, as a handheld device, can communicate with each positioning device, thereby enabling the mobile phone to accurately locate the position of each device in the home environment.

[0183] For example, Figure 6 Figure (a) illustrates an incorrect installation method where an obstacle obstructs communication between positioning device 1 and device A, potentially preventing the three positioning devices from locating device A in a home environment. During the installation of the positioning devices, it is possible to... Figure 6 As shown in Figure (b), it is ensured that any device B in the home scenario can communicate with positioning device 1, positioning device 2, and positioning device 3 respectively, and there are no obstacles between device B and any positioning device.

[0184] (2) Rules for the installation height of positioning devices.

[0185] Specifically, during the installation of the positioning device, it is ensured that the height of each positioning device from the ground plane is greater than or equal to 2 meters, or referred to as the minimum installation height H-min. The minimum installation height H-min can ensure that the three positioning devices have a larger coverage range, and can locate any device in the home scene.

[0186] Exemplarily, Figure 6 (c) in FIG. 1 shows an incorrect installation method, the height H2 of the positioning device 2 from the ground plane, and the height H3 of the positioning device 3 from the ground plane are less than 2 meters, which may result in a small coverage range of the three positioning devices, and the three positioning devices cannot locate any device in the home scene. During the installation of the positioning device, it can be ensured that Figure 6 (d) in FIG. 1 shows that the heights of the positioning device 1, the positioning device 2, and the positioning device 3 from the ground plane are greater than H-min, that is, the positioning device 1, the positioning device 2, and the positioning device 3 are installed in the shadow area with a distance greater than H-min from the ground plane.

[0187] (3) Large coverage range rule of the positioning device.

[0188] Specifically, during the installation of the positioning device, it is ensured that the combined coverage range of the three positioning devices is maximized, that is, the smart screen, air conditioner, sound box, and other devices in the entire home scene are covered as much as possible.

[0189] Exemplarily, Figure 6 (e) in FIG. 1 shows an incorrect installation method, the area of the triangular region formed after the connection of the positioning device 1, the positioning device 2, and the positioning device 3 is small, which may result in a small coverage range of the three positioning devices, and the three positioning devices cannot locate any device in the home scene. During the installation of the positioning device, it can be ensured that Figure 6 (f) in FIG. 1 shows that the area of the triangular region formed after the connection of the positioning device 1, the positioning device 2, and the positioning device 3 is maximized.

[0190] (4) Vertical installation rule of the positioning device.

[0191] Specifically, during the installation of the positioning device, it is ensured that the three positioning devices are installed on a vertical wall surface, and the positioning device is not installed on a wall surface with an inclined angle from the ground plane.

[0192] Exemplarily, Figure 6Figure (g) in the above (1) illustrates an incorrect installation mode, where the positioning device 1 is installed on an inclined wall, which may result in a small coverage of the three positioning devices and the three positioning devices cannot locate any device in the home scene. During the installation of the positioning devices, it can be ensured that the positioning device 1, the positioning device 2 and the positioning device 3 are installed on vertical walls. Figure 6 Figure (h) in the above (1) illustrates that the positioning device 1, the positioning device 2 and the positioning device 3 are installed on vertical walls.

[0193] (5) Uniform installation rule of the positioning devices.

[0194] Specifically, during the installation of the positioning devices, it is ensured that the three positioning devices are installed on different walls, so as to avoid installing two or more positioning devices on the same wall.

[0195] For example, Figure 6 Figure (i) in the above (1) illustrates an incorrect installation mode, where the positioning device 1 and the positioning device 2 are installed on the same wall, which may result in a small coverage of the three positioning devices and the three positioning devices cannot locate any device in the home scene. During the installation of the positioning devices, it can be ensured that the positioning device 1, the positioning device 2 and the positioning device 3 are installed on different walls. Figure 6 Figure (h) in the above (1) illustrates that the positioning device 1, the positioning device 2 and the positioning device 3 are installed on different walls.

[0196] (6) No obstacle (wall or object) between the positioning devices

[0197] Specifically, the obstacle-free communication rule introduced in the rule (1) ensures that there is no obstacle between any device in the home scene and each positioning device, and the devices can communicate with each other. In addition, during the installation of the positioning devices, it is also ensured that there is no obstacle between the three positioning devices, and there is no obstacle in the direction perpendicular to the plane where the ground is located and the direction parallel to the plane where the ground is located, so as to further ensure a large coverage of the three positioning devices, and the three positioning devices can locate any device in the home scene.

[0198] For example, Figure 6 Figure (j) in the above (1) illustrates an incorrect installation mode, where there is an obstacle between the positioning device 1 and the positioning device 3, and it is ensured that there is no obstacle between the three positioning devices.

[0199] The above introduces six rules for installing the positioning devices, and it should be understood that the embodiments of the present application can include more or fewer rules under the condition that the three positioning devices can locate any device in the home scene, and the embodiments of the present application do not limit this.

[0200] After the three positioning devices are installed, a connection between the handheld device and the three positioning devices needs to be established, so that the handheld device can interact with the three positioning devices, and the position of each device in the home scene and the distance between the handheld device and each device are further determined.

[0201] In a possible implementation, the connection between the handheld device and the three positioning devices can be established through a smart life application. For example, Figure 7 is a schematic diagram provided by an embodiment of the present application for establishing the connection between the handheld device and the positioning device. In the diagram, Figure 7 (a) of FIG. shows a main interface 701 displayed by the phone in the unlocked mode. The user clicks an icon of the smart life application on the main interface 701. In response to the click operation of the user, the phone displays a main interface 702 of the smart life application as shown in Figure 7 (b) of FIG. For example, Figure 7 (b) of FIG. shows that the main interface 702 of the smart life application can include a virtual experience menu and an add device menu, and function areas such as "home", "mall", "smart" and "mine".

[0202] As shown in (a) of FIG. Figure 7 , the user clicks the "add device" menu on the main interface 702 of the smart life application. In response to the click operation of the user, the phone displays an automatic scanning interface 703 as shown in Figure 7 (c) of FIG. For example, the phone can automatically scan devices or devices that can be connected around. It should be understood that during the process, the three positioning devices installed in the home scene must be in a powered-on working state and can be discovered by the phone.

[0203] Alternatively, the user can also discover the three positioning devices through "manual addition" or "code scanning addition" and the like on the main interface 702 of the smart life application. The manner in which the phone discovers the positioning device is not limited by the embodiments of the present application, and will not be described here.

[0204] When the phone scans the three positioning devices, an interface 704 as shown in Figure 7 (d) of FIG. can be displayed. On the interface 704, the names of the three positioning devices can be displayed, for example, Tag 1, Tag 2 and Tag 3. The user clicks the positioning device Tag 1. In response to the click operation of the user, the Tag 1 can be directly connected with the phone. Alternatively, the phone can display a network configuration interface 705 as shown in (e) of FIG. 7. In the network configuration interface 705, the user can input the current network name and password, and click the "next" button, and then the phone jumps to a connection success interface 706 as shown in (f) of FIG. 7.

[0205] By the above steps, the connection of the mobile phone and the positioning device 1 is realized. Similarly, the connection of the mobile phone and the positioning device 2, the positioning device 3 can be realized by repeating the processes shown in (d), (e) and (f) of FIG. 1. Figure 7 Alternatively, the mobile phone and the positioning device can also be connected through a quick way such as "bump", which will not be described herein.

[0206] When three positioning devices are arranged in the home scenario, the handheld device can determine the position of each device in the home scenario and the distance between the handheld device and each device according to the three positioning devices.

[0207] Figure 8 is a schematic diagram for determining the position of a device in a home scenario provided by an embodiment of the present application. In the diagram, a space coordinate system is established in the space where the room is located, for example, a space coordinate system is established with the length, width and height of the room as the coordinate axes, and the present application is not limited to the way of establishing the space coordinate system. It should be understood that as long as the same space coordinate system is used when calculating the position of each device in the home scenario through the three positioning devices.

[0208] The space coordinates of the positioning device 1 (x1, y1, z1), the space coordinates of the positioning device 2 (x2, y2, z2), and the space coordinates of the positioning device 3 (x3, y3, z3) are known, and the distance d1 between the device A and the positioning device 1, the distance d2 between the device A and the positioning device 2, and the distance d3 between the device A and the positioning device 3 can be measured. After obtaining the above known parameters, assume the space coordinates of any device A (x0, y0, z0) in the home scenario, and determine the values of x0, y0 and z0 according to the following calculation process.

[0209] (x1-x0) 2 +(y1-y0) 2 +(z1-z0) 2 =d1 2

[0210] (x2-x0) 2 +(y2-y0) 2 +(z2-z0) 2 =d2 2

[0211] (x3-x0) 2 +(y3-y0) 2 +(z3-z0) 2 =d3 2

[0212] After d1, d2, d3 are obtained, a circle is made with the position of the positioning device 1 as the center and d1 as the radius, a circle is made with the position of the positioning device 2 as the center and d2 as the radius, and a circle is made with the position of the positioning device 3 as the center and d3 as the radius, and the position coordinates of the intersection of the three circles are the position coordinates of the device A, that is, the position of any device in the home scene can be determined by the three positioning devices 3. The mobile phone as a handheld device can obtain the known parameters above, and determine the position of any device according to the above calculation process.

[0213] In summary, the mobile phone as a handheld device can already communicate with the three positioning devices arranged in the home scene. To realize further quick gestures based on the user's pointing gesture and use the user's handheld device to control other devices in the home scene, the other devices in the home scene need to be endowed with the ability to be controlled. The following Figure 9 introduces an implementation manner, which can endow the other devices in the home scene with the ability to be controlled by the user's handheld device after detecting the user's pointing gesture.

[0214] Figure 9 is an example of setting devices in a home scene provided by an embodiment of the present application. In (a) of Figure 9 the (a) shows the main interface 901 currently displayed by the mobile phone in the unlocked mode, and the user clicks the icon of the smart life application on the main interface 901. In response to the click operation of the user, the mobile phone displays the main interface 902 of the smart life application as shown in (b) of Figure 9 .

[0215] Optionally, the main interface 902 of the smart life application has already added the infrared remote controller, my sound box, Huawei router, my air conditioner and Huawei smart screen in the home scene. In other words, the main interface 902 of the smart life application can display the cards of multiple devices that have been paired with the handheld device.

[0216] It should be understood that the way of adding multiple electronic devices in the home scene in the smart life application can refer to the way in the prior art, such as scanning addition, manual addition, and bump-to-pair addition, and the like. Therefore, details are not described herein.

[0217] In a possible implementation manner, on the main interface 902 of the smart life application, the user selects the area where the card of a device in a certain home scene is located through a preset operation, and then points the handheld device to the device corresponding to the area where the card is located, so as to activate the device, so as to agree that the device can be controlled by the handheld device when detecting the pointing gesture of the user.

[0218] For example, as Figure 9As shown in Figure (b), on the main interface 902 of the smart living application, the user long-presses the area where the "My Speaker" card is located, and at the same time or within a certain period of time, the user uses their mobile phone to point to the location where the speaker is placed.

[0219] It should be understood that the long press operation can activate the speaker, and by pointing the phone at the location where the speaker is placed, the phone and the speaker can exchange information, so that when the phone detects the user's finger gesture, the speaker can be controlled by the phone.

[0220] It should also be understood that during this process, the user can enable information exchange between the phone and the speaker by pointing the phone at the location where the speaker is placed. It can also be understood that the phone locates the speaker through the three aforementioned positioning devices, thus determining the speaker's specific location in the home environment.

[0221] Optionally, the method of triggering speaker activation is not limited to... Figure 9 The operation of long-pressing the area where the "My Speaker" card is located and pointing the phone at the location of the speaker, as shown in (b) of the figure, can also be achieved through other preset operations or gestures such as swiping and pointing the phone at the location of the speaker. This application embodiment does not limit this.

[0222] In another possible implementation, the user can select the area where a device card is located in a home setting, enter the device's details interface, and enable the device's one-finger control function. When the user's one-finger gesture is detected, the device can be controlled by the handheld device. It should be understood that in this implementation, the user can also point the handheld device at the device corresponding to the area where the card is located, thereby activating the device. The phone then uses the three aforementioned positioning devices to locate the device, determining its specific position in the home setting, and agreeing that the device can be controlled by the handheld device when the user's one-finger gesture is detected.

[0223] Optionally, users can select the area where a device card is located in a home scene on the main interface 902 of the smart living application, or select the area where a device card is located in a home scene through a shortcut.

[0224] For example, such as Figure 9 As shown in Figure (c), on the main interface 903 of the mobile phone, when the user performs a pull-down operation as indicated by the black arrow, the phone displays the following in response to the user's pull-down operation: Figure 9 The control center interface 904 shown in Figure (d) can display the My Speaker and Huawei Smart Screen cards in the added home scenes.

[0225] The user clicks the area where the Huawei Smart Screen card is located, and the mobile phone can enter the detail interface 905 of the Huawei Smart Screen as shown in (e) of FIG. 10. Figure 9 In the detail interface 905 of the Huawei Smart Screen, a "point-to-control function" switch can be added. When the user clicks to turn on the "point-to-control function" switch, the switch is in an open state, the Huawei Smart Screen can be activated, the mobile phone locates the Huawei Smart Screen through the three positioning devices connected as described above, determines the specific position of the Huawei Smart Screen in the home scene, and agrees that the Huawei Smart Screen can be controlled by the mobile phone when the mobile phone detects the point-to-control gesture of the user.

[0226] In another possible implementation, when the user activates a device in the home scene through the handheld device, the user can be prompted with prompt information indicating that the device has been successfully activated.

[0227] Optionally, when the mobile phone completes the positioning of a device in the home scene, the prompt information can be displayed in a pop-up window or the like on a specific application or interface, informing the user that the device has been successfully activated.

[0228] Optionally, if the mobile phone activates a device and finds that the distance between the device and the mobile phone is very close, the user can be prompted to "move the mobile phone or the XX device away, as the distance between the mobile phone and the positioned XX device is close". Alternatively, multiple possible devices can be displayed, and the user can further select the currently positioned device in actual operation, which is not limited in the embodiments of the present application.

[0229] Similarly, by repeating the above operations, the mobile phone can position and activate any other device such as a smart screen in the home scene. Details are not repeated here.

[0230] In combination with the multiple possible methods introduced in Figure 9 , the method can be performed in a specific application or interface (such as smart life), or the control center interface of the handheld device, or the device detail interface of a device, and the like, through a preset operation (long press, swipe, or the like) or clicking a preset switch, so that the other device in the home scene that has been paired with the handheld device and is currently being pointed to by the handheld device of the user enters an activated state, agreeing that the device can be controlled by the handheld device when the handheld device detects the point-to-control gesture of the user.

[0231] In summary, through the introduction of Figures 5 to 9 , at least three positioning devices have been deployed in the current home scene, the user has turned on the point-to-control function of the mobile phone to control other devices, and multiple devices in the home scene have been activated, agreeing that the device pointed to by the user can be controlled by the mobile phone when the mobile phone detects the point-to-control gesture of the user. The followingFigures 10 to 19 This document details how users can control other devices in their home environment with a simple tap.

[0232] Figure 10 This is a schematic diagram illustrating a user pointing at a speaker, provided in an embodiment of this application. Figure 10 The process in Figures (a) to (c) illustrates one possible pointing operation process for the user.

[0233] For example, a user can use, as Figure 10 The hand gesture shown in Figure (a) is used to hold the phone, and via as shown in Figure (a) Figure 10 The posture shown in Figure (b) ultimately results in the following: Figure 10 The posture shown in Figure (c) points towards the speaker in the home scene. The phone can detect the displacement process of the user holding the phone, as shown in the figure, ultimately pointing towards the speaker's location.

[0234] In one possible implementation, the mobile phone, as the user's handheld device, can recognize the user's pointing gesture using one or more sensors on the phone, or the phone's camera, etc. In the following description of embodiments, the user's pointing gesture can be referred to as a "point-and-click operation".

[0235] Optionally, the mobile phone's sensors may include an accelerometer, gyroscope, magnetometer (M-sensor), inertial measurement unit (IMU), etc. The mobile phone's camera is not limited to a front-facing or rear-facing camera; or, the mobile phone's camera is not limited to one or more of a main camera, wide-angle camera, and telephoto camera. The user's pointing operation is recognized based on the image captured by the camera; this application embodiment does not limit this.

[0236] In the process of determining whether the user has pointed at the speaker, establish such as Figure 10 The coordinate system O-XYZ is shown in Figure (d). Specifically, this coordinate system O-XYZ has the phone's center of gravity as the origin O, the line passing through the origin O along the long edge of the phone as the Y-axis, the line along the short edge of the phone as the X-axis, and the line perpendicular to the ground as the Z-axis. Therefore, when the user performs a pointing operation, the movement of the coordinate system O-XYZ can be described as follows: Figure 10 As shown in Figures (b) and (c) above.

[0237] For example, the user's operation of pointing at the speaker may include the following action change process:

[0238] (1) The user's starting position can be as follows: Figure 10Figure (a) shows the user's hand holding the phone in a naturally extended position, or as shown in Figure (a). Figure 10 Figure (b) shows the user holding the phone naturally in front of their chest.

[0239] In other words, the user's pointing operation can be from (a) graph directly to (c) graph, including different trajectories; or from (b) graph directly to (c) graph; or via the trajectory of (a) graph - (b) graph - (c) graph. This application embodiment does not limit this.

[0240] (2) When the user extends their arm forward, the phone accelerates along the line containing the Y-axis. The phone's accelerometer detects that the acceleration along the Y-axis is greater than or equal to a first preset threshold a. A The acceleration change curve can be shown as follows: Figure 10 As shown in Figure (e), at time T2, the acceleration of the mobile phone along the line containing the Y-axis is greater than or equal to the first preset threshold a. A .

[0241] Simultaneously, it was detected that the acceleration of the phone along the X-axis and the Z-axis was less than or equal to a second preset threshold a. B .

[0242] Optionally, the acceleration of the mobile phone in different directions can be detected by the phone's IMU. Alternatively, the acceleration of the mobile phone in different directions can be detected based on the image captured by the phone's front-facing camera and a constantly open optical flow algorithm. Specifically, the front-facing camera captures adjacent frames in the image, where the two-dimensional vector field of feature points in adjacent frames during translation represents the velocity field of the object point's three-dimensional motion through a two-dimensional image. That is, based on the velocity field of the object point's three-dimensional motion, reflecting the image changes caused by the object point's motion within a certain time interval, the direction and speed of the object point's motion in the image can be determined, thereby determining whether the mobile phone's trajectory is a straight line or a near-straight line.

[0243] In one possible implementation, when the handheld device detects the movement of the phone along the straight line containing the Y-axis, and the acceleration value is greater than or equal to a first preset threshold a... A At times, such as Figure 10 As shown in Figure (e), the first prompt message can appear on the handheld device at time T2, guiding the user to continue accelerating along the Y-axis.

[0244] (3) The acceleration time of the mobile phone along the straight line containing the Y-axis, detected by the mobile phone's IMU, reaches the first preset duration t. a .

[0245] In one possible implementation, when the handheld device detects the phone's motion along the Y-axis, the acceleration value is greater than or equal to a first preset threshold a.A and the acceleration time of the acceleration reaches a first preset time length t a When the acceleration time reaches the first preset time length t, the handheld device can display a second prompt information to prompt the user that the handheld device has started the wireless positioning function.

[0246] (4) The mobile phone detects that the mobile phone is away from the user's body.

[0247] Optionally, the mobile phone can detect that the mobile phone is away from the user's body or close to the user's body through a magnetometer. Since the human body itself is a magnetic field, when the mobile phone is away from the human body or close to the human body, the magnetic field strength detected by the magnetometer will change obviously. Therefore, the state of the mobile phone being away from the user's body or close to the user's body can be determined according to the magnetic field strength detected by the magnetometer.

[0248] Alternatively, the mobile phone can detect that the mobile phone is away from the user's body or close to the user's body through images obtained by a camera. Specifically, the camera of the mobile phone can obtain images in real time, and perform face feature detection according to the obtained images. Within a certain period of time, the mobile phone first detects face information in the obtained images, and determines that the mobile phone is close to the user's body; then, the mobile phone does not detect face information in the obtained images, and determines that the mobile phone is away from the user's body. When the above change process occurs, it can be determined that the mobile phone gradually changes from being close to the user's body to being away from the user's body.

[0249] (5) The IMU of the mobile phone detects that the mobile phone stops accelerating, and the hovering time of the mobile phone reaches a second preset time length t dwell .

[0250] When the mobile phone detects the above motion change processes (1), (2), (3), (4), and (5), it can be determined that the current user performs a pointing operation, and then triggers the mobile phone to determine the position of the device pointed by the user through a wireless positioning technology. The process can combine the data collected by multiple sensors to more accurately determine that the user performs a pointing operation. It should be understood that the mobile phone can also trigger the mobile phone to determine the position of the device pointed by the user through a wireless positioning technology when detecting one or more of the above motion change processes (1), (2), (3), (4), and (5), and the embodiments of the present application are not limited in this regard.

[0251] In another possible implementation, in order to accurately identify the pointing operation of the user and reduce the false touch rate, the embodiments of the present application can trigger the handheld device to detect the pointing operation of the user only in a preset scene. The preset scene includes: the handheld device is in a bright screen state; and / or, the handheld device is in an unlocked state; and / or, the handheld device displays a main interface; and / or, a certain angle is detected between the normal direction of the display screen of the handheld device and the ground. The certain angle between the normal direction of the display screen of the handheld device and the ground can be understood as the normal direction of the display screen not being perpendicular to the ground.

[0252] In another possible implementation, when the IMU of the mobile phone detects that the acceleration of the mobile phone stops, and the hovering time of the mobile phone reaches a second preset time length t dwell When the handheld device displays the third prompt information, the third prompt information prompts the target device pointed by the user. When at least two devices are detected in the direction pointed by the user, the third prompt information can display information of the at least two devices for the user to select the target device from the at least two devices.

[0253] Alternatively, at this time, a fourth prompt information can also be displayed on the pointed device, for prompting the user to perform a correct pointing operation to point to the device. For example, if the user points to the smart screen through the mobile phone, since the user gazes at the smart screen during the pointing process, a pop-up window including the fourth prompt information can be displayed on the smart screen, for indicating the correct pointing operation required by the user to control the smart screen.

[0254] The above-mentioned visual and non-visual prompt manners guide and feedback the pointing operation on the handheld device and the target device, thereby improving the success rate of the pointing operation and improving the user experience.

[0255] The above-mentioned embodiments introduce possible cases of displaying prompt information on the handheld device or the target device. It should be understood that, during the pointing process of the target device, the user can have a hand-eye separation phenomenon, i.e., the user cannot see the interface of the mobile phone. Therefore, the embodiments of the present application do not limit the prompt manner of any one of the above-mentioned prompt information (the first prompt information, the second prompt information, the third prompt information, and the fourth prompt information), for example, the prompt manner can be a visual or non-visual prompt recognizable by the user, such as different interface prompts, vibrations, indicator lights, voices, and various prompt manners.

[0256] Alternatively, the prompt effect of the prompt information on the handheld device can be matched with the direction of the pointing action of the user, the acceleration of the handheld device, and the like.

[0257] Through the above-mentioned preset scene setting, the mobile phone can recognize whether the user performs the pointing operation only in a preset scene or in multiple scenes at the same time, for example, the mobile phone can recognize whether the user performs the pointing operation only in a bright screen state and / or an unlocked state, or only when the mobile phone displays a home interface, or only when the mobile phone meets the conditions of the bright screen state, the unlocked state, and the display of the home interface at the same time. The preset scene setting can increase the accuracy of recognizing the pointing operation of the user, avoid detection in possible scenes such as the user only handing over the mobile phone to another person, thereby reducing the false touch rate and improving the user experience.

[0258] In another possible implementation, when the mobile phone determines that the user is currently performing a pointing operation on the sound box, the user can press a physical button, or perform a preset screen gesture, or an air gesture, or the like, to send a signal to the mobile phone to determine the detection result currently detected by the mobile phone. Figure 10 dwell

[0259] For example, when the user performs the operation shown in (c) of FIG. 6 and the mobile phone hovers for a second preset time t Figure 10 dwell For example, when the user performs the operation shown in (c) of FIG. 6 and the mobile phone hovers for a second preset time t

[0260] Through the above process, the accuracy of the mobile phone in recognizing the pointing operation of the user can be improved. The process can be understood as the user sending a confirmation signal to the mobile phone through the above pressing operation or preset gesture, confirming that the action currently detected by the mobile phone is the operation of the user pointing to the sound box, and the positioning process of the mobile phone on the sound box can be triggered more accurately. In contrast, if the mobile phone detects that the user performs the operation shown in (c) of FIG. 6 and the mobile phone hovers for a second preset time t Figure 10 dwell

[0261] According to the above steps, after the mobile phone has recognized the current pointing operation of the user on the sound box, the wireless positioning function of the mobile phone is further started to search for devices around the mobile phone, determine the positions of the devices around the mobile phone, and the distances between the devices around the mobile phone and the mobile phone, and determine whether there is a device in the pointing direction of the user. For example, when the mobile phone detects the pointing operation of the user as shown in (c) of FIG. 6, the mobile phone can trigger the wireless positioning technology. Figure 10

[0262] In one possible implementation, when the target device pointed to by the user is installed with a positioning structure, the mobile phone can perform wireless positioning according to the bidirectional connection between the mobile phone and the target device. For example, the target device pointed to by the user can be installed with one or more of a Bluetooth positioning chip, a UWB positioning chip, a GPS positioning structure, a WIFI positioning structure, and a laser sensor, an infrared sensor, or the like having a positioning function, so that the target device can be positioned based on the bidirectional connection between the target device and the mobile phone.

[0263] ​​​​​​Exemplarily, if the smart screen, air conditioner, sound box and the like have UWB chips, the positioning of any one of the devices in the home scene can be realized through the interaction between the mobile phone and the smart screen, air conditioner, sound box and the like.

[0264] It should be understood that both the mobile phone and the target device to be pointed have hardware capable of transmitting or receiving wireless positioning signals, and the distance and angle between the mobile phone and the target device to be pointed are calculated and determined according to the transmission or reception of wireless positioning signals between the mobile phone and the target device to be pointed.

[0265] In another possible implementation, when three positioning devices are arranged in the home scene, the mobile phone can determine the distance and direction between the mobile phone and the target device to be pointed in combination with the three-point positioning capability of the three positioning devices.

[0266] Exemplarily, according to the method shown in Figure 6 , after three positioning devices are installed in the home scene, the connection between the handheld device and the three positioning devices is established according to the method shown in Figure 7 , so that the handheld device can interact with the three positioning devices, and the position of each device in the home scene and the distance between the handheld device and each device are further determined according to the three-point positioning method introduced in Figure 8 . Details are not described herein again.

[0267] In summary, the mobile phone can determine the distance and angle between the mobile phone and the target device to be pointed by triangulation among multiple devices. It should be understood that the mobile phone can transmit or receive wireless positioning signals, as long as the space of the home scene includes three or more fixed positioning devices capable of transmitting or receiving wireless positioning signals, such as Tag 1, Tag 2 and Tag 3 shown in (d) of Figure 10 . Further, the mobile phone calculates the absolute positioning (x, y, z, ) of the mobile phone by time difference of arrival (TDOA). That is, the position of each device in the home scene and the distance between the handheld device and each device can be determined by the three-point positioning method.

[0268] Alternatively, when there are enough positioning devices capable of transmitting or receiving wireless positioning signals in the home scene, the six degrees of freedom (6DoF) positioning of the mobile phone can be determined, and details are not described herein again.

[0269] It should be understood that the positioning device can exist independently or be installed on other devices in the home scene, and be a component of the smart screen, air conditioner, sound box and the like, and the embodiments of the present application do not limit this.

[0270] Through the above process, the mobile phone has recognized the user's current pointing operation on the sound box, and further starts the wireless positioning function of the mobile phone, and determines the position of the sound box pointed by the user. After the above pointing operation of the user, the sound box pointed by the user can be controlled on the mobile phone.

[0271] Figure 11 is a schematic diagram of a user pointing to a sound box provided by an embodiment of the present application, Figure 12 is a schematic diagram of a user controlling a sound box through a handheld device provided by an embodiment of the present application.

[0272] Exemplarily, Figure 11 The processes of (a), (b) and (c) in FIG. 1 show a possible pointing operation process of a user. Specifically, Figure 12 (a) of FIG. 1 shows the main interface 1201 of the mobile phone. When the mobile phone is in a bright screen state and displays the main interface of the mobile phone, if the user uses the mobile phone to perform the above pointing operation on the sound box, the mobile phone can display the interface 1202 as shown in (b) of FIG. 1. Figure 12 (b) of FIG. 1. In the interface 1202, the sound box control window 20 is automatically popped up, and the sound box control window 20 can be suspended on the main interface of the mobile phone.

[0273] Optionally, the sound box control window 20 can include a window zoom option 21, a window close option 22, a sound box icon and name display area 23, and a song playing control area 24. The user can click the window zoom option 21 to display the sound box control window 20 in full screen or to suspend the display. The song playing control area 24 can display the singer, the singer's photo, the album picture, the song name, the playing switch, the playing progress bar, the previous song button, the next song button, etc. The user can click any one button according to his / her own needs to control the playing of the sound box. The song playing control area 24 also includes a sound projection button for controlling the playing of the music of the mobile phone through the sound box, which will not be described herein.

[0274] It should be understood that the sound box control window 20 can include the above-mentioned control options, menus, buttons or other functional areas, or the sound box control window 20 can also include other display contents not mentioned above, such as a song lyrics display area, etc. The sound box control window 20 can display different styles and contents according to the type and screen size of the handheld device, which will not be described herein.

[0275] Figure 13 is a schematic diagram of a user pointing to a sound box provided by an embodiment of the present application, Figure 14 is a schematic diagram of a user controlling a sound box through a handheld device provided by an embodiment of the present application.

[0276] Exemplarily,Figure 13 The processes of (a), (b) and (c) in FIG. 13 show a possible pointing operation process of the user. Specifically, Figure 14 The (a) in FIG. 13 shows the main interface 1401 of the mobile phone. When the mobile phone is in the bright screen state and displays the main interface of the mobile phone, if the user uses the mobile phone to perform the above-mentioned pointing smart screen operation, the mobile phone can display the interface 1402 as shown in the (b) of FIG. 13. Figure 14 The (b) in FIG. 13 shows the interface 1402. In the interface 1402, the smart screen control window 30 is automatically popped up, which can be suspended on the main interface of the mobile phone.

[0277] Optionally, the trajectory of the user pointing to the smart screen in this embodiment Figure 13 may be different from the trajectory of the user pointing to the sound box in Figure 11 , or the trajectories can be the same, and the embodiments of the present application do not limit this.

[0278] Optionally, the smart screen control window 30 can include a window zoom option 31, a window close option 32, a smart screen icon and name display area, and a smart screen play control area 33. The user can click the window zoom option 31 to display the smart screen control window 30 full screen or to reduce it to suspended display. The smart screen play control area 33 can display the film name, the selected list, the play switch, the play progress bar, the previous episode, the next episode and other buttons. The user can click any one button according to his own needs to control the play of the film on the smart screen.

[0279] It should be understood that the smart screen control window 30 can include the above-mentioned control options, menus, buttons or other functional areas, or the smart screen control window 30 can also include other display contents not mentioned above, and the embodiments of the present application do not limit this. The smart screen control window 30 can display different styles and contents according to the type and screen size of the handheld device, which will not be described here.

[0280] Figure 15 is a schematic diagram of the user pointing to the air conditioner provided by an embodiment of the present application, Figure 16 is a schematic diagram of the user controlling the air conditioner through the handheld device provided by an embodiment of the present application.

[0281] Exemplarily, Figure 15 The processes of (a), (b) and (c) in FIG. 13 show a possible pointing operation process of the user. Specifically, Figure 16 The (a) in FIG. 13 shows the main interface 1401 of the mobile phone. When the mobile phone is in the bright screen state and displays the main interface of the mobile phone, if the user uses the mobile phone to perform the above-mentioned pointing smart screen operation, the mobile phone can display the interface 1402 as shown in the (b) of FIG. 13. Figure 16The interface 1602 shown in (b) of FIG. 16. In the interface 1602, the air conditioner control window 40 is automatically popped up, which can be suspended on the main interface of the mobile phone.

[0282] Optionally, the air conditioner control window 40 can include a window zoom option 41, a window close option 42, an air conditioner icon and name display area 43, and an air conditioner control area 44. The user can click the window zoom option 41 to display the air conditioner control window 40 in full screen or to reduce it to a suspended display. The air conditioner control area 44 can display whether the air conditioner is on, the cooling operation mode, the indoor temperature, the working power, the temperature increase / decrease button, the wind speed setting button, and the like. The user can click any button according to his / her needs to control the start of the air conditioner and the temperature and operation mode settings.

[0283] It should be understood that the air conditioner control window 40 can include the above-mentioned control options, menus, buttons, or other functional areas, or the air conditioner control window 40 can also include other display contents not mentioned above, which are not limited in the embodiments of the present application. The air conditioner control window 40 can display different styles and contents according to the type and screen size of the handheld device, which will not be described here.

[0284] By the above method, based on the recognition function and positioning function of the handheld device, when the handheld device detects the user's pointing operation to the target device, a window for controlling the target device can be displayed on the handheld device, and the user can control the target device through the window. The method is simple to operate, reduces the steps of the user operating the target device, and further helps the user to select a device that is far away, reduces the distance limitation, and improves the user experience.

[0285] In another possible implementation, during the user's pointing operation to the target device using the handheld device, a prompt information can be displayed on the handheld device, which guides the user to perform the correct pointing operation to control the target device according to the embodiments of the present application.

[0286] Optionally, the prompt information can be displayed through the window displayed on the handheld device, or in the form of an icon on the handheld device, or in other visual guidance manners to guide the user to perform the correct pointing operation to control the target device according to the embodiments of the present application.

[0287] Figure 17 is another example of the user controlling the sound box through the handheld device according to the embodiments of the present application.

[0288] Exemplarily, Figure 17The processes illustrated in diagrams (a), (b), and (c) show one possible user-directed operation. Specifically, Figure 17 Figure (d) shows the phone's main interface 1701. When the phone screen is on and displaying the main interface 1701, if the user performs the following actions on the phone... Figure 17 The operations shown in Figures (a) and (b) indicate that the phone did not detect them for a certain period of time. Figure 17 As shown in Figure (c), the operation of pointing at the speaker and hovering, etc., can be displayed on the main interface of the phone as follows. Figure 17 The interface 1702 is shown in Figure (e). In this interface 1702, a guide icon 50 is automatically displayed, which can float on the main screen of the phone in the form of an arrow.

[0289] When the user continues to execute according to the guide icon 50, as follows Figure 17 When the pointing operation shown in Figure (c) is performed, the phone's interface can be displayed as interface 1703 as shown in Figure (f) of 17. On interface 1703, icon 60 is displayed to indicate that the phone is currently re-detecting the user's pointing operation. When such a pointing operation is detected... Figure 17 When the user points to the speaker as shown in Figure (c), the speaker control window 20 can be displayed on the handheld device, which will not be described in detail here.

[0290] In another possible implementation, during the user's operation of pointing the handheld device at the target device, in addition to Figure 17 In addition to the guide icon shown in Figure (e), non-visual methods such as sound feedback and vibration feedback can be used to guide the user to perform the correct pointing operation, so as to realize the target device control by pointing operation provided in the embodiments of this application.

[0291] Optionally, the sound feedback can be emitted by a handheld device or by the target device being pointed at, such as a mobile phone prompting the user to "move to the right" via voice.

[0292] Optionally, the vibration feedback can be emitted by a handheld device so that it can be perceived by the user more quickly and sensitively. It should be understood that the embodiments of this application do not limit this.

[0293] Alternatively, after a user successfully points their handheld device at a target device, feedback can be provided on either the handheld device or the target device to inform the user that the pointing operation was successful. This feedback can include one or more of the following: visual feedback, audio feedback, vibration feedback, etc. For example, ... Figure 17 As shown in Figure (f), after the user successfully points the phone at the speaker, the phone can vibrate to notify the user that the pointing operation is complete.

[0294] It should be understood that after the user successfully points to the target device using the handheld device, the target device can transmit the currently logged-in account information and device information to the handheld device, and then display the target device control window shown in the (g) view of FIG. 1 1 on the handheld device. Figure 18

[0295] It should also be understood that when the handheld device is determined to point to the target device, the axis of the handheld device is identified to be within a preset range of the position of the target device, that is, the target device pointed to by the user is determined. Specifically, when the axis of the handheld device is identified to substantially intersect the physical position of the target device with a predetermined accuracy, or when the handheld device is identified to be aligned with the target device, the target device pointed to by the user is determined, that is, feedback can be provided on the handheld device or the target device pointed to, and the logged-in account information and device information of the target device are transmitted to the handheld device, and the target device control window 20 or the interface related to the target device is displayed on the handheld device.

[0296] In summary, the method for implementing cross-device interaction based on pointing operation provided by the embodiments of the present application can be used for the user to point to the target device using the handheld device, and the pointing operation of the user is detected based on one or more sensors such as the acceleration sensor, the gyroscope, the IMU, and / or the camera of the handheld device, and the wireless positioning function of the handheld device is triggered. The handheld device determines the distance between the handheld device and other devices and the position and direction of the other devices according to the wireless positioning function. When the axis of the handheld device is identified to intersect or substantially intersect the physical position of the target device with a predetermined accuracy, at least one feedback such as visual feedback, sound feedback, vibration, and the like can be provided on the handheld device and / or the target device, and the logged-in account information and device information of the target device are transmitted to the handheld device, and the control window of the target device is displayed on the handheld device, so that the user can control the target device through the control window on the handheld device. When the axis of the handheld device is identified to not intersect the physical position of the target device, at least one guiding mode such as visual guidance, sound guidance, vibration, and the like can be provided on the handheld device and / or the target device to guide the user to perform the correct pointing operation, and the control window of the target device is further displayed on the handheld device and the control function of the target device is implemented.

[0297] ​In another possible implementation, when the user performs the action of pointing to the target device, there can be two or more devices in the same location that are relatively close to each other. In other words, the axis of the y-axis of the mobile phone during the movement of the mobile phone can be located between the two devices, and the accuracy between the axis and the physical locations of the two devices is within a predetermined accuracy. When the axis of the y-axis of the mobile phone is determined to intersect or substantially intersect with device 1 or device 2 at the same time, that is, the pointing operation of the user holding the device can be identified as pointing to device 1 or device 2. For this scenario, the embodiments of the present application further provide a method, which can provide a selection window for the user to select the target device during the detection of the pointing operation of the user.

[0298] Figure 18 Fig. 1 is a schematic diagram of another example of a user controlling a sound box by using a handheld device according to an embodiment of the present application. As shown in Fig. 1, Figure 18 Fig. 1(a) shows the main interface 1801 of the mobile phone. When the mobile phone is in a bright screen state and displays the main interface 1801 of the mobile phone, the mobile phone can display, for example, Figure 18 Fig. 1(b) shows the interface 1802 on which a guide icon 50 is displayed to guide the user to perform a correct pointing operation.

[0299] If the user uses the mobile phone to point to the air conditioner in the living room, since the air conditioner and the sound box are located close to each other, when the mobile phone detects the pointing operation of the user, it is determined that the pointing operation can trigger the air conditioner to be controlled by the mobile phone and can also trigger the sound box to be controlled by the mobile phone.

[0300] In this scenario, the mobile phone can further display, for example, the interface 1803 shown in Fig. 1(c), which simultaneously displays the sound box card 70 corresponding to the sound box that the user currently points to and the air conditioner card 80 corresponding to the air conditioner. Figure 18 Optionally, the sound box card 70 can only display part of the contents such as the device name, icon, zoom option, and window closing option; similarly, the air conditioner card 80 can only display part of the contents such as the device name, icon, zoom option, and window closing option, and the embodiments of the present application are not limited in this regard.

[0301] The user can perform, for example, the operation shown in Fig. 1(c), click any area of the sound box card 70 that the user currently desires to operate or control, and in response to the click operation of the user, the mobile phone displays, for example, the interface 1804 shown in Fig. 1(d). Figure 18 Figures 5 to 18 The interface 1804 displays the sound box control window 20, which will not be described herein again.

[0302] ​The above introduces the implementation process of the mobile phone identifying that the user points to two devices. When the mobile phone identifies that the user points to more devices, the method can also be used to display cards of the multiple devices for the user to select at least one target device according to the user's own needs. In addition, the user can also click to select multiple devices as target devices at the same time. Then, multiple device control windows can be displayed on the interface 1804, and the user can replace different device control windows through sliding switching. Details are not described herein.

[0303] Through the above process, for multiple devices close to the position, the handheld device can identify multiple devices pointed by the user, or the user expects to control the multiple devices close to the position. The cards of the multiple devices can be displayed for the user to select at least one target device according to the user's own needs. The design is more humanized, can better meet different needs of the user, and improves the user experience.

[0304] The above embodiments combine Figure 19 to introduce the interaction method between devices based on pointing from the user interaction level. Next, the interaction method between devices provided by the embodiments of the present application will be introduced from the software implementation strategy level. Figure 1 It should be understood that the method can be implemented in an electronic device (such as a mobile phone, a tablet computer, etc.) with a touch screen, a positioning chip, and a camera assembly, as shown in Figure 2 , Figure 19 .

[0305] Figure 19 is a schematic flowchart of an example of the interaction method between devices based on pointing provided by the embodiments of the present application. Taking a mobile phone as a handheld device, as shown in Figure 5 , the method can include the following steps:

[0306] 1901, the user starts the one-finger control function of the handheld device.

[0307] For example, the user can start the one-finger control function of the mobile phone by starting the one-finger control function switch in the settings application according to the method introduced in Figure 6 .

[0308] Alternatively, the one-finger control function switch can be added in other applications, for example, the one-finger control function switch can be added in the smart life application. The user can start the one-finger control function of the mobile phone through the one-finger control function switch of the smart life application.

[0309] Alternatively, in addition to enabling the quick interaction function between devices in the settings application and smart life application installed on the mobile phone, the "point-to-point control function" switch can be added in the notification bar of the mobile phone. For example, the user can perform a downward pull operation from the top of the mobile phone on any interface of the mobile phone, and the mobile phone displays a notification interface in response to the downward pull operation of the user. The user can click the "point-to-point control function" switch in the notification interface to enable the quick gesture interaction function between devices on the mobile phone. The embodiments of the present application do not limit the manner of enabling the quick interaction function between devices.

[0310] It should be understood that the mobile phone as a handheld device of the user also needs to realize positioning of the smart TV (smart screen), air conditioner, sound box, router and other devices in the home scene, that is, the mobile phone needs to determine the position of each device, and then more accurately control through the quick gesture.

[0311] In a possible implementation, at least three fixed positioning devices are installed in the home scene, and the three positioning devices are ensured to be in a powered-on working state, and the at least three positioning devices can communicate with the handheld device of the user.

[0312] Optionally, the at least three positioning devices can be installed according to the method and rules introduced in Figure 7 The mobile phone can establish a connection with the positioning devices according to the method shown in Figure 8 , that is, the mobile phone can send signals to the positioning devices or receive signals sent by the positioning devices. The mobile phone can determine the position of any device in the home scene in combination with the at least three positioning devices according to the process introduced in Figure 10 .

[0313] Optionally, the positioning device can be a sensor with positioning function or a structure with positioning function, for example, the sensor can be a laser sensor, an infrared sensor, etc.; the structure with positioning function can be a chip. For example, the structure with positioning function can be a positioning structure based on a Bluetooth module, a positioning chip based on ultra-wideband (UWB) wireless sensing capability, a positioning structure based on GPS, a positioning structure based on a WIFI module, etc.

[0314] Optionally, in addition to pre-arranging the above-mentioned positioning devices in the home scene, the positioning devices can also be installed on the smart TV (smart screen), air conditioner, sound box, router and other devices in the home scene. For example, if the smart screen, air conditioner, sound box and other devices themselves have UWB chips, at least three fixed positioning devices do not need to be additionally arranged in the home scene, and the positioning function of any device in the home scene can be realized through interaction between the mobile phone and the smart screen, air conditioner, sound box and other devices.

[0315] 1902, detecting the pointing operation of the user.

[0316] For example, in one possible implementation, the mobile phone as the handheld device of the user can recognize the pointing operation of the user by means of one or more sensors of the mobile phone or a camera of the mobile phone.

[0317] Optionally, the sensors of the mobile phone can include an acceleration sensor, a gyroscope, a magnetometer, an inertial detection unit (IMU), etc. The camera of the mobile phone is not limited to the front camera or the rear camera of the mobile phone; or, the camera of the mobile phone is not limited to one or more of the main camera, the wide-angle camera, and the telephoto camera of the mobile phone, and the pointing operation of the user is recognized according to the picture acquired by the camera, which is not limited in the embodiments of the present application. For the specific detection process, reference can be made to the specific description of the operation process of the user pointing at the sound box in the foregoing Figure 17

[0318] 1903, there is another device in the pointing direction of the user.

[0319] 1904, when no other device is detected in the pointing direction, the user is guided and fed back on the handheld device and the target device pointed at by the handheld device through visual and / or non-visual prompting on the handheld device.

[0320] It should be understood that, during the process of the user pointing at the target device, the hand-eye separation phenomenon may occur, i.e., the user's eyes cannot see the interface of the mobile phone. Therefore, the embodiments of the present application do not limit the prompting manner of the above-mentioned any one of the prompting information (the foregoing first, second, third, and fourth prompting information, etc.), which can be, for example, visual or non-visual prompting recognizable by the user, such as different interface prompts, vibrations, indicator lights, voices, and various other prompting manners.

[0321] In one possible implementation, during the process of the user pointing at the target device, as shown in (e) of FIG. 13, the mobile phone can automatically display a guide icon on the interface, which can be in the form of an arrow floating on the interface of the mobile phone. Figure 10

[0322] In another possible implementation, during the operation process of the user pointing at the target device by using the handheld device, in addition to the guide icon, the user can be guided to perform the correct pointing operation through non-visual manners such as sound feedback and vibration feedback, so as to realize the control of the target device by the pointing operation provided in the embodiments of the present application.

[0323] Optionally, the sound feedback can be issued by the handheld device or the target device pointed at, for example, the mobile phone prompts the user in the form of voice to "please move to the right".

[0324] ​​Optionally, the vibration feedback can be emitted by the handheld device to be perceived more quickly and sensitively by the user. It should be understood that the embodiments of the present application do not limit this.

[0325] 1905, when detecting the presence of other devices in the pointing direction, the handheld device determines the distance and direction of the handheld device and the other devices according to the wireless positioning technology or the three-point positioning technology between multiple devices.

[0326] For example, after the mobile phone identifies the user's operation of pointing to the target device, the mobile phone further starts the wireless positioning function of the mobile phone, searches for the surrounding devices of the mobile phone, and determines the positions of the surrounding devices and the distances between the surrounding devices and the mobile phone, etc., to determine whether there are devices in the pointing direction of the user. For example, as shown in (c) of FIG. 1, when the mobile phone detects the pointing operation of the user as shown in the figure, the mobile phone can trigger the wireless positioning technology of the mobile phone. Figure 17

[0327] In a possible implementation, when the target device pointed by the user is installed with a positioning structure, the mobile phone can perform wireless positioning according to the bidirectional connection between the mobile phone and the target device. For example, the target device pointed by the user can be installed with a Bluetooth positioning chip, a UWB positioning chip, or one or more of a GPS positioning structure, a WIFI positioning structure, and a laser sensor, an infrared sensor, etc. with a positioning function, so that the target device can be positioned based on the bidirectional connection between the target device and the mobile phone.

[0328] For example, if the smart screen, air conditioner, sound box, etc. have a UWB chip, the positioning of any device in the home scene can be realized through the interaction between the mobile phone and the smart screen, air conditioner, sound box, etc. in the home scene.

[0329] It should be understood that the mobile phone and the target device pointed by the user both have hardware capable of emitting or receiving wireless positioning signals, and the distance and angle between the mobile phone and the target device pointed by the user are calculated and determined according to the emission or reception of wireless positioning signals between the mobile phone and the target device pointed by the user.

[0330] In another possible implementation, when three positioning devices are arranged in the home scene, the mobile phone can determine the distance and direction between the mobile phone and the target device pointed by the user in combination with the three-point positioning capability of the three positioning devices.

[0331] 1906, it is determined whether the axis of the handheld device substantially intersects the physical position of the target device with a preset accuracy.

[0332] ​When the handheld device is determined to be pointed at the target device, the axis of the handheld device and the location of the target device are identified to be within a preset range, and the target device pointed at by the user is determined. Specifically, when the axis of the handheld device is identified to substantially intersect the physical location of the target device with a predetermined accuracy, or when the handheld device is identified to be aligned with the target device, the target device pointed at by the user is determined, that is, feedback can be provided on the handheld device or the target device pointed at, and the logged account information and device information of the target device are transmitted to the handheld device, and a target device control window 20 or an interface related to the target device is displayed on the handheld device.

[0333] Optionally, the pointing operation of the user can be described as a pointing operation in which the user holds the mobile phone and moves from a first position to a second position, and at the second position, the axis along the long side of the mobile phone intersects or approximately intersects one or more other devices. Here, intersection can be understood as the presence of other devices in the direction pointed at by the user, and approximate intersection can be understood as the presence of other devices within a preset accuracy range in the direction pointed at by the user, and both cases can be understood as detecting the pointing operation of the user.

[0334] It should be understood that the axis of the mobile phone is related to the antenna layout of the mobile phone. When the antenna layout is at the front frame of the mobile phone, the user can point the axis along the long side of the mobile phone at the target device, or when the antenna layout is inside the shell of the mobile phone, the user can also stand the mobile phone up in a similar posture to taking a photo, and point the axis along the normal line perpendicular to the display screen of the mobile phone at the target device, and the embodiments of the present application do not limit this.

[0335] 1907, feedback is provided on the handheld device, such as visual feedback, sound feedback, vibration feedback, etc.

[0336] It should be understood that the target device pointed at by the user is determined according to step 1906, that is, the user successfully points at the target device using the handheld device. At this time, feedback can also be provided on the handheld device or the target device pointed at, and the feedback is used to inform the user that the pointing operation is successful. The feedback information can include one or more of visual feedback, sound feedback, vibration feedback, etc.

[0337] For example, as shown in (f) of FIG. 1 of the prior art, Figure 17 When the user successfully points at the sound box using the mobile phone, the mobile phone can vibrate to inform the user that the pointing operation is complete, as shown in (f) of FIG. 1 of the prior art.

[0338] 1908, the logged account information and device information of the target device are transmitted to the handheld device, and an interface related to the pointed device is displayed on the handheld device.

[0339] When the user successfully points to the target device using the handheld device, the target device can transmit the currently logged-in account information and device information to the handheld device, and then display a target device control window as shown in the (g) view of FIG. 10 on the handheld device. Figure 13

[0340] 1909, end.

[0341] In summary, the method for implementing cross-device interaction based on pointing operation provided in the embodiments of the present application can be used for a user to point to a target device using a handheld device, and the pointing operation of the user is detected based on one or more sensors such as an acceleration sensor, a gyroscope, an IMU, and / or a camera of the handheld device, and a wireless positioning function of the handheld device is triggered. The handheld device determines the distance between the handheld device and other devices and the position and direction of the other devices according to the wireless positioning function. When it is identified that the axis of the handheld device intersects or substantially intersects with the physical position of the target device with a predetermined accuracy, at least one feedback such as visual feedback, sound feedback, and vibration can be provided on the handheld device and / or the target device, and the logged-in account information and device information of the target device are transmitted to the handheld device, and a control window of the target device is displayed on the handheld device. The user can control the target device through the control window on the handheld device. When it is identified that the axis of the handheld device does not intersect with the physical position of the target device, at least one guiding mode such as visual guidance, sound guidance, and vibration can be provided on the handheld device and / or the target device to guide the user to perform a correct pointing operation, and the control window of the target device is further displayed on the handheld device and the control function of the target device is implemented.

[0342] It can be understood that, in order to implement the above functions, the electronic device comprises hardware and / or software modules corresponding to the respective functions. The algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0343] The embodiments can divide the functional modules of the electronic device according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated in one processing module. The integrated module can be implemented in the form of hardware. It should be noted that the division of modules in the embodiments is illustrative, and is only a logical functional division. Actual implementation can have another division manner.

[0344] ​In the case of dividing each functional module according to each function, Figure 13 A possible component diagram of the electronic device 1300 involved in the above embodiments is shown as follows. Figure 1 As shown, the electronic device 1300 can include a display unit 1301, a detection unit 1302, and a processing unit 1303.

[0345] The display unit 1301 can be configured to support the electronic device 1300 to perform the above steps 1201 and 1204, etc., and / or other processes of the technology described herein.

[0346] The detection unit 1302 can be configured to support the electronic device 1300 to perform the above step 1203, etc., and / or other processes of the technology described herein.

[0347] The processing unit 1303 can be configured to support the electronic device 1300 to perform the above step 1202, etc., and / or other processes of the technology described herein.

[0348] It should be noted that all related content of each step involved in the above method embodiments can be referred to the function description of the corresponding functional module, which will not be repeated here.

[0349] The electronic device provided by the embodiment can be used to execute the above method for playing a video, and thus the same effects as the above method can be achieved.

[0350] In the case of using an integrated unit, the electronic device can include a processing module, a storage module, and a communication module. The processing module can be configured to control and manage the actions of the electronic device, for example, to support the electronic device to perform the steps performed by the display unit 1301, the detection unit 1302, and the processing unit 1303. The storage module can be configured to support the electronic device to store program codes and data, etc. The communication module can be configured to support the electronic device to communicate with other devices.

[0351] The processing module can be a processor or a controller. It can implement or execute various exemplary logical blocks, modules, and circuits described in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can be a device for interacting with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0352] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in the embodiment can be a mobile phone, a computer, a server, a network device, a personal digital assistant, an electronic diary, a multimedia device, a consumer electronic device, a terminal device, a peripheral device, or any device with similar properties.​ The device of the illustrated structure.

[0353] The embodiment further provides a computer readable storage medium, which stores computer instructions, and when the computer instructions are run on an electronic device, the electronic device executes the related method steps to realize the interaction method between devices based on pointing in the above embodiment.

[0354] The embodiment further provides a computer program product, which, when run on a computer, causes the computer to execute the related steps to realize the interaction method between devices based on pointing in the above embodiment.

[0355] In addition, the embodiment of the present application further provides an apparatus, which can be a chip, a component or a module, and the apparatus can include a processor and a memory connected to each other; and the memory is used to store computer execution instructions, and when the apparatus is running, the processor can execute the computer execution instructions stored in the memory to enable the chip to execute the interaction method between devices based on pointing in the above method embodiments.

[0356] The electronic device, the computer readable storage medium, the computer program product or the chip provided by the embodiment are used to execute the corresponding method provided above, and thus the beneficial effects achieved thereby can refer to the beneficial effects of the corresponding method provided above, which will not be described herein again.

[0357] Through the above description of the implementation mode, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above.

[0358] In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiment described above is only illustrative, and for example, the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0359] The units described as separate components may or may not be physically separate, and the components displayed as units may be a physical unit or multiple physical units, that is, may be located in one place, or also can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0360] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0361] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical scheme of the embodiments of the present application essentially or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0362] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of interaction between devices, characterized in that, The method is applied to a first electronic device, and the method comprises: obtaining acceleration of the first electronic device in a first axis direction, acceleration of the first electronic device in a second axis direction, and acceleration of the first electronic device in a third axis direction, wherein the first axis is parallel to a long side of the first electronic device, the second axis is parallel to a short side of the first electronic device, and the third axis is perpendicular to a plane determined by the first axis and the second axis; determining that the first electronic device performs a pointing motion from a first position to a second position, wherein determining that the first electronic device performs the pointing motion comprises one or more of: the acceleration in the first axis direction is greater than or equal to a first threshold value or the acceleration in the first axis direction is greater than or equal to the first threshold value for a duration greater than or equal to a first preset duration; the acceleration in the second axis direction is less than or equal to a second threshold value; the acceleration in the third axis direction is less than or equal to a third threshold value; the first electronic device is away from a user's body; and a hovering time when the first electronic device stops accelerating reaches a second preset duration; determining that a target device of the pointing motion of the first electronic device is at least one second electronic device, comprising: detecting that, at the second position, the first axis of the first electronic device intersects with the at least one second electronic device or the first axis of the first electronic device approximately intersects with the at least one second electronic device within a preset accuracy range; receiving account information and device information of the at least one second electronic device sent by the at least one second electronic device; displaying a second window, wherein the second window comprises information of the at least one second electronic device; in response to a first operation in the second window, displaying a first window, wherein the first window displays an interface for controlling the at least one second electronic device.

2. The method of claim 1, wherein, The method comprises: obtaining the acceleration of the first electronic device in the first axis direction, the acceleration of the first electronic device in the second axis direction, and the acceleration of the first electronic device in the third axis direction based on one or more of an acceleration sensor, an inertial detection unit (IMU), and a gyroscope.

3. The method according to claim 1 or 2, characterized in that, Before detecting that, at the second position, the first axis of the first electronic device intersects with the at least one second electronic device or the first axis of the first electronic device approximately intersects with the at least one second electronic device within a preset accuracy range, the method further comprises: capturing a picture in a motion process by a camera of the first electronic device; determining that the picture captured by the first electronic device at the first position comprises facial feature information of a user, and the picture captured by the first electronic device at the second position does not comprise the facial feature information of the user.

4. The method according to claim 1 or 2, characterized in that, The detection that the first axis of the first electronic device intersects with at least one second electronic device at the second position or that the first axis of the first electronic device approximately intersects with the at least one second electronic device within a preset accuracy range, the method further comprises: Collecting a magnetic induction intensity in a motion process by a magnetometer of the first electronic device; Determining that the magnetic induction intensity collected by the first electronic device at the first position is greater than or equal to a fourth threshold value, the magnetic induction intensity collected at the second position is less than or equal to a fifth threshold value, and the fourth threshold value is greater than the fifth threshold value.

5. The method according to claim 1 or 2, characterized in that, When the acceleration in the direction of the first axis is greater than or equal to the first threshold value, the method further comprises: Displaying first prompt information for guiding a user to continue accelerating in the direction of the first axis, the first prompt information comprising one or more of text, icons, sound, and vibration.

6. The method of claim 1 or 2, wherein, When the duration that the acceleration in the direction of the first axis is greater than or equal to the first threshold value is greater than or equal to the first preset duration, the method further comprises: Displaying second prompt information for prompting the user that the duration of acceleration in the direction of the first axis reaches the first preset duration, the second prompt information comprising one or more of text, icons, sound, and vibration.

7. The method according to claim 1 or 2, characterized in that, When the hovering time reaches a second preset duration when the first electronic device stops accelerating, the method further comprises: Displaying third prompt information for indicating whether the first axis of the first electronic device intersects with the second electronic device at the second position, the third prompt information comprising one or more of text, icons, sound, and vibration.

8. The method of claim 1 or 2, wherein, The detection that the first axis of the first electronic device intersects with at least one second electronic device at the second position or that the first axis of the first electronic device approximately intersects with the at least one second electronic device within a preset accuracy range, the method further comprises: Displaying fourth prompt information for indicating that the first electronic device is pointing to the at least one second electronic device, the fourth prompt information comprising one or more of text, icons, sound, and vibration.

9. The method of claim 1 or 2, wherein, The detection that the first axis of the first electronic device intersects with at least one second electronic device at the second position or that the first axis of the first electronic device approximately intersects with the at least one second electronic device within a preset accuracy range, comprises: According to a positioning device, detecting that the first axis of the first electronic device intersects with the at least one second electronic device at the second position or that the first axis of the first electronic device approximately intersects with the at least one second electronic device within a preset accuracy range, The positioning device is a separately installed positioning sensor, or the positioning device is a positioning chip installed on the first electronic device and the second electronic device, and the positioning chip includes any one of a Bluetooth positioning chip, an ultra-wideband (UWB) positioning chip, and a wireless fidelity (WIFI) positioning chip.

10. The method of claim 1 or 2, wherein, The method further includes: detecting a second operation in the first window; in response to the second operation, the first electronic device sends a control instruction to the at least one second electronic device, and the control instruction is used to control a behavior of the at least one second electronic device.

11. The method of claim 1 or 2, wherein, The second threshold value is equal to the third threshold value.

12. An electronic device, comprising: including: a processor and a memory, the memory storing one or more instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1-11.

14. A computer program product comprising instructions, characterized in that, The computer program product, when executed on an electronic device, causes the electronic device to perform the method of any one of claims 1-11.

Images