Inter-device interaction activation by pointing gesture recognition

Abstract

A method and a handheld device for remotely interacting with a second device are provided. The method and apparatus identify the second device from a plurality of devices based on gestures of a user. When the user makes gestures, motion sensors that sense the motion of the gestures can generate signals that can be processed by rule-based and / or learning-based methods. The results of the processing of the signals can be used to identify the second device. To improve performance, the user can be prompted to confirm that the identified second device is the device that the user wishes to remotely control. The results of the processing of the signals can also enable the user to remotely interact with the second device.

Description

[0001] Cross-referencing related applications

[0002] This is the first application filed for this invention. Technical Field

[0003] This invention relates to remote interaction with electronic devices, and more particularly to methods and apparatus for recognizing user gestures and applying these gestures to remotely interact with electronic devices. Background Technology

[0004] As more smart devices enter the consumer market, an increasing number of consumers are demanding the ability to remotely control these devices.

[0005] As handheld electronic devices (cellular phones) become increasingly popular and powerful, the demand for remotely controlling smart devices using consumers' handheld devices is growing. However, current products designed to meet this demand typically do not select the smart devices that users want to control. Therefore, there is a need for products that improve the user experience by selecting the smart devices that users would want to control remotely every time.

[0006] This background information is provided to disclose information that the applicant believes may be relevant to the present invention. No such information is permitted or should be construed as constituting prior art that conflicts with the present invention. Summary of the Invention

[0007] Embodiments of the present invention provide a system for implementing a pointing gesture recognition system (PGRS). Embodiments also provide a method for implementing an architecture to provide a PGRS that enables a user to remotely control one or more second devices by recognizing the user's gestures.

[0008] According to an embodiment of the present invention, a method for remote interaction between a handheld electronic device and a second device is provided. The method includes sensing motion of the handheld electronic device based on signals generated by one or more motion sensors of the handheld electronic device. The method further includes identifying the sensed motion as a motion-based gesture, the motion-based gesture including moving the handheld electronic device. The method further includes identifying the second device based on one or both of the signals and other signals. The other signals are from: the one or more motion sensors; one or more other components of the handheld device; or a combination thereof. Device orientation (which may be determined based on motion sensor signals, other signals, or both) and these other signals indicate the direction in which the handheld electronic device points toward the end of the motion-based gesture. After a predetermined condition is met and the second device is identified, the method initiates a user interaction for remote interaction with the second device, wherein the predetermined condition is at least partially met when the identified motion-based gesture is a predetermined motion-based gesture for interacting with the second device.

[0009] The technical advantage of these embodiments is that user interaction is initiated only after a predetermined motion-based gesture is executed. This prevents handheld electronic devices from incorrectly recognizing that the user intends to interact with a second device, which could negatively impact the user experience and unnecessarily consume battery or processing resources. Furthermore, the motion-based gesture is combined with the recognition of the second device, since the second device is based on pointing, which can be integrated with the motion-based gesture. This combination integrates the recognition of motion-based gestures and the recognition of the second device.

[0010] In some embodiments, the predetermined condition further includes recognizing confirmation input from the user. A technical advantage of these embodiments is that user interaction is only initiated after a predetermined motion-based gesture and confirmation input are performed. This further prevents false recognitions of movements corresponding to the predetermined motion-based gesture by the handheld electronic device from incorrectly identifying that the user wishes to interact with a second device.

[0011] In other embodiments, recognizing the confirmation input includes using the one or more motion sensors to recognize that the handheld electronic device remains in place after the predetermined motion-based gesture, without further movement for a predetermined period of time. The technical advantage of this embodiment is that confirmation input is automatically performed by pointing at the device without requiring further user interaction with the handheld device, which improves the user experience.

[0012] In some other embodiments, recognizing the motion-based gesture means that the predetermined motion-based gesture includes recognizing signals generated by the one or more motion sensors indicating that the handheld electronic device moves from a first position to a second position in an upward arc. The first position corresponds to the handheld electronic device being close to the user's hip and pointing downwards, and the second position corresponds to the handheld electronic device being held at the end of a straight arm and pointing towards the second device.

[0013] In other embodiments, recognizing the motion-based gesture means that the predetermined motion-based gesture includes recognizing signals generated by the one or more motion sensors indicating that the handheld electronic device moves linearly from a first position to a second position. In these embodiments, the first position corresponds to the handheld electronic device being held in front of the user's body with a bent arm, and the second position corresponds to the handheld electronic device being held at the end of a straight arm and pointing towards the second device.

[0014] In some embodiments, the second device is identified after it is determined that the motion sensed by the handheld device has stopped. The technical advantage of this embodiment is that the second device can be identified more reliably, and other devices inadvertently pointed at during motion-based gestures are prevented from being identified as the second device.

[0015] In some embodiments, the one or more motion sensors include one or more of the following: accelerometer, magnetometer, proximity sensor, gyroscope, ambient light sensor, camera, microphone, radio frequency receiver, near-field communication device, and temperature sensor. A technical advantage of these embodiments is that motion-based gestures can be recognized by sensors that directly respond to motion, sensors that directly respond to parameters indirectly related to motion or position (e.g., body proximity, radio frequency signals, sound, or temperature), or combinations thereof. Various inputs are provided that can be processed to obtain motion- or position-based information.

[0016] In some embodiments, the one or more other components of the handheld device are used to: detect the position of the one or more other electronic devices, at least in part, based on the angle of arrival measurement of the signal emitted by each of the one or more other electronic devices. A technical advantage of this embodiment is that signals such as radio frequency signals can be used to locate the second device. Therefore, for example, an antenna array system can be used to perform physical positioning.

[0017] In some embodiments, after predetermined conditions are met and the second device is identified, an icon representing the second device is displayed on the handheld electronic device. The position of the icon on the display is changed according to one or both of the following: the angle between the pointing direction of the handheld electronic device and the direction of the second device relative to the handheld electronic device, and a measurement of the probability that the handheld device is pointing at the second device. A technical advantage of this embodiment is that it provides a visual correlation between user actions and device responses, which can be used in user-involved feedback loops to facilitate the second device selection process.

[0018] According to other embodiments, a handheld electronic device is provided for performing operations commensurate with the methods described above. The device may include: one or more motion sensors for generating signals representing movement of the handheld device; and processing electronics for implementing these operations.

[0019] Embodiments have been described above in conjunction with aspects of the invention, and these embodiments can be implemented thereon. Those skilled in the art will understand that embodiments can be implemented in conjunction with the aspects described therein, but may also be implemented together with other embodiments of that aspect. It will be apparent to those skilled in the art that embodiments are mutually exclusive or incompatible with each other. Some embodiments may be described in conjunction with one aspect, but may also be applicable to other aspects, as will be apparent to those skilled in the art. Attached Figure Description

[0020] Furthermore, the features and advantages of the invention will be readily understood by reading the following detailed description taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 The method provided by an embodiment of the present invention is illustrated;

[0022] Figure 2 This illustrates the selection of one of several electronic devices provided by an embodiment of the present invention;

[0023] Figure 3A The angle of arrival of a signal from a selectable second electronic device, as provided in an embodiment of the present invention, is shown.

[0024] Figure 3B The pointing direction provided by the embodiments of the present invention is shown;

[0025] Figure 4 An exemplary angle of arrival measurement operation provided by an embodiment of the present invention is shown;

[0026] Figure 5 The present invention illustrates possible gestures that users can use to remotely interact with electronic devices, as provided by embodiments of the present invention.

[0027] Figure 6 This illustrates a rule-based pointing gesture recognition operation provided by an embodiment of the present invention.

[0028] Figure 7 This illustrates a learning-based pointing gesture recognition operation provided by an embodiment of the present invention;

[0029] Figure 8 This illustrates a learning-based similarity pointing gesture recognition operation provided by an embodiment of the present invention;

[0030] Figure 9 The present invention illustrates sensors that may be included in a handheld device provided in an embodiment of the present invention;

[0031] Figure 10 This illustrates a handheld electronic device provided by an embodiment of the present invention.

[0032] It should be noted that in all the accompanying drawings, the same features are identified by the same element symbols. Detailed Implementation

[0033] Embodiments of the present invention relate to providing methods, handheld electronic devices, and systems for pointing gesture recognition (PGR). The handheld electronic device is used for remote interaction with a second electronic device. Non-limiting examples of the handheld electronic device may include smartphones, handheld remote controls, smart rings, smart bracelets, and smartwatches. Non-limiting examples of the second electronic device may include smart TVs, tablets, smart glasses, smartwatches, smartphones, personal computers, smart LEDs, robots such as robotic vacuum cleaners, speakers, and other household appliances.

[0034] According to embodiments of the present invention, a user of a handheld electronic device (or wearing the handheld electronic device on their wrist or finger) can remotely interact with a second electronic device by moving the handheld electronic device with one or more predefined motion-based gestures. These predefined motion-based gestures may include the user raising their hand holding the handheld electronic device from a position close to their chest or below their waist to a position where the handheld electronic device points towards the second electronic device the user wishes to control. When the user moves the handheld electronic device, the handheld electronic device can sense the movement based on signals received from one or more motion sensors of the handheld device. The handheld electronic device can also recognize the motion-based gesture based on the sensed movement and generate predetermined conditions when the recognized motion-based gesture corresponds to a predefined motion-based gesture. The handheld electronic device can also recognize the second electronic device based on signals from a radio frequency sensor of the handheld electronic device after the predetermined conditions are met. The handheld electronic device may also include a processor that processes these predetermined conditions using the methods described herein, so that the user can control the second electronic device using the handheld electronic device. The user's recognition of predefined motion-based gestures triggers the handheld electronic device to initiate interaction with the second electronic device, enabling the user to control the second electronic device using the handheld electronic device.

[0035] The interaction involves wireless communication between a handheld electronic device and a second device. The interaction may include the handheld electronic device transmitting a message containing a command or query from the second device. A command may cause the second device to perform an appropriate operation, such as changing a volume or light level, performing a hardware or software operation, etc. A query may cause the second device to send a response back to the handheld electronic device, such as a response containing information held by the second device and requested in the query. The interaction can be performed with or without user input.

[0036] Figure 1 In one embodiment, a method 100 for remotely interacting with a second device by a handheld electronic device is illustrated. Method 100, as well as other methods described herein, can be executed by routines and subroutines of a pointing gesture recognition system (PGRS) 200 of the handheld electronic device 210. The PGRS 200 may include software (e.g., a computer program) that can be executed by a processor 910 of the handheld electronic device 210 (see...). Figure 9The PGRS is a machine-readable instruction executed by the processor 910 of the handheld electronic device 210. The PGRS may additionally or alternatively include dedicated electronic components, and in some embodiments may include hardware-related firmware. The encoding of the PGRS 200 is entirely within the scope of those skilled in the art in consideration of this invention. Method 100 may include more or fewer operations than those shown and described, and may be performed in different orders. The computer-readable instructions of the PGRS 200 executed by the processor 910 of the handheld electronic device 210 may be stored in a non-transitory computer-readable medium.

[0037] Method 100 begins with operation 110. In operation 110, the method includes sensing motion of the handheld electronic device 110 based on signals generated by one or more motion sensors of the handheld electronic device 110. Method 100 then proceeds to operation 120.

[0038] In operation 120, method 100 identifies the sensed motion as a motion-based gesture based on signals received from one or more motion sensors of handheld electronic device 110 during movement of handheld electronic device 120. Method 100 then proceeds to operation 130.

[0039] In operation 130, method 110 identifies the second device based on other signals derived from: the one or more motion sensors; one or more other components of the handheld device; or a combination thereof. These other signals indicate the direction in which the handheld electronic device points to the end of a motion-based gesture. Therefore, the motion-based gesture acts as a trigger to initiate interaction with the second electronic device and also provides a means for the user to point at the second device, thereby recognizing the second device and launching an appropriate application for interacting with it. Operation 130 can use... Figure 4 The angle of arrival measurement is performed as shown. Method 100 then proceeds to operation 140.

[0040] In operation 140, after a predetermined condition is met and the second device is identified, method 110 initiates a user interaction for remotely interacting with the second device, wherein the predetermined condition is at least partially met when the identified motion-based gesture is a predetermined motion-based gesture for interacting with the second device 140.

[0041] Although operations 110, 120, 130, and 140 are performed sequentially in method 100, the operation of identifying the second device can be performed partially or entirely in parallel with the operation of identifying motion-based gestures and determining whether predetermined conditions are met. Performing the operations in the indicated order allows for device identification, particularly at the end of a motion-based gesture, and allows the user to use the same gesture to identify the second device and indicate the need for interaction with it.

[0042] Figure 2 Examples of a handheld electronic device 210 and several possible second devices provided in embodiments of the present invention, and their roles, are illustrated. Figure 5 As shown, a user of handheld device 210 can control multiple second devices (e.g., by selecting one at a time), including a smart TV 220, a tablet 230, smart glasses 240, a smartwatch 250, and a personal computer 260. Handheld device 210 and the second devices are part of operating environment 205. Figure 2 As shown, a user of handheld device 200 can control smart TV 220 by performing a predefined motion-based gesture, which ends with the user pointing handheld electronic device 210 at smart TV 220. Pointing handheld electronic device 210 at smart TV 220 causes the PGRS 200 of handheld device 210 to project a (real, virtual, or conceptual) ray 270 towards smart TV 220, and the PGRS 200 recognizes smart TV 220 as a secondary device. Ray 270 is a pointing direction known to those skilled in the art of ray tracing.

[0043] Figure 3A An example is shown where the handheld electronic device 210 identifies the smart TV 220 when the ray 270 projected by the handheld electronic device 210 does not terminate at the smart TV 220. In some embodiments, the PGRS 200 of the handheld device 210 performs a pointing-based selection based on a device-to-device angle of arrival measurement. Using the pointing-based angle of arrival measurement, the PGRS 200 of the handheld device 210 is able to identify a second device that the handheld electronic device 210 is not directly pointing at. Figure 3A As shown, the PGRS 200 of the handheld device 210 identifies the smart TV 220 based on a pointing-based selection of the device-to-device arrival angle 320. The arrival angle 320 is the angle between ray 270 and the second ray 310. Ray 270 is projected along the long axis of the handheld device 210 and extends from the center of the handheld device 210. Ray 310 is projected from the center of the handheld device 210 to the center of the second device. The handheld device 210 includes a radio frequency (RF) sensor 920 (see...). Figure 9 The radio frequency sensor includes an RF transmitter, an RF receiver, and one or more RF antennas. Similarly, the second electronic device includes an RF sensor, which includes an RF transmitter, an RF receiver, and one or more RF antennas. The RF sensor 920 of the second electronic device can be any RF sensor according to one of several known technical standards, including IEEE 802.11 (known to those skilled in the art). ), Low power consumption (known to those skilled in the art as BLE), ultra-wideband (known to those skilled in the art as UWB), and ultrasonic waves specifying the required angle of arrival value. In some embodiments of the invention, the angle of arrival 320° conforms to UWB, BLE and ultrasonic requirements.

[0044] The device-to-device angle of arrival 320° can be measured using several methods. One method involves measuring the propagation direction of the radio frequency wave incident on the RF sensor antenna. A second method involves measuring the phase of the radio frequency wave incident on multiple antenna array elements of the RF sensor. In the second method, the angle of arrival can be determined by calculating the difference between the measured phases of the incident radio frequency waves.

[0045] In some embodiments, to facilitate angle of arrival measurement, the handheld electronic device may send a request to the second device to transmit an appropriate RF signal. The RF signal can then be received, for example, using the antenna array of the handheld electronic device, and processed by the handheld electronic device to determine the angle of arrival 320°. Alternatively, the handheld electronic device may send an RF signal and a request for angle of arrival measurement to the second device. The second device can then receive the RF signal from the handheld electronic device, for example, using its own antenna array, and process the RF signal to determine the angle of arrival of the handheld electronic device from the angle of the second device. The result can be transmitted back to the handheld electronic device and used therefrom.

[0046] In some embodiments, UWB, WiFi, BLE, and ultrasonic technology standards require that the second ray 310 be projected onto the center of the second device. However, if the second device is large, the detector of the second device 330 used to measure the angle of arrival can be located at a large distance from the center of the second device. This large distance can, and in fact, move the second ray 310 to ray 340. Ray 340 has an associated angle 350. Angle 350 is added to the angle of arrival 320. The result of ray 340 and the offset angle 350 is that PGRS 200 is able to detect a pointing direction that is not projected onto the center of the second device.

[0047] Figure 3BExamples of pointing directions for a tablet computer 365, a smartwatch 375, a smart bracelet 385, a handheld electronic device 210, and smart glasses 395 are shown, referred to herein as device directions. For illustrative purposes, the direction of each device is defined by rays 360, 370, 380, 387, and 390 projected along the long axis of their respective devices. In each case, the ray extends from or passes through the center of the device. However, in other embodiments, the rays may be oriented differently. For the purposes of this discussion, pointing direction or device direction may be equivalent to the direction of the ray. According to various embodiments, a second electronic device can be selected based on the device direction (pointing direction) of the handheld electronic device. This direction can be determined based on signals from device components. For example, angle of arrival measurements as described above can be used to determine the device direction (pointing direction). In some embodiments, components such as gyroscopes and magnetometers can be used to determine the absolute device direction (pointing direction). Accelerometers and ranging processing can also be used to determine or support the determination of the device direction (pointing direction).

[0048] Figure 4 An exemplary flowchart is shown, illustrating the operation performed by a handheld electronic device to identify a second electronic device. Figure 4 The operation can be a sub-operation of operation 130 of method 100 performed by handheld device 210. Method 400 identifies the second device using a point-based selection based on the angle of arrival, wherein handheld electronic device 210 sends an angle of arrival measurement request to all second devices 410. The second devices determine the angle of arrival of these second devices using ray 270 and second ray 310 (or second ray 340 in some embodiments). The handheld electronic device then receives each angle of arrival response from all second devices 420. It should be noted that, here and elsewhere, the processing operation may be offloaded to other devices such as cloud computing devices, which promptly return the results to the handheld electronic device for use. In cases where handheld electronic device 210 can communicate with multiple second devices, the handheld electronic device uses the angle of arrival received from all second devices to identify 450 which second device can communicate with handheld electronic device 210. This identification 450 can be accomplished through two actions, namely 430 and 440. The first action 430 is a comparison of the angle of arrival received from each second device. The maximum angle of arrival is a predefined parameter that can vary depending on the device. The angle of arrival can also be determined based on the wireless technology used, for example, specified by the supported technology standard, which may include WiFi, BLE, UWB, and ultrasonic standards. The maximum angle of arrival can represent the pointing error tolerance. The second action 440 is to determine which second device has the minimum angle of arrival.

[0049] In some embodiments, the predetermined conditions further include recognizing confirmation input from the user. To improve the performance of the PGRS 200, enabling the PGRS 200 to select a second device intended by the user, the handheld electronic device 210 may vibrate to provide feedback to the user once the PGRS 200 has recognized the second device. This vibration may prompt the user to press a key or button on the handheld electronic device 210 to confirm that the recognized second device is the one the user intended to select.

[0050] In some embodiments, recognizing the input includes recognizing a second predetermined motion-based gesture of the mobile handheld electronic device 210. After the predetermined motion-based gesture has been recognized by the handheld device 210, the second predetermined motion-based gesture is recognized based on the sensed motion of the handheld electronic device.

[0051] In some embodiments, recognizing the confirmation input includes recognizing the handheld electronic device as rotated into place based on signals received from one or more motion sensors. As a non-limiting example, when the handheld electronic device 210 prompts confirmation that the correct second device has been selected, the user may twist the wrist of the hand holding the handheld electronic device 210.

[0052] In some embodiments, recognizing the confirmation input includes: recognizing, based on signals received from the one or more motion sensors, that the handheld electronic device remains in place after the predetermined motion-based gesture without further movement for a predetermined period of time. A non-limiting example of this confirmation is pointing the handheld electronic device 210 at a second device that the user wishes to control for one second. It should be understood that holding the electronic device 210 as confirmation is a "hover" known to those skilled in the art.

[0053] In some embodiments, recognizing the confirmation input includes detecting the presence of a signal indicating that a physical button on the handheld electronic device or a virtual button displayed on the touchscreen of the handheld electronic device has been pressed. A non-limiting example of this confirmation is pressing the power button of the handheld electronic device 210. Another non-limiting example of this confirmation is pressing a soft key on the keyboard of the handheld electronic device 210.

[0054] In some embodiments, the method further includes prompting the user to provide the confirmation input to confirm the intention to interact with the second device after recognizing the motion-based gesture as a predetermined motion-based gesture, after recognizing the second device, and before detecting the confirmation input.

[0055] In some embodiments, the predetermined condition further includes the presence of a detection signal, which indicates that a physical button of the handheld electronic device or a virtual button displayed on the touchscreen of the handheld electronic device is pressed.

[0056] In some embodiments, the predetermined condition includes detecting the presence of the signal indicating that the physical button or the virtual button was pressed at the start of the motion-based gesture.

[0057] In some embodiments, recognizing the motion-based gesture means that the predetermined motion-based gesture includes recognizing signals generated by the one or more motion sensors indicating that the handheld electronic device 210 moves from a first position to a second position in an upward arc. The first position corresponds to the handheld electronic device being close to the user's hip and pointing downwards, and the second position corresponds to the handheld electronic device 210 being held at the end of a straight arm and pointing towards the second device.

[0058] In some embodiments, recognizing the motion-based gesture means that the predetermined motion-based gesture includes recognizing signals generated by the one or more motion sensors indicating that the handheld electronic device moves linearly from a first position to a second position. The first position corresponds to the handheld electronic device being held in front of the user's body with a bent arm, and the second position corresponds to the handheld electronic device being held at the end of a straight arm and pointing towards the second device.

[0059] Figure 5 The illustration shows a user 510 holding a handheld electronic device 210 and moving the device according to three specific motion-based gestures that the user can use to remotely interact with a second device. These three motion-based gestures are included among predetermined motion-based gestures that can be recognized by the PGRS 200 of the handheld device 210. It should also be understood that signals generated by one or more motion sensors of the handheld device 210 can be processed by the PGRS 200 of the handheld device 210 and can be analyzed using models that may include human models and machine learning models.

[0060] Analysis using a human model can be performed by the PGRS 200, as shown below. These signals can be processed using an operation that classifies signals from motion sensors according to the types of movements a human body is typically capable of performing. Therefore, signals from one or more sensors can be mapped to movements performed by the human body to facilitate gesture recognition by the PGRS 200. These signals can be instantaneous readings from the motion sensors or samples acquired from the motion sensors over time intervals.

[0061] Analysis using a machine learning model can be performed by the PGRS 200, as follows. A machine learning model for recognizing motion-based gestures can be learned during the training phase by instructing the user to perform a predefined motion-based gesture and monitoring the resulting signals from one or more motion sensors. The generated signals can be used to generate a labeled dataset. The trained model can then be deployed in the PGRS 200 to recognize other instances of motion-based gestures based on new signals received from one or more motion sensors. The machine learning model can then process these additional signals to determine when to perform the gesture, and can output the same instruction.

[0062] When user 510 raises the handheld electronic device 210 held by hand 530 from position 540 to position 550 via moving arm 520, user 510 performs a motion-based gesture 560. It should be understood that when user 510 moves handheld device 210 in response to motion-based gesture 560, handheld electronic device 210 remains close to user 510's body. Motion-based gesture 560 can be sensed by handheld device 210, which senses the movement of handheld device 210 when the user performs motion-based gesture 560, including sensing displacement, rotation, and acceleration of handheld electronic device 210.

[0063] A motion-based gesture 580 occurs when user 510 extends handheld electronic device 210 from position 550 to position 570 using arm 520. It should be understood that handheld electronic device 210 is close to user 510's body at position 550, and this proximity decreases as handheld electronic device 210 extends to position 570 for gesture 580. The motion-based gesture 580 can also be sensed by sensing motion, including displacement, rotation, and acceleration of handheld electronic device 210 when the device is pointed at a second device.

[0064] A motion-based gesture 590 occurs when the user 510 rotates the arm 520 to move the handheld electronic device 210 directly from position 540 to position 570. It should be understood that the handheld electronic device 210 is close to the user 510's body at position 540, and this proximity to the user 510's body decreases as the handheld electronic device 210 extends to position 570 for gesture 590.

[0065] In some embodiments, identifying the motion-based gesture as the predetermined motion-based gesture includes performing pattern recognition on signals generated by the one or more motion sensors.

[0066] Implementations of PGRS 200 can recognize motion-based gestures using rule-based operations and learning-based operations, or a combination thereof. These operations can analyze signals generated by one or more motion sensors of the handheld electronic device 210. PGRS 200 can use acceleration patterns, rotation patterns, or magnetic field amplitude patterns to recognize that the motion-based gesture is a predefined motion-based gesture. PGRS 200 can use one or more of a variety of computational methods to recognize that the motion-based gesture is a predefined motion-based gesture. Computational methods may include performing similarity measurements, including Euclidean distance and cosine distance, and using methods including support vector machines (SVM), dynamic time warping (DTW), deep learning including autoencoders, long short-term memory (LSTM), and convolutional neural networks (CNN).

[0067] In some embodiments, the handheld electronic device 210 includes a gesture recognizer for recognizing motion-based gestures performed by a user holding (or wearing) the handheld electronic device 210 based on signals received from a motion sensor of the handheld device 210. The gesture recognition component may be implemented by a processor executing instructions stored in memory. In a non-limiting embodiment, the gesture recognition component implements rules for recognizing motion-based gestures based on signals from the motion sensor. In a non-limiting embodiment, the gesture recognition component implements a machine learning model that receives signals from the motion sensor and outputs predicted motion-based gestures based on the signals from the motion sensor. In a non-limiting embodiment, the gesture recognition component implements templates for recognizing motion-based gestures based on signals from the motion sensor, as described in further detail below. The machine learning model can be trained using supervised learning algorithms such as deep neural networks, support vector machines (SVMs), similarity learning, etc.

[0068] As a non-limiting example, when a user moves the handheld electronic device 210 forward and performs motion-based gestures 560 and 580 or 590, rule-based operation can process the electromagnetic field measured by the user, determine that the user is pointing forward at the handheld electronic device 210, and perform motion-based gestures 560 and 580 or 590 based on the intensity changes of the electromagnetic field measured by the user. Another non-limiting example of rule-based processing is that when a motion-based gesture 580 is performed based on the processing acceleration and / or rotation of the handheld electronic device 210, it is determined that the user has extended their arm toward a second device. The motion-based gesture 580 may include measuring the linear motion of the handheld electronic device 210, the acceleration of the handheld electronic device 210, and the user's arm without rotation. The motion-based gesture 580 may alternatively or additionally include only the rotation of the user's shoulder.

[0069] Figure 6 A non-limiting example embodiment of a gesture recognition method 600 performed by the PRGS 200 of a handheld electronic device 210 is shown. The gesture recognition method 600 begins at operation 610. During movement of the handheld electronic device 210, one or more motion sensors of the handheld electronic device 210 generate signals based on the movement of the handheld electronic device 210. The one or more motion sensors may include an accelerometer, a gyroscope, a magnetometer, and a barometer. In operation 610, sensor measurements are determined based on signals received from the one or more motion sensors of the handheld electronic device 210. Determining sensor measurements may include receiving signals, initial interpretation as numerical values, initial filtering, etc., or combinations thereof. Method 600 then proceeds to operation 620.

[0070] In operation 620, rule checks are performed, such as magnetic, motion, and acceleration rule checks. A magnetic rule check operation may process signals generated by a magnetometer. A motion rule check operation may process signals generated by an accelerometer or other sensors representing motion. An acceleration rule check operation may also process signals generated by an accelerometer. Rule checks include processing sensor measurements to determine if they represent a predetermined motion-based gesture. This may include checking for rule compliance, where rule checks (tests) whether sensor measurements exhibit predetermined features indicating that a predetermined motion-based gesture has been recognized. If all rules are followed (satisfied) 630, then PGRS 200 recognizes that the motion-based gesture performed by a user holding the handheld electronic device 201 (or wearing the handheld electronic device 210) is a predetermined motion-based gesture. In other words, PGRS 200 determines that the handheld electronic device 210 is being used in a pointing operation 640. Alternatively, if at least one rule is violated 650, then PGRS 200 determines that the predetermined motion-based gesture has not been recognized, and the handheld electronic device 210 is not being used in a pointing operation 660.

[0071] Figure 7 Another non-limiting example embodiment of a gesture recognition method 700 performed by the PRGS 200 of a handheld electronic device 210 is shown. In this exemplary embodiment, when a user performs a motion-based gesture by moving the handheld electronic device 210, one or more motion sensors of the handheld electronic device 210 generate signals, such as... Figure 5 As shown. Then, in 720, the signals generated by these motion sensors are received by a pre-trained model, which is used to infer the probability of each type of motion-based gesture in the set of motion-based gestures recognized by the pre-trained model based on the received signals. One or more motion sensors may include accelerometers, gyroscopes, magnetometers, and barometers. The pre-trained model may be implemented by SVM, CNN, and LSTM. The pre-trained model 720 outputs an identifier (i.e., a label) of the motion-based gesture type with the highest probability in the set of motion-based gestures as the recognized motion-based gesture. Then, PGRS 200 determines whether the label of the recognized motion-based gesture corresponds to a predetermined motion-based gesture.

[0072] Learning-based processing can be used to analyze a user pointing the handheld electronic device 210 forward during a motion-based gesture. This learning-based processing can include classification-based and similarity-based processing methods. Classification-based processing methods can include generating binary labels indicating that the user is pointing forward at the handheld electronic device 210 while performing a motion-based gesture. Classification-based processing methods can be performed using SVM, CNN, or LSTM. Similarity-based processing methods can include using pre-built pointing gesture sensor measurement templates.

[0073] Figure 8 Another non-limiting example embodiment of a gesture recognition method 800 performed by the PRGS 200 of a handheld electronic device 210 is shown. The gesture recognition method begins with operation 810, wherein receiving 810 corresponds to a template of sensor measurements of a predefined motion-based gesture.

[0074] In some embodiments, identifying the motion-based gesture as the predetermined motion-based gesture includes processing signals generated by the one or more motion sensors using a human model. When a user of the handheld electronic device 210 performs a pointing gesture, one or more motion sensors of the handheld electronic device 210 generate signals based on the motion of the handheld electronic device 210. In operation 820, signals received from the one or more motion sensors are processed to generate sensor measurements 820 for the one or more motion sensors. In operation 830, signal similarity processing 830 is performed using the template received in operation 810 and the sensor measurements generated in 820. In operation 840, PGRS 200 determines that the similarity is greater than a threshold θ. In operation 850, PGRS 200 determines that the sensor measurement does not correspond to the predetermined motion-based gesture. In operation 860, PGRS 200 determines that the similarity is less than or equal to the threshold θ 860, and proceeds to operation 870, where PGRS 200 determines that the sensor measurement corresponds to the predetermined motion-based gesture.

[0075] In some embodiments, the second device is identified after it is determined that the sensed motion of the handheld electronic device 210 has stopped.

[0076] In some embodiments, initiating the user interaction includes launching an application on the handheld electronic device 210 for interacting with the second device.

[0077] In some embodiments, the method further includes sensing other movements of the handheld electronic device based on other signals generated by one or more motion sensors after the application is launched.

[0078] In some embodiments, the method further includes identifying other sensed movements as predetermined based on a gesture to deselect a movement, including moving the electronic device 210 and ceasing interaction with the second device.

[0079] In some embodiments, the method further includes closing the application after identifying that the other sensed motion is the predetermined gesture based on deselecting the motion. Figure 5 A non-limiting example of a gesture based on deselection movement is the reverse movement of the previously described gesture 590. The reverse movement of gesture 590, which can be a gesture based on deselection movement, can be a movement of the handheld electronic device 210 from position 570 to position 540. As a non-limiting example, a gesture based on deselection movement sensing the reverse gesture 590 can be identified by detecting an increase in the user's electromagnetic field strength as the handheld electronic device 210 approaches.

[0080] In some embodiments, the one or more motion sensors include one or more of the following: accelerometer, magnetometer, proximity sensor, gyroscope, ambient light sensor, camera, microphone, radio frequency receiver, near field communication device, and temperature sensor.

[0081] Figure 9 Several motion sensors are shown, which can be included in a handheld electronic device 210 to generate signals corresponding to the user's motion-based gestures as the user moves the handheld electronic device 210. A processor 910 of the handheld electronic device 210 processes signals from a radio-frequency (RF) sensor 920, a camera 930, a microphone 940, a temperature sensor 950, a near-field sensor 960, a light sensor 970, an accelerometer 980, and a gyroscope 990. The processor 910 may need to process signals generated by multiple of these components to determine a predefined gesture. Alternatively, the processor 910 may need to process signals generated by a single motion sensor to determine a motion-based gesture. Various sensors can be used, wherein these sensors output signals that are directly responsive to or related to motion. Accelerometers respond to motion-based acceleration. Gyroscopes and magnetometers react to motion because they respond to changes in orientation. Magnetometers also react to motion in response to magnetic fields, such as magnetic fields that move them toward or away from the body. Other sensors react to changes in conditions that may be a result of motion. Signals from multiple sensors can potentially be used to detect predetermined motion-based gestures by processing these signals to identify specific ranges of values, signatures, waveforms, waveform combinations, etc., typically generated by a predetermined motion-based gesture being performed.

[0082] In some embodiments, the one or more motion sensors are used to detect one or more of the following: displacement motion, rotational motion, and user proximity.

[0083] A non-limiting example of determining displacement motion is determining displacement based on predetermined conditions generated by the accelerometer 980 of the handheld electronic device 210. When a user moves the handheld electronic device 210 according to a motion-based gesture, the signal generated by the accelerometer 980 may correspond to acceleration and / or deceleration of the handheld electronic device 210. It should be understood that displacement motion may include sensing the proximity of the handheld electronic device 210 to the user's body via the accelerometer 980.

[0084] Non-limiting examples of rotational motion of the handheld electronic device 210 can be determined using the gyroscope 990 of the handheld electronic device 210. When a user moves the handheld electronic device 210 according to a motion-based gesture, the gyroscope 990 can generate a signal corresponding to the rotation of the handheld electronic device 210.

[0085] A non-limiting example of determining the proximity of the handheld device 210 to the user's body is to use an RF detector 920 to detect the strength of the electromagnetic field generated by the user's body. The electromagnetic field strength can represent the proximity of the handheld electronic device 210 to the user's body or a radio frequency source. For example, as the handheld electronic device 210 moves towards the user's body, the RF detector 920 can detect a gradually stronger electromagnetic field from the user. As another example, as the handheld electronic device 210 moves away from the user's body, the RF detector 920 can detect a gradually weaker electromagnetic field from the user.

[0086] According to some embodiments, the handheld electronic device 210 may include (e.g., in addition to) Figure 9 In addition to the processor 910, there is an artificial intelligence (AI) processor 915. The AI ​​processor may include one or more of a graphics processing unit (GPU), a tensor processing unit (TPU), a field-programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). The AI ​​processor can be used to perform computations on machine learning models (i.e., machine learning operations). The model itself can be deployed and stored in the memory of a handheld electronic device.

[0087] In some embodiments, the other components of the handheld device include one or more of the following: a magnetometer, a proximity sensor, a camera, a microphone, a radio frequency receiver, and a near-field communication device.

[0088] In some embodiments, the one or more other components of the handheld device are used to detect the location of one or more other electronic devices, including the second device.

[0089] In some embodiments, the one or more other components of the handheld device are used to: detect the position of the one or more other electronic devices at least in part based on the angle of arrival measurement of the signal emitted by each of the one or more other electronic devices.

[0090] In some embodiments, the method further includes: after the predetermined conditions are met and the second device is identified, displaying an icon representing the second device on the display of the handheld electronic device, and changing the position of the icon on the display according to one or both of the following: the angle between the pointing direction of the handheld electronic device and the direction of the second device relative to the handheld electronic device, and a measurement of the probability that the handheld device is pointing at the second device.

[0091] In some embodiments, the handheld electronic device includes one or more motion sensors for generating signals representing the motion of the handheld device.

[0092] In some embodiments, the handheld electronic device further includes: processing electronics configured to sense motion of the handheld device based on signals generated by one or more motion sensors. The device is further configured to: identify the sensed motion as a motion-based gesture, the motion-based gesture including moving the handheld electronic device. The device is further configured to identify a second device based on other signals from: the one or more motion sensors; one or more other components of the handheld device; or a combination thereof. The other signals indicate a direction in which the handheld electronic device points towards the end of the motion-based gesture. The device is further configured to initiate a user interaction for remotely interacting with the second device after a predetermined condition is met and the second device is identified. The predetermined condition is at least partially met when the identified motion-based gesture is a predetermined motion-based gesture for interacting with the second device.

[0093] It should be understood that embodiments of handheld electronic devices can be used to perform the methods described herein.

[0094] Figure 10 A non-limiting example of a handheld electronic device 210 with functional modules is shown, which may be provided using components of processing electronics 1015. The processing electronics may include a computer processor that executes program instructions stored in memory 1030. As previously described, device 210 may include a motion sensor 1020, other components 1035, a user interface 1025, and a transmitter and receiver 1040. The user interface 1025 may be used for user-guided interaction with a second device. The transmitter and receiver 1040 may be used to communicate with the second device and, in some embodiments, may also be used, for example, to locate the second device using angle of arrival measurement and processing.

[0095] like Figure 10The illustrated device 210 includes a pointing gesture recognition module 1045. The pointing gesture recognition module can perform various operations of the PGRS described elsewhere herein. Device 210 may include a second device recognition module 1055 for recognizing a second device that device 210 is pointing to at the end of a predetermined gesture. Device 210 may include a user interaction module 1050 that can initiate and execute appropriate applications for user-oriented interaction with the second device. Device 210 may include a confirmation module 1060 that monitors confirmation input as described elsewhere herein and can also prompt the user for confirmation input, for example, by vibrating device 210, emitting a sound, or generating a prompt on the display of device 210.

[0096] Although the invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations thereof can be made without departing from the invention. The specification and drawings are to be regarded merely as an illustration of the invention as defined in the appended claims and any and all modifications, variations, combinations or equivalents falling within the scope of this specification are to be considered.

Claims

1. A method for remotely interacting with a second device via a handheld electronic device, characterized in that, The method includes: The motion of the handheld electronic device is sensed based on signals generated by one or more motion sensors of the handheld electronic device; The sensed motion is identified by motion-based gestures, wherein the motion-based gestures include moving the handheld electronic device; The second device is identified based on other signals from: the one or more motion sensors; one or more other components of the handheld electronic device; or a combination thereof, wherein the other signals indicate the direction in which the handheld electronic device points toward the end of the motion-based gesture; the other signals also indicate the angle of arrival between the handheld electronic device and the second device, the angle of arrival between the handheld device and the second device being the smallest angle of arrival among the angles of arrival between the handheld electronic device and a plurality of second devices to be identified; After a predetermined condition is met and the second device is identified, a user interaction for remotely interacting with the second device is initiated, wherein the predetermined condition is at least partially met when the identified motion-based gesture is a predetermined motion-based gesture for interacting with the second device.

2. The method according to claim 1, characterized in that, The predetermined conditions also include recognizing confirmation input from the user.

3. The method according to claim 2, characterized in that, Recognizing the confirmation input includes recognizing the sensed motion and also includes a second predetermined motion-based gesture, the second predetermined motion-based gesture including moving the handheld electronic device and following the predetermined motion-based gesture.

4. The method according to claim 2 or 3, characterized in that, Recognizing the confirmation input includes using the one or more motion sensors to recognize that the handheld electronic device has been rotated into place.

5. The method according to claim 2 or 3, characterized in that, Recognizing the confirmation input includes using the one or more motion sensors to recognize that the handheld electronic device remains in place after the predetermined motion-based gesture, without further movement for a predetermined period of time.

6. The method according to claim 2 or 3, characterized in that, Identifying the confirmation input includes detecting the presence of a signal indicating that a physical button on the handheld electronic device or a virtual button displayed on the touchscreen of the handheld electronic device has been pressed.

7. The method according to any one of claims 2 or 3, characterized in that, Also includes: After recognizing the motion-based gesture as a predetermined motion-based gesture, and after recognizing the second device but before detecting the confirmation input, the user is prompted to provide the confirmation input to confirm the intention to interact with the second device.

8. The method according to any one of claims 1 to 3, characterized in that, The predetermined condition also includes the presence of a detection signal, which indicates that a physical button of the handheld electronic device or a virtual button displayed on the touchscreen of the handheld electronic device is pressed.

9. The method according to claim 8, characterized in that, The predetermined conditions include detecting the presence of the signal, which indicates that the physical button or the virtual button was pressed when the motion-based gesture began.

10. The method according to any one of claims 1 to 3, characterized in that, Recognizing the motion-based gesture includes recognizing signals generated by the one or more motion sensors, the signals indicating that the handheld electronic device moves from a first position to a second position in an upward arc, wherein the first position corresponds to the handheld electronic device being close to the user's hip and pointing downwards, and the second position corresponds to the handheld electronic device being held at the end of a straight arm and pointing towards the second device.

11. The method according to any one of claims 1 to 3, characterized in that, Recognizing the motion-based gesture includes: recognizing signals generated by the one or more motion sensors, the signals indicating that the handheld electronic device moves linearly from a first position to a second position, wherein the first position corresponds to the handheld electronic device being held in front of the user's body with a bent arm, and the second position corresponds to the handheld electronic device being held at the end of a straight arm and pointing towards the second device.

12. The method according to any one of claims 1 to 3, characterized in that, Recognizing the motion-based gesture as the predetermined motion-based gesture includes performing pattern recognition on signals generated by the one or more motion sensors.

13. The method according to any one of claims 1 to 3, characterized in that, Recognizing the motion-based gesture as the predetermined motion-based gesture includes: processing signals generated by the one or more motion sensors using a human model.

14. The method according to any one of claims 1 to 3, characterized in that, Identifying the second device based on the other signals includes: identifying the second device based on the orientation of the handheld electronic device relative to the second device.

15. The method according to any one of claims 1 to 3, characterized in that, After determining that the motion sensed by the handheld electronic device has stopped, the second device is identified.

16. The method according to any one of claims 1 to 3, characterized in that, Initiating the user interaction includes launching an application on the handheld electronic device and interacting with the second device.

17. The method according to claim 16, characterized in that, This also includes, after the application is launched: Other movements of the handheld electronic device are sensed based on other signals generated by the one or more motion sensors; Other detected movements are pre-defined gestures based on deselecting the movement, including: moving the handheld electronic device and ceasing interaction with the second device; After recognizing that the other detected motion is the predetermined gesture based on deselecting the motion, the application is closed.

18. The method according to any one of claims 1 to 3, characterized in that, The one or more motion sensors include one or more of the following: accelerometer, magnetometer, proximity sensor, gyroscope, ambient light sensor, camera, microphone, radio frequency receiver, near field communication device, and temperature sensor.

19. The method according to any one of claims 1 to 3, characterized in that, The one or more motion sensors are used to detect one or more of the following: displacement motion, rotational motion, and user proximity.

20. The method according to any one of claims 1 to 3, characterized in that, The other components of the handheld electronic device include one or more of the following: a magnetometer, a proximity sensor, a camera, a microphone, a radio frequency receiver, and a near-field communication device.

21. The method according to any one of claims 1 to 3, characterized in that, The one or more other components of the handheld electronic device are used to detect the location of one or more other electronic devices, including the second device.

22. The method according to claim 21, characterized in that, The one or more other components of the handheld electronic device are used to: detect the position of the one or more other electronic devices, at least in part, based on the angle of arrival measurement of the signal emitted by each of the one or more other electronic devices.

23. The method according to any one of claims 1 to 3, characterized in that, Also includes: After the predetermined conditions are met and the second device is identified, an icon representing the second device is displayed on the display of the handheld electronic device, and the position of the icon on the display is changed according to one or both of the following: the angle between the pointing direction of the handheld electronic device and the direction of the second device relative to the handheld electronic device, and a measurement of the probability that the handheld electronic device is pointing at the second device.

24. A handheld electronic device, characterized in that, include: One or more motion sensors are used to generate signals representing the motion of the handheld electronic device; Processing electronic components for: The motion of the handheld electronic device is sensed based on signals generated by the one or more motion sensors; The sensed motion is identified by motion-based gestures, which include moving the handheld electronic device. The second device is identified based on other signals from: the one or more motion sensors; one or more other components of the handheld electronic device; or a combination thereof, wherein the other signals indicate the direction in which the handheld electronic device points toward the end of the motion-based gesture; the other signals also indicate the angle of arrival between the handheld electronic device and the second device, the angle of arrival between the handheld device and the second device being the smallest angle of arrival among the angles of arrival between the handheld electronic device and a plurality of second devices to be identified; After a predetermined condition is met and the second device is identified, a user interaction for remotely interacting with the second device is initiated, wherein the predetermined condition is at least partially met when the identified motion-based gesture is a predetermined motion-based gesture for interacting with the second device.

25. The handheld electronic device according to claim 24, characterized in that, The predetermined conditions also include recognizing confirmation input from the user.

26. The handheld electronic device according to claim 25, characterized in that, Recognizing the confirmation input includes recognizing the sensed motion and also includes a second predetermined motion-based gesture, the second predetermined motion-based gesture including moving the handheld electronic device and following the predetermined motion-based gesture.

27. The handheld electronic device according to claim 25 or 26, characterized in that, Recognizing the confirmation input includes using the one or more motion sensors and the processing electronics to recognize that the handheld electronic device has been rotated into place.

28. The handheld electronic device according to claim 25 or 26, characterized in that, Recognizing the confirmation input includes using the one or more motion sensors and the processing electronics to recognize that the handheld electronic device remains in place after the predetermined motion-based gesture, without further movement for a predetermined period of time.

29. The handheld electronic device according to claim 25 or 26, characterized in that, Identifying the confirmation input includes detecting the presence of a signal using processing electronics, the signal indicating that a physical button on the handheld electronic device or a virtual button displayed on the touchscreen of the handheld electronic device has been pressed.

30. The handheld electronic device according to claim 25 or 26, characterized in that, It also includes prompting the user to provide the confirmation input to confirm the intention to interact with the second device after recognizing the motion-based gesture as a predetermined motion-based gesture, after recognizing the second device and before detecting the confirmation input.

31. The handheld electronic device according to any one of claims 24 to 26, characterized in that, The predetermined condition also includes the presence of a detection signal, which indicates that a physical button of the handheld electronic device or a virtual button displayed on the touchscreen of the handheld electronic device is pressed.

32. The handheld electronic device according to claim 31, characterized in that, The predetermined conditions include detecting the presence of the signal, which indicates that the physical button or the virtual button was pressed when the motion-based gesture began.

33. The handheld electronic device according to any one of claims 24 to 26, characterized in that, Recognizing the motion-based gesture includes recognizing signals generated by the one or more motion sensors, the signals indicating that the handheld electronic device moves from a first position to a second position in an upward arc, wherein the first position corresponds to the handheld electronic device being close to the user's hip and pointing downwards, and the second position corresponds to the handheld electronic device being held at the end of a straight arm and pointing towards the second device.

34. The handheld electronic device according to any one of claims 24 to 26, characterized in that, Recognizing the motion-based gesture includes: recognizing signals generated by the one or more motion sensors, the signals indicating that the handheld electronic device moves linearly from a first position to a second position, wherein the first position corresponds to the handheld electronic device being held in front of the user's body with a bent arm, and the second position corresponds to the handheld electronic device being held at the end of a straight arm and pointing towards the second device.

35. The handheld electronic device according to any one of claims 24 to 26, characterized in that, Recognizing the motion-based gesture as the predetermined motion-based gesture includes performing pattern recognition on signals generated by the one or more motion sensors.

36. The handheld electronic device according to any one of claims 24 to 26, characterized in that, Recognizing the motion-based gesture as the predetermined motion-based gesture includes: processing signals generated by the one or more motion sensors using a human model.

37. The handheld electronic device according to any one of claims 24 to 26, characterized in that, Identifying the second device based on the other signals includes: identifying the second device based on the orientation of the handheld electronic device relative to the second device.

38. The handheld electronic device according to any one of claims 24 to 26, characterized in that, After determining that the motion sensed by the handheld electronic device has stopped, the second device is identified.

39. The handheld electronic device according to any one of claims 24 to 26, characterized in that, Initiating the user interaction includes launching an application on the handheld electronic device and interacting with the second device.

40. The handheld electronic device according to claim 39, characterized in that, It is also used after the application is launched: Other movements of the handheld electronic device are sensed based on other signals generated by the one or more motion sensors; Other detected movements are pre-defined gestures based on deselecting the movement, including: moving the handheld electronic device and ceasing interaction with the second device; After recognizing that the other detected motion is the predetermined gesture based on deselecting the motion, the application is closed.

41. The handheld electronic device according to any one of claims 24 to 26, characterized in that, The one or more motion sensors include one or more of the following: accelerometer, magnetometer, proximity sensor, gyroscope, ambient light sensor, camera, microphone, radio frequency receiver, near field communication device, and temperature sensor.

42. The handheld electronic device according to any one of claims 24 to 26, characterized in that, The one or more motion sensors are used to detect one or more of the following: displacement motion, rotational motion, and user proximity.

43. The handheld electronic device according to any one of claims 24 to 26, characterized in that, The other components of the handheld electronic device include one or more of the following: a magnetometer, a proximity sensor, a camera, a microphone, a radio frequency receiver, and a near-field communication device.

44. The handheld electronic device according to any one of claims 24 to 26, characterized in that, The one or more other components of the handheld electronic device are used to detect the location of one or more other electronic devices, including the second device.

45. The handheld electronic device according to claim 44, characterized in that, The one or more other components of the handheld electronic device are used to: detect the position of the one or more other electronic devices, at least in part, based on the angle of arrival measurement of the signal emitted by each of the one or more other electronic devices.

46. ​​The handheld electronic device according to any one of claims 24 to 26, characterized in that, It is also configured to, after the predetermined conditions are met and the second device is identified, display an icon representing the second device on the display of the handheld electronic device, and change the position of the icon on the display according to one or both of the following: the angle between the pointing direction of the handheld electronic device and the direction of the second device relative to the handheld electronic device, and a measurement of the probability that the handheld electronic device is pointing at the second device.

47. A computer-readable medium including instructions, characterized in that, When the instruction is executed by the processor of the handheld electronic device, it causes the handheld electronic device to perform the method according to any one of claims 1 to 23.

48. A handheld electronic device, characterized in that, Used to perform the method according to any one of claims 1 to 23.

Images