Activating Cross-Device Interactions Using Pointing Gesture Recognition

Abstract

A method and a handheld device for remotely interacting with a second device. The method and apparatus identify a second device from a plurality of devices based on a user's gestures. As the user gestures, a motion sensor that senses these gestural movements can generate signals that can be processed by rule-based and / or learning-based methods. The results of processing these signals can be used to identify the second device. To improve performance, the user can be prompted to confirm that the identified second device is the device the user wants to control remotely. The results of processing these signals can also be used to enable the user to remotely interact with the second device.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This is the first application filed for this invention.

[0002] FIELD OF THEINVENTION The present invention relates to remotely interacting with electronic devices, and in particular to methods and apparatus used to recognize user gestures and then apply these gestures to remote interactions with electronic devices. [Background technology]

[0003] As more smart devices enter the consumer market, there is an increasing consumer demand for the ability to remotely control these smart devices.

[0004] As handheld electronic devices (cell phones) become more common and powerful, there is an increasing demand for the ability to remotely control smart devices using consumers' handheld electronic devices. However, current products aimed at addressing this demand do not generally allow users to select which smart device they wish to control. As a result, there is a need for products that enhance the user's experience by allowing users to select each time which smart device they wish to remotely control.

[0005] This background information is provided to identify information believed by the applicant to be of possible relevance to the present invention. It is not necessarily intended, nor should it be construed, that any of the preceding information constitutes prior art against the present invention. Summary of the Invention [Problem to be solved by the invention]

[0006] Embodiments of the present invention provide a system for implementing a pointing gesture recognition system (PGRS), as well as a method for implementing an architecture that provides a PGRS that enables a user to remotely control one or more second devices through recognition of the user's gestures. [Means for solving the problem]

[0007] According to an embodiment of the present invention, there is provided a method for remotely interacting with a second device by a handheld electronic device, the method comprising: determining whether the handheld electronic device is capable of remotely interacting with a second device based on signals generated by one or more movement sensors of the handheld electronic device; electronic and sensing motion of the device. The method also includes recognizing that the sensed motion is a motion-based gesture that includes movement of the handheld electronic device. The method further includes identifying the second device based on one or both of the signal or a further signal. The further signal may be one or more of a motion sensor; a handheld electronic device; electronic or one or more other components of the device; or a combination thereof. The orientation of the device (which may be determined based on the motion sensor signal) and these further signals indicate the direction in which the handheld electronic device is pointing at the end of the motion-based gesture. After the predetermined condition is met and the second device is identified, the method proceeds to initiate a user interaction for remotely interacting with the second device, where the predetermined condition is at least partially met when the recognized motion-based gesture is a predetermined motion-based gesture for interacting with the second device.

[0008] A technical advantage of such an embodiment is that user interaction is initiated only after a predefined motion-based gesture is performed. This prevents the handheld electronic device from erroneously identifying that the user desires to interact with the second device, which may adversely affect the user experience, for example consuming battery or processing resources unnecessarily. Furthermore, the motion-based gesture is incorporated into the identification of the second device in that the second device is identified based on pointing, but such pointing can be integrated with the motion-based gesture. This combination can integrate both the recognition of the motion-based gesture and the identification of the second device.

[0009] In some embodiments, the predefined condition further includes recognizing a confirmatory input from the user. A technical advantage of such an embodiment is that user interaction is initiated only after the predefined motion-based gesture and the confirmatory input are performed. This further prevents the handheld electronic device from erroneously identifying that the user desires to interact with the second device based on false recognition of movements corresponding to the predefined motion-based gesture.

[0010] In a further embodiment, recognizing the confirmation input includes using one or more motion sensors to recognize that following a predetermined motion-based gesture, the handheld electronic device is held in a predetermined in position without further movement for a predetermined period of time. electronic The confirmation input is performed automatically by pointing at the device without further user interaction with the device, which enhances the user experience.

[0011] In some further embodiments, recognizing the motion-based gesture as a predetermined motion-based gesture includes recognizing a signal generated by the one or more motion sensors indicative of movement of the handheld electronic device in an upward arcing motion from a first position to a second position, the first position corresponding to the handheld electronic device being near a user's waist and pointing downward and the second position corresponding to the handheld electronic device being held at the end of an outstretched arm and pointing towards the second device.

[0012] In another further embodiment, recognizing the motion-based gesture as a predetermined motion-based gesture includes recognizing a signal generated by one or more motion sensors indicative of movement of the handheld electronic device from a first position to a second position in a linear motion, the first position corresponding to the handheld electronic device being held by a user with a bent arm in front of the user's body and the second position corresponding to the handheld electronic device being held at the end of an outstretched arm and pointing towards the second device.

[0013] In some embodiments, identifying the second device comprises: electronic This is performed after determining that the sensed movement of the device has stopped. A technical advantage of this embodiment is that it may more reliably identify the second device and inhibit other devices unintentionally pointed to during the motion-based gesture from being identified as the second device.

[0014] In some embodiments, the one or more motion sensors include one or more of an accelerometer, a magnetometer, a proximity sensor, a gyroscope, an ambient light sensor, a camera, a microphone, a radio frequency receiver, a near field communication device, and a temperature sensor. A technical advantage of such embodiments is that motion-based gestures can be recognized by sensors that respond directly to motion, by sensors that respond directly to parameters that indirectly correlate with motion or position (e.g., physical proximity, radio frequency, sound, or temperature), or by combinations thereof. This provides a variety of inputs that can be processed to obtain motion-based or position information.

[0015] In some embodiments, a handheld electronic The one or more other components of the device are configured to detect a location of the one or more other electronic devices based at least in part on measuring an angle of arrival of a signal emitted by each of the one or more other electronic devices. A technical advantage of this embodiment is that a signal, such as a radio frequency signal, may be used to locate the second device. Thus, the antenna array system may be utilized to perform, for example, physical positioning.

[0016] In some embodiments, after the predetermined condition is met and the second device is identified, an icon representing the second device is displayed on the display of the handheld electronic device. The position of the icon on the display is determined by the angle between the pointing direction of the handheld electronic device and the orientation of the second device relative to the handheld electronic device and the position of the icon on the display relative to the handheld electronic device. electronic and / or a measure of the likelihood that the device is pointed at a second device. A technical advantage of this embodiment is that it provides a visual correlation between a user's actions and the device's response, which can be used in a feedback loop involving the user to facilitate the second device selection process.

[0017] According to another embodiment, there is provided a handheld electronic device configured to perform operations consistent with the above-described methods. electronic It may include one or more motion sensors configured to generate signals indicative of device movement, and processing electronics configured to implement such operations.

[0018] The embodiments are described above in relation to the aspects of the invention in which they can be implemented. Those skilled in the art will understand that the embodiments may be implemented in relation to the aspect in which they are described, but may also be implemented with other embodiments of that aspect. It will be clear to those skilled in the art when the embodiments are mutually exclusive or otherwise mutually inconsistent. Some embodiments may be described in relation to one aspect, but may also be applicable to other aspects, as will be clear to those skilled in the art. [Brief description of the drawings]

[0019] Further features and advantages of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0020] [Figure 1] 1 illustrates a method provided in accordance with one embodiment of the present disclosure.

[0021] [Diagram 2] 1 illustrates the selection of one of several electronic devices according to one embodiment of the present disclosure.

[0022] [Figure 3A] 13 illustrates an angle of arrival of a signal from a selectable second electronic device according to one embodiment of the present disclosure.

[0023] [Figure 3B] 1 illustrates pointing directions according to one embodiment of the present disclosure.

[0024] [Figure 4] 4 illustrates an exemplary angle of arrival measurement operation according to one embodiment of the present disclosure.

[0025] [Diagram 5] 1 illustrates potential gestures that a user may use to remotely interact with an electronic device, according to one embodiment of the present disclosure.

[0026] [Figure 6] 1 illustrates a rule-based pointing gesture recognition operation according to one embodiment of the present disclosure.

[0027] [Figure 7] 1 illustrates a learning-based pointing gesture recognition operation according to one embodiment of the present disclosure.

[0028] [Figure 8] 1 illustrates a learning-based similarity pointing gesture recognition operation according to one embodiment of the present disclosure.

[0029] [Figure 9] 1 illustrates a sensor that may be included in a handheld electronic device according to one embodiment of the present disclosure.

[0030] [Figure 10] 1 illustrates a handheld electronic device according to one embodiment of the present disclosure.

[0031] It should be noted that throughout the accompanying drawings, like features are identified by like reference numbers. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The present invention relates to providing a method, handheld electronic device and system for pointing gesture recognition (PGR). The handheld electronic device is used to remotely interact with a second electronic device. Non-limiting examples of handheld electronic devices include smartphones, handheld remote controls, smart rings, smart bands and smart watches. Non-limiting examples of second electronic devices include smart TVs, tablets, smart glasses, smart watches, smartphones, personal computers, smart LEDs, robots such as robot vacuum cleaners, speakers and other home appliances.

[0033] According to an embodiment of the invention, a user of a handheld electronic device holding the handheld electronic device (or wearing the handheld electronic device on their wrist or finger) can remotely interact with a second electronic device by moving the handheld electronic device in one or more predefined motion-based gestures. These predefined motion-based gestures can include the user raising the hand holding the handheld electronic device from a position close to the chest or below the waist to a position where the handheld electronic device is pointing towards the second electronic device that the user wishes to control. The handheld electronic device may display a motion-based gesture to indicate when the user moves the handheld electronic device. electronicThe handheld electronic device may sense movement of the handheld electronic device based on signals received from one or more movement sensors of the device. The handheld electronic device may also recognize a motion-based gesture based on the sensed movement and generate a predefined condition when the recognized motion-based gesture corresponds to a predefined motion-based gesture. The handheld electronic device may also identify a second electronic device based on signals from a radio frequency sensor of the handheld electronic device after the predefined condition is met. The handheld electronic device may also include a processor that processes these predefined conditions using the methods described herein such that a user may control the second electronic device using the handheld electronic device. Recognition of the performance of a predefined motion-based gesture by a user of the handheld electronic device triggers the handheld electronic device to initiate an interaction with the second electronic device to enable the user to control the second electronic device using the handheld electronic device.

[0034] An interaction involves wireless communication between the handheld electronic device and a second device. An interaction can include the handheld electronic device sending a message containing a command or query to which the second device responds. A command can cause the second device to perform an operation appropriate to the second device, such as changing a volume or light level, performing a hardware or software action, etc. A query can cause the second device to send a response to the handheld electronic device, such as a response maintained by the second device and containing the information requested in the query. electronic The interaction can be performed with or without input from the user.

[0035] FIG. 1 illustrates a method 100 used by a handheld electronic device to remotely interact with a second device in one embodiment. The method 100, as well as other methods described herein, may be performed by routines and subroutines of a pointing gesture recognition system (PGRS) 200 of the handheld electronic device 210. The PGRS 200 may include software (e.g., a computer program) including machine-readable instructions executable by a processor 910 (see FIG. 9) of the handheld electronic device 210. The PGRS may additionally or alternatively include dedicated electronics, which in some embodiments may include hardware-related firmware. Coding of the PGRS 200 is within the purview of one of ordinary skill in the art in view of this disclosure. The method 100 may include additional or fewer operations than those illustrated and described, and operations may be performed in a different order. Computer-readable instructions of the PGRS 200 executable by the processor 910 of the handheld electronic device 210 may be stored on a non-transitory computer-readable medium.

[0036] The method 100 begins at operation 110. At operation 110, the method includes: 210 based on signals generated by one or more motion sensors of the handheld electronic The method 100 then proceeds to operation 120, which includes sensing the movement of the device.

[0037] At operation 120, the method 100 comprises: 210 On the move, handheld electronic devices 210 The method 100 then proceeds to operation 130. The method 100 then recognizes that the sensed movement is a motion-based gesture based on signals received from the one or more motion sensors of the

[0038] In operation 130, the method 100 one or more motion sensors, handheld electronicThe second device is identified based on a further signal from one or more other components of the device or a combination thereof. Such further signal indicates the direction in which the handheld electronic device is pointing at the end of the motion-based gesture. The motion-based gesture thus serves as a trigger to initiate an interaction with the second electronic device, and also provides a means by which the user can point towards the second device so that it can be recognized, and can launch an appropriate application for interacting with the second device. Operation 130 may be performed using angle of arrival measurements, as illustrated in FIG. 4. Method 100 then proceeds to operation 140.

[0039] In operation 140, the method 100 initiates a user interaction for remotely interacting with the second device after a predetermined condition is met and the second device is identified, where the predetermined condition is at least partially met when the recognized motion-based gesture is a predetermined motion-based gesture for interacting with the second device 140.

[0040] In method 100, operations 110, 120, 130, and 140 are performed in sequence, although the operation of identifying the second device may be performed partially or fully in parallel with the operation of recognizing the motion-based gesture and determining that a predetermined condition is met. Performing the operations in the illustrated order allows for identifying the device, particularly at the end of the motion-based gesture, which may allow a user to use the same gesture to both identify the second device and to indicate that interaction with the second device is desired.

[0041] FIG. 2 illustrates a handheld electronic device 210 and an example of some potential second devices and the roles they play, according to one embodiment of the present disclosure. As shown in FIG. electronicA user of the device 210 can control multiple second devices (e.g., one at a time via selection), including a smart TV 220, a tablet 230, smart glasses 240, a smart watch 250, and a personal computer 260. electronic The device 210 and the second device are part of the operating environment 205. As shown in FIG. electronic device 210 A user of the handheld electronic device 210 can control the smart television 220 by performing predefined motion-based gestures that are completed by the user pointing the handheld electronic device 210 at the smart television 220. Pointing the handheld electronic device 210 at the smart television 220 can electronic The PGRS 200 of the device 210 can be caused to project a (real, virtual or conceptual) ray 270 towards the smart TV 220, causing the PGRS 200 to identify the smart TV 220 as a second device. The ray 270 is known to those skilled in the art of ray tracing as a pointing direction.

[0042] 3A illustrates an example of a handheld electronic device 210 identifying a smart television 220 when a light beam 270 projected by the handheld electronic device 210 does not terminate at the smart television 220. electronic The PGRS 200 of the device 210 performs pointing-based selection based on device-to-device angle of arrival measurements. electronic The PGRS 200 of the device 210 can identify a second device that is not directly pointed to by the handheld electronic device 210. As shown in FIG. electronicThe PGRS 200 of the device 210 identifies the smart TV 220 based on the pointing-based selection using an angle of arrival 320 between the devices. The angle of arrival 320 is the angle between the ray 270 and a second ray, i.e., ray 310. The ray 270 is projected along the long axis of the handheld electronic device 210 and extends from the center of the handheld electronic device 210. The ray 310 is projected from the center of the handheld electronic device 210 to the center of the second device. electronic The device 210 includes a radio frequency (RF) sensor 920 (see FIG. 9 ) including an RF transmitter, an RF receiver, and one or more RF antennas. Similarly, the second electronic device includes an RF sensor including an RF transmitter, an RF receiver, and one or more RF antennas. The RF sensor 920 and the RF sensor of the second electronic device can be any RF sensor based on one of several known technology standards, including IEEE 802.11 (known to those skilled in the art as WiFi), Bluetooth® Low Energy (known to those skilled in the art as BLE), Ultra Wideband (known to those skilled in the art as UWB), and Ultrasound specifies the required angle of arrival value. In some embodiments of the invention, the angle of arrival 320 complies with the requirements of UWB, WiFi®, BLE, and Ultrasound.

[0043] Several methods can be used to measure the angle of arrival 320 between the devices. One method involves measuring the propagation direction of the radio waves incident on the antenna of the RF sensor. A second method is to measure the phase of the radio waves incident on multiple antenna array elements of the RF sensor. With this second method, the angle of arrival can be determined by calculating the difference between the measured phases of the incident radio waves.

[0044] In some embodiments, to facilitate the angle of arrival measurement, the handheld electronic device may transmit a request to the second device to transmit an appropriate RF signal. The RF signal is then received, for example, using an antenna array of the handheld electronic device, and the angle of arrival is measured. cornerThe RF signal can be processed by the handheld electronic device to determine the angle of arrival of the handheld electronic device from the perspective of the second device. Additionally or alternatively, the handheld electronic device may transmit an RF signal as well as a request for an angle of arrival measurement of the second device. The second device may then receive the RF signal from the handheld electronic device, for example using an antenna array of the second device, and process the RF signal to determine the angle of arrival of the handheld electronic device from the perspective of the second device. The results can be transmitted back to the handheld electronic device and used thereby.

[0045] In some embodiments, UWB, WiFi, BLE and ultrasound technology standards require that the second light ray 310 be projected at the center of the second device. However, if the second device is large, the detector of the second device 330 used to measure the angle of arrival may be a significant distance from the center of the second device. This significant distance may result in, in effect, moving the second light ray 310 to light ray 340. Light ray 340 has an associated angle 350. Angle 350 adds an offset to the angle of arrival 320. The result of light ray 340 and offset angle 350 is that the PGRS 200 can detect a pointing direction that is not projected at the center of the second device.

[0046] FIG. 3B illustrates examples of pointing directions, also referred to herein as device orientations, for a tablet 365, a smart watch 375, a smart ring 385, a handheld electronic device 210, and smart glasses 395, respectively. For purposes of explanation, each device orientation is defined by a ray 360, 370, 380, 387, and 390 projected along the long axis of its respective device. The ray in each case extends from or passes through the center of the device. However, in other embodiments, the ray may be directed differently. For purposes of this discussion, the pointing direction or device orientation may be equivalent to the direction of the ray. According to various embodiments, the second electronic device may be selected based on the device orientation (pointing direction) of the handheld electronic device. This direction may be determined based on signals from device components. For example, angle of arrival measurements as described above may be used to determine the device orientation (pointing direction). In some embodiments, components such as gyroscopes and magnetometers may be used to determine the absolute device orientation (pointing direction). The accelerometer may also be used in conjunction with a dead reckoning process to determine or support the determination of the device's orientation (pointing direction).

[0047] 4 illustrates an example flow chart of operations performed by a handheld electronic device to identify a second electronic device. electronicIt may be a sub-operation of operation 130 of method 100 performed by device 210. Method 400 identifies second devices using pointing-based selection based on arrival angle, where handheld electronic device 210 sends arrival angle measurement requests to all second devices 410. The second devices determine their arrival angles using light ray 270 and second light ray 310 (or second light ray 340 in some embodiments). The handheld electronic device then receives 420 each arrival angle response from all of the second devices. Note here and elsewhere that processing operations may be offloaded to other devices, such as a cloud computing device, which may return the results to the handheld electronic device for use in a timely manner. In a situation where handheld electronic device 210 may communicate with multiple second devices, handheld electronic device uses the arrival angles received from all second devices to identify 450 which second devices may communicate with handheld electronic device 210. This identification 450 may be facilitated by two actions, namely 430 and 440. The first action 430 is a comparison of the angles of arrival received from each second device. The maximum angle of arrival is a predefined parameter that may be device dependent. The angle of arrival may also be dependent on the wireless technology being used, as specified by the technology standard supported, which may include, for example, WiFi, BLE, UWB, and ultrasonic standards. The maximum angle of arrival may represent a pointing error tolerance. The second action 440 is a determination of which second device has the smallest angle of arrival.

[0048] In some embodiments, the predetermined condition further includes recognizing a confirmatory input from the user. To improve the performance of the PGRS 200 so that the PGRS 200 selects the second device that the user intended to select, the handheld electronic device 210 may vibrate to provide feedback to the user once the PGRS 200 identifies the second device. The vibration may prompt the user to press a key or button on the handheld electronic device 210 to confirm that the identified second device is the second device that the user intended to select.

[0049] In some embodiments, recognizing the confirmation input includes recognizing a second predefined motion-based gesture of moving the handheld electronic device 210. The second predefined motion-based gesture is a gesture that is generated when the predefined motion-based gesture is detected by the handheld electronic device 210. electronic Once recognized by the device 210, it is recognized based on the sensed movement of the handheld electronic device.

[0050] In some embodiments, recognizing the confirmation input includes recognizing that the handheld electronic device is rotating in place based on signals received from one or more motion sensors. As a non-limiting example, a user may twist the wrist of the hand holding the handheld electronic device 210 when prompted by the handheld electronic device 210 to confirm that the correct second device has been selected.

[0051] In some embodiments, recognizing the confirmation input includes recognizing, based on signals received from one or more motion sensors, that the handheld electronic device is held in a predetermined location without further movement for a predetermined time following a predetermined motion-based gesture. A non-limiting example of this confirmation is pointing the handheld electronic device 210 toward a second device that the user wishes to control for one second. HandheldIt should be appreciated that holding the electronic device 210 is known to those skilled in the art as "hovering."

[0052] In some embodiments, recognizing the confirmatory input includes detecting the presence of a signal indicative of a press of a physical button on the handheld electronic device or a virtual button displayed on a touch screen of the handheld electronic device. A non-limiting example of this confirmation is pressing a power button on the handheld electronic device 210. Another non-limiting example of this confirmation is pressing a soft key on the keyboard of the handheld electronic device 210.

[0053] In some embodiments, the method further includes, after recognizing that the motion-based gesture is a predetermined motion-based gesture, prompting the user to provide a confirmation input to confirm an intent to interact with the second device after the second device has been identified and before detecting the confirmation input.

[0054] In some embodiments, the predetermined condition further comprises detecting the presence of a signal indicative of a press of a physical button on the handheld electronic device or a virtual button displayed on a touch screen of the handheld electronic device.

[0055] In some embodiments, the predetermined condition includes detecting the presence of a signal indicating a press of a physical or virtual button at the initiation of the motion-based gesture.

[0056] In some embodiments, recognizing that the motion-based gesture is a predetermined motion-based gesture includes recognizing a signal generated by one or more motion sensors indicative of movement of the handheld electronic device 210 from a first position to a second position in an upward arcing motion, the first position corresponding to the handheld electronic device being near a user's waist and pointing downward, and the second position corresponding to the handheld electronic device 210 being held at the end of an outstretched arm and pointing towards the second device.

[0057] In some embodiments, recognizing the motion-based gesture as a predetermined motion-based gesture includes recognizing a signal generated by one or more motion sensors indicative of movement of the handheld electronic device from a first position to a second position in a linear motion, the first position corresponding to the handheld electronic device being held by a user in front of the user's body with a bent arm and the second position corresponding to the handheld electronic device being held at the end of an outstretched arm and pointing towards the second device.

[0058] 5 illustrates a user 510 holding a handheld electronic device 210 and moving it according to three specific motion-based gestures that can be used by the user to remotely interact with a second device. electronic The handheld device 210 includes a PGRS 200 that can recognize certain motion-based gestures. electronic The signals generated by one or more motion sensors of the device 210 are electronic It should also be appreciated that the PGRS 200 of the device 210 may be processed and analyzed using models that may include mannequin models and machine learning models.

[0059] Analysis using the mannequin model may be performed by the PGRS 200 as follows: The signals from the motion sensors may be processed using operations that classify the signals based on the types of movements that a human body can typically make. Thus, signals from one or more sensors may be mapped to movements made by a human body to facilitate gesture recognition by the PGRS 200. The signals may be instantaneous readings from the motion sensors or samples taken from the motion sensors at a time interval.

[0060] Analysis using the machine learning model may be performed by the PGRS 200 as follows: A machine learning model for recognizing motion-based gestures may be learned during a training phase by instructing a user to perform predefined motion-based gestures and monitoring the resulting signals from one or more motion sensors. The resulting signals may be used to generate a labeled data set. The trained model may then be PGRS The method may expand at 200 to recognize additional instances of the motion-based gesture based on new signals received from one or more motion sensors. The additional signals may then be processed by a machine learning model to determine when the gesture is being performed, and the machine learning model may output an indication of the same.

[0061] The motion-based gesture 560 is performed by the user 510 when the user 510 raises the handheld electronic device 210 held by the hand 530 from position 540 to position 550 by moving the arm 520. electronic It should be appreciated that the handheld electronic device 210 is maintained close to the body of the user 510 when moving the device 210. The motion-based gesture 560 involves sensing the displacement, rotation, and acceleration of the handheld electronic device 210 when the user performs the motion-based gesture 560. electronic A handheld device that senses motion of the device 210 electronic It can be sensed by the device 210.

[0062] Motion-based gesture 580 occurs when user 510 uses arm 520 to extend handheld electronic device 210 from position 550 to position 570. It should be appreciated that handheld electronic device 210 is close to the body of user 510 when in position 550 and this proximity to the body of user 510 decreases as handheld electronic device 210 is extended to position 570 for gesture 580. Motion-based gesture 580 can also be sensed by sensing movement, including sensing displacement, rotation and acceleration of handheld electronic device 210 when the handheld electronic device is pointed towards a second device.

[0063] Motion-based gesture 590 occurs when user 510 rotates arm 520 to move handheld electronic device 210 directly from position 540 to position 570. It should be appreciated that handheld electronic device 210 is close to the body of user 510 when in position 540, and this proximity to the body of user 510 decreases as handheld electronic device 210 is extended to position 570 for gesture 590.

[0064] In some embodiments, recognizing that the motion-based gesture is a predetermined motion-based gesture includes performing pattern recognition on signals generated by the one or more motion sensors.

[0065] An embodiment of the PGRS 200 may recognize a motion-based gesture using rule-based and learning-based operations or a combination thereof. These operations may analyze signals generated by one or more motion sensors of the handheld electronic device 210. The PGRS 200 may recognize a motion-based gesture as a predetermined motion-based gesture using acceleration patterns, rotation patterns, or magnetic field magnitude patterns. The PGRS 200 may recognize a motion-based gesture as a predetermined motion-based gesture using one or more of a variety of computational methods. The computational methods may include performing a similarity measurement, which may include Euclidean distance, cosine distance, and using dynamic programming techniques, which may include support vector machines (SVM), dynamic time warping (DTW), deep learning, which may include autoencoders, long short-term memory (LSTM), and convolutional neural networks (CNN).

[0066] In some embodiments, the handheld electronic device 210 includes a gesture recognizer configured to recognize motion-based gestures performed by a user holding (or wearing) the handheld electronic device 210 based on signals received from a motion sensor of the handheld electronic device 210. The gesture recognition component may be implemented by a processor executing instructions stored in a memory. In a non-limiting embodiment, the gesture recognition component implements rules for recognizing motion-based gestures based on signals from the motion sensor. In a non-limiting embodiment, the gesture recognition component implements a machine learning model that receives signals from the motion sensor and outputs predicted motion-based gestures based on the signals from the motion sensor. In a non-limiting embodiment, the gesture recognition component implements templates that are used to recognize motion-based gestures based on signals from the motion sensor, as described in more detail below. The machine learning model may be implemented using a supervised learning algorithm (such as a deep neural network, a support vector machine (SVM), similarity learning, etc.). ) You can learn by using

[0067] As a non-limiting example, when a user moves the handheld electronic device 210 forward and performs a motion-based gesture 560 and 580 or 590, the rule-based operation can process the user's measured electromagnetic field and determine that the user pointed the handheld electronic device 210 forward and performed the motion-based gesture 560 and 580 or 590 based on the measured change in the strength of the user's electromagnetic field. Another non-limiting example of rule-based processing is determining that the user extended their arm towards a second device when performing a motion-based gesture 580 based on processing acceleration and / or rotation of the handheld electronic device 210. The motion-based gesture 580 can involve measuring the linear motion of the handheld electronic device 210, the acceleration of the handheld electronic device 210, and the lack of rotation of the user's arm. The motion-based gesture 580 can alternatively or additionally include only the rotation of the user's shoulder.

[0068] Handheld electronic device 210 PGRS A non-limiting example embodiment of a gesture recognition method 600 performed by the handheld electronic device 200 is illustrated in FIG. 6. The gesture recognition method 600 begins at operation 610. During movement of the handheld electronic device 210, one or more motion sensors of the handheld electronic device 210 generate signals based on the movement of the handheld electronic device 210. The one or more motion sensors may include an accelerometer, a gyroscope, a magnetometer, a barometer. At operation 610, a sensor measurement is determined based on a signal received from the one or more motion sensors of the handheld electronic device 210. Determining the sensor measurement may include receiving a signal, an initial interpretation as a numerical value, initial filtering, or the like, or a combination thereof. The method 600 then proceeds to operation 620.

[0069] In operation 620, rule checks are performed, such as magnetic, motion, and acceleration rule check operations. The magnetic rule check operation may process signals generated by a magnetometer. The motion rule check operation may process signals generated by an accelerometer or other sensor indicative of motion. The acceleration rule check operation may also process signals generated by an accelerometer. The rule checks include processing the sensor measurements to determine whether they are indicative of a predefined motion-based gesture. This may include checking whether rules are satisfied, where the rules check (test) whether the sensor measurements exhibit predefined characteristics that indicate a predefined motion-based gesture has been recognized. If all rules are followed (satisfied) 630, the PGRS 200 recognizes that the motion-based gesture performed by the user holding the handheld electronic device 201 (or wearing the handheld electronic device 210) is the predefined motion-based gesture. In other words, the PGRS 200 determines 640 that the handheld electronic device 210 is being used in a pointing operation. Alternatively, if at least one rule is violated 650, the PGRS 200 determines that the predefined motion-based gesture is not recognized and the handheld electronic device 210 is not being used 660 in a pointing operation.

[0070] Handheld electronic device 210 PGRSAnother non-limiting example embodiment of the gesture recognition method 700 performed by the handheld electronic device 200 is illustrated in FIG. 7. In this example embodiment, as illustrated in FIG. 5, one or more motion sensors of the handheld electronic device 210 generate signals when a user performs a motion-based gesture by moving the handheld electronic device 210 that is being held. The signals generated by these motion sensors are then received at 720 by a pre-trained model configured to infer a probability of each type of motion-based gesture in a set of motion-based gestures recognized by the pre-trained model based on the received signals. The one or more motion sensors may include an accelerometer, a gyroscope, a magnetometer, and a barometer. The pre-trained model may be implemented by SVM, CNN, and LSTM. The pre-trained model 720 outputs an identifier (i.e., a label) of the type of motion-based gesture that has the highest probability in the set of motion-based gestures as the recognized motion-based gesture. The PGRS 200 then determines whether the label of the recognized motion-based gesture corresponds to a predefined motion-based gesture.

[0071] A learning-based process can be used to analyze a user pointing the handheld electronic device 210 forward during a motion-based gesture. Such learning-based processes can include classification-based processing methods and similarity-based processing methods. The classification-based processing method can include generating a binary label indicating that the user is pointing the handheld electronic device 210 forward when performing the motion-based gesture. The classification-based processing method can be performed using an SVM, a CNN, or an LSTM. The similarity-based processing method can include the use of a pre-built pointing gesture sensor measurement template.

[0072] Handheld electronic device 210 PGRSAnother non-limiting example embodiment of a gesture recognition method 800 performed by 200 is illustrated in Figure 8. The gesture recognition method begins at operation 810 where a template of sensor measurements corresponding to a predefined motion-based gesture is received 810.

[0073] In some embodiments, recognizing that the motion-based gesture is a predefined motion-based gesture includes processing signals generated by one or more motion sensors using a mannequin model. The one or more motion sensors of the handheld electronic device 210 generate signals based on a movement of the handheld electronic device 210 when a pointing gesture is performed by a user holding the handheld electronic device 210. In operation 820, the signals received from the one or more motion sensors are processed to generate sensor measurements 820 of the one or more motion sensors. In operation 830, a signal similarity process 830 is performed using the template received in operation 810 and using the sensor measurements generated in 820. In operation 840, the PGRS 200 determines that the similarity is greater than a threshold theta. In operation 850, the PGRS 200 determines that the sensor measurements do not correspond to the predefined motion-based gesture. At operation 860, the PGRS 200 determines that the similarity is less than or equal to the threshold theta 860 and proceeds to operation 870 where the PGRS 200 determines that the sensor measurements correspond to a predetermined motion-based gesture.

[0074] In some embodiments, identifying the second device is performed after determining that the sensed movement of the handheld electronic device 210 has stopped.

[0075] In some embodiments, initiating the user interaction includes launching an application on the handheld electronic device 210 to interact with the second device.

[0076] In some embodiments, the method further comprises, after launching the application, determining whether the handheld is running a motion sensor based on further signals generated by the one or more motion sensors. Electronic Devices 210 This also includes sensing further movement of the

[0077] In some embodiments, the method also includes recognizing that the sensed further movement is a predefined de-selection motion-based gesture that includes movement of the electronic device 210 and for stopping interaction with the second device.

[0078] In some embodiments, the method also includes closing the application after recognizing that the sensed further movement is a predetermined deselection motion-based gesture. A non-limiting example of a deselection motion-based gesture is the inverse of gesture 590 from FIG. 5, described above. The inverse of gesture 590, which may be a deselection motion-based gesture, may be a movement of handheld electronic device 210 from position 570 to position 540. As a non-limiting example, sensing the deselection motion-based gesture of inverse gesture 590 may be recognized by a radio frequency motion sensor that detects an increase in the strength of the user's electromagnetic field as the proximity of handheld electronic device 210 increases.

[0079] In some embodiments, the one or more motion sensors include one or more of an accelerometer, a magnetometer, a proximity sensor, a gyroscope, an ambient light sensor, a camera, a microphone, a radio frequency receiver, a near field communication device, and a temperature sensor.

[0080] FIG. 9 illustrates several motion sensors that may be included in the handheld electronic device 210 to generate signals corresponding to a user's motion-based gestures as the user moves the handheld electronic device 210. The processor 910 of the handheld electronic device 210 processes predefined conditions generated by a radio frequency (RF) sensor 920, a camera 930, a microphone 940, a temperature sensor 950, a near-field sensor 960, a light sensor 970, an accelerometer 980, and a gyroscope 990. The processor 910 may require processed signals generated by a number of these components to determine a predefined gesture. Alternatively, the processor 910 may require processed signals generated by a single motion sensor to determine a motion-based gesture. A variety of sensors may be used, where such sensors output signals that are either directly responsive to motion or correlated with motion. Accelerometers respond to motion-based acceleration. Gyroscopes and magnetometers respond to motion as they respond to changes in orientation. Magnetometers also respond to motion toward or away from a magnetic field, such as the human body magnetic field. Other sensors respond to changes in conditions that may be the result of motion. Potentially, signals from multiple sensors can be used to detect a given motion-based gesture by processing the signals to identify a particular range of values, signature, waveform, combination of waveforms, etc. that are attributable to a given motion-based gesture that is typically being performed.

[0081] In some embodiments, the one or more motion sensors are configured to detect one or more of displacement motion, rotational motion, and proximity to a user.

[0082] A non-limiting example of determining the displacement motion is determining the displacement based on a predetermined condition generated by the accelerometer 980 of the handheld electronic device 210. The signal generated by the accelerometer 980 can correspond to the acceleration and / or deceleration of the handheld electronic device 210 as the user moves it according to the motion-based gesture. It should be appreciated that the displacement motion can include sensing, by the accelerometer 980, the proximity of the handheld electronic device 210 to the user's body.

[0083] A non-limiting example of rotational movement of the handheld electronic device 210 can be determined using the gyroscope 990 of the handheld electronic device 210. As a user moves the handheld electronic device 210 according to a motion-based gesture, the gyroscope 990 can generate a signal corresponding to the rotation of the handheld electronic device 210.

[0084] Handheld against the user's body electronic A non-limiting example of determining the proximity of the device 210 is to detect the strength of the electromagnetic field generated by the user's body using the RF detector 920. The strength of the electromagnetic field can indicate the proximity of the handheld electronic device 210 to the user's body or to a source of RF waves. For example, as the handheld electronic device 210 is moved toward the user's body, the RF detector 920 can detect an increasingly stronger user's electromagnetic field. As a further example, as the handheld electronic device 210 is moved away from the user's body, the RF detector 920 can detect an increasingly weaker user's electromagnetic field.

[0085] According to some embodiments, the handheld electronic device 210 may include an artificial intelligence (AI) processor 915 (e.g., in addition to the processor 910 of FIG. 9). The AI ​​processor may comprise one or more of a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), and an application specific integrated circuit (ASIC). The AI ​​processor may be configured to perform computations of machine learning models (i.e., machine learning operations). The models themselves may be deployed and stored in the memory of the handheld electronic device.

[0086] In some embodiments, a handheld electronic The one or more other components of the device include one or more of a magnetometer, a proximity sensor, a camera, a microphone, a radio frequency receiver, and a near field communication device.

[0087] In some embodiments, a handheld electronic One or more other components of the device are configured to detect the location of one or more other electronic devices, including the second device.

[0088] In some embodiments, a handheld electronic The one or more other components of the device are configured to detect a position of the one or more other electronic devices based at least in part on angle-of-arrival measurements of signals emitted by each of the one or more other electronic devices.

[0089] In some embodiments, the method further includes, after the predetermined condition is met and the second device is identified, displaying an icon representing the second device on a display of the handheld electronic device, and determining an angle between a pointing direction of the handheld electronic device and an orientation of the second device relative to the handheld electronic device and a position of the handheld electronic device relative to the handheld electronic device. electronic and a measurement of the likelihood that the device is pointed at the second device.

[0090] In some embodiments, the handheld electronic device is electronic One or more motion sensors are provided that are configured to generate signals indicative of device movement.

[0091] In some embodiments, the handheld electronic device may determine whether or not the handheld electronic device is capable of detecting a motion based on signals generated by the one or more motion sensors. electronic The device further includes processing electronics configured to sense a motion of the device. The device is further configured to recognize that the sensed motion is a motion-based gesture that includes a movement of the handheld electronic device. The device includes one or more motion sensors, a handheld electronic The device is further configured to identify the second device based on a further signal from one or more other components of the device or a combination thereof. The further signal indicates a direction in which the handheld electronic device is pointing at the end of the motion-based gesture. The device is further configured to initiate a user interaction to remotely interact with the second device after the predetermined condition is met and the second device is identified. The predetermined condition is at least partially met when the recognized motion-based gesture is a predetermined motion-based gesture for interacting with the second device.

[0092] It should be appreciated that embodiments of the handheld electronic device may be configured to perform the methods described herein.

[0093] 10 illustrates a non-limiting example of a handheld electronic device 210 having functional modules that may be provided using components such as processing electronics 1015. The processing electronics may include a computer processor that executes program instructions stored in memory 1030. As previously described, the device 210 may include a motion sensor 1020, additional components 1035, a user interface 1025, and a transmitter and receiver 1040. The user interface 1025 may be used to direct a user's interaction with the second device. The transmitter and receiver 1040 may be used to communicate with the second device and, in some embodiments, may also locate the second device using, for example, angle of arrival measurements and processing.

[0094] The device 210 illustrated in FIG. 10 includes a pointing gesture recognition module 1045. The pointing gesture recognition module can perform various operations of the PGRS, as described elsewhere herein. The device 210 can include a second device identification module 1055, configured to identify a second device to which the device 210 is pointing, e.g., upon completion of a predefined gesture. The device 210 can include a user interaction module 1050, which can launch and execute an appropriate application for user-directed interaction with the second device. The device 210 can include a confirmation module 1060, which can monitor for a confirmation input and also prompt the user for a confirmation input, e.g., by vibrating the device 210, generating a sound, or generating a prompt on a display of the device 210, as described elsewhere herein.

[0095] While the invention has been described with reference to particular features and embodiments thereof, it will be apparent that various modifications and combinations can be made thereto without departing from the invention. Accordingly, the specification and drawings are to be considered merely as illustrative of the invention as defined by the appended claims, and it is intended to cover any and all modifications, variations, combinations or equivalents that are within the scope of the invention.

Claims

1. 1. A method for remotely interacting with a second device by a handheld electronic device, the method comprising: sensing motion of the handheld electronic device based on signals generated by one or more motion sensors of the handheld electronic device; recognizing that the sensed movement is a motion-based gesture that includes a movement of the handheld electronic device; identifying the second device based on a further signal from the one or more motion sensors, one or more other components of the handheld electronic device, or a combination thereof, the further signal indicating a direction in which the handheld electronic device is pointing at an end of the motion-based gesture; initiating a user interaction for remotely interacting with the second device after a predetermined condition is met and the second device is identified, the predetermined condition being at least partially met when the recognized motion-based gesture is a predetermined motion-based gesture for interacting with the second device; and wherein identifying the second device includes providing feedback indicating that the second device has been identified, and the predetermined condition further includes recognizing a confirmatory input from a user responsive to the feedback, and recognizing the confirmatory input includes: (i) recognizing that the sensed movement further includes a second predefined motion-based gesture following the predefined motion-based gesture, the second predefined motion-based gesture moving the handheld electronic device; (ii) recognizing, using the one or more motion sensors, that the handheld electronic device is rotated in place; (iii) using the one or more motion sensors to recognize that following the predetermined motion-based gesture, the handheld electronic device is held in a predetermined location without further movement for a predetermined time; The method includes at least one of the following:

2. after recognizing that the motion-based gesture is a predetermined motion-based gesture, prompting the user to provide the confirmation input to confirm an intent to interact with the second device after the second device has been identified and prior to detecting the confirmation input. The method of claim 1.

3. the predetermined condition further comprises detecting the presence of a signal indicative of a press of a physical button of the handheld electronic device or a virtual button displayed on a touch screen of the handheld electronic device. The method of claim 1.

4. the predetermined condition includes detecting the presence of the signal indicating a press of the physical or virtual button at the initiation of the motion-based gesture. The method according to claim 3.

5. Recognizing that the motion-based gesture is the predetermined motion-based gesture includes recognizing a signal generated by the one or more motion sensors indicative of movement of the handheld electronic device in an upward arcing motion from a first position to a second position, the first position corresponding to the handheld electronic device being near a user's waist and pointing downward, and the second position corresponding to the handheld electronic device being held at the end of an outstretched arm and pointing towards the second device. The method of claim 1.

6. 1. A handheld electronic device, comprising: one or more motion sensors configured to generate signals indicative of motion of the handheld electronic device; processing electronics, sensing motion of the handheld electronic device based on the signals generated by the one or more motion sensors; Recognizing that the sensed movement is a motion-based gesture that includes a movement of the handheld electronic device; identifying a second device based on a further signal from the one or more motion sensors, one or more other components of the handheld electronic device, or a combination thereof, where the further signal indicates a direction in which the handheld electronic device is pointing at an end of the motion-based gesture; after a predetermined condition is met and the second device is identified, initiating a user interaction for remotely interacting with the second device, where the predetermined condition is at least partially met when the recognized motion-based gesture is a predetermined motion-based gesture for interacting with the second device. and processing electronics configured to and identifying the second device includes providing feedback indicating that the second device has been identified, and the predetermined condition further includes recognizing a confirmatory input from a user responsive to the feedback, and recognizing the confirmatory input includes: (i) recognizing that the sensed movement further comprises a second predefined motion-based gesture following the predefined motion-based gesture, the second motion-based gesture comprising a movement of the handheld electronic device; (ii) using the one or more motion sensors and the processing electronics to recognize that the handheld electronic device is rotated in a predetermined position; (iii) using the one or more motion sensors and the processing electronics to recognize that following the predetermined motion-based gesture, the handheld electronic device is held in a predetermined location without further movement for a predetermined time; A handheld processing device comprising at least one of:

7. and after recognizing that the motion-based gesture is a predetermined motion-based gesture, prompting the user to provide the confirmation input to confirm an intention to interact with the second device after the second device has been identified and before detecting the confirmation input.

7. The handheld processing device of claim 6.

8. the predetermined condition further comprises detecting the presence of a signal indicative of a press of a physical button of the handheld electronic device or a virtual button displayed on a touch screen of the handheld electronic device; 7. The handheld processing device of claim 6.

9. the predetermined condition includes detecting the presence of the signal indicating a press of the physical or virtual button at the initiation of the motion-based gesture.

9. The handheld processing device of claim 8.

10. 13. A computer readable medium comprising instructions that, when executed by a processor of a handheld electronic device, cause the handheld electronic device to perform the method of claim 1.

Images