Radar-based gesture recognition situation-aware control

Abstract

To provide a method and a system for radar-based gesture-recognition with context-sensitive gating and other context-sensitive controls.SOLUTION: In an environment 400, sensor data from a proximity sensor such as a radar system 104 and / or a movement sensor such as an inertial measurement unit (IMU) 408 produces a context of user equipment 102. The method and the system enable the user equipment to recognize contexts when a radar system can be unreliable and should not be used for gesture-recognition, enabling the user equipment to automatically disable or "gate" the output from the radar system according to the context. The user equipment prevents the radar system from transitioning to a high-power state to perform gesture-recognition in contexts where radar data detected by the radar system is likely due to unintentional input.SELECTED DRAWING: Figure 4

Description

Background Art

[0001] Background Some computing devices (also referred to as “user devices”) include a radar system for detecting input. For example, the radar system provides a radar field and recognizes two-dimensional and three-dimensional (also referred to as “touch-independent”) radar-based gestures that occur within or through the radar field. The radar system may continuously evaluate reflections within the radar field and frequently transition to a gesture recognition state to interpret what may be a radar-based gesture input. However, transitioning to the gesture recognition state in response to an unintended or misjudged radar input wastes power and may cause malfunction if a misrecognized radar-based gesture is used to trigger or perform a function.

Summary of the Invention

[0002] Summary This document describes techniques and systems for radar-based gesture recognition using context-aware gating and other context-aware controls. The techniques and systems use sensor data from multiple sensors to define the context of the user device. The multiple sensors include low-power sensors such as an inertial measurement unit (IMU). It may include the sensor device and may exclude high-power sensor devices such as cameras. The sensor data can be inertial sensor data from an IMU, proximity data from a proximity sensor, radar data from a radar system, or any other sensor data. The sensor data defines the situation of the user device, such as user activities or characteristics of the computing environment. In certain situations, when the radar system is not reliable or not very reliable for radar-based gesture recognition, the method and system enable the user device to automatically disable or "gate" the radar-based gesture recognition. To do so, the user device may limit the input to or the output from the gesture recognition model. The user device may also disable the gesture recognition model to prevent the radar system from performing radar-based gesture recognition entirely. When the situation changes to a different situation that is unlikely to cause errors or misjudgments in gesture recognition, the user device can enable the radar-based gesture recognition again. When the user device is operating in a situation where radar-based gestures are unlikely to occur, the user device automatically gates the gesture recognition. Gating the gesture recognition prevents an application or other subscriber running on the user device from performing functions in response to the radar input obtained during gating. By doing so, the method prevents misjudged gesture recognition from triggering actions by the subscriber of the gesture recognition. Preventing misjudgments saves power for a computing system using a radar-based gesture recognition system and improves usability and user satisfaction.

[0003] For example, a method for situation-aware control of radar-based gesture recognition is described. The method includes receiving sensor data from a plurality of sensors of a user device, determining a situation of the user device based on the sensor data, determining whether the situation meets requirements for radar-based gesture recognition, and in response to a determination that the situation does not meet the requirements for radar-based gesture recognition, gating the radar system to prevent the radar system from outputting radar-based gesture cues to an application subsystem of the user device. and including gating the radar system to prevent the radar system from outputting radar-based gesture cues to an application subsystem of the user device.

[0004] In another example, after receiving second sensor data from a plurality of sensors, the method includes determining a second situation of the user device based on the second sensor data, determining whether the second situation meets requirements for radar-based gesture recognition, in response to a determination that the second situation meets the requirements for radar-based gesture recognition, inputting radar data obtained by the radar system into a model that determines radar-based gestures from the input radar data, and performing an operation in response to the model determining a radar-based gesture. The method is described in the case where it includes these steps.

[0005] This document also describes a computer-readable medium having instructions for performing the methods outlined above and other methods described herein, as well as systems and means for performing these methods.

[0006] This summary is provided to introduce a simplified concept of radar-based gesture recognition using situation-aware gating and other situation-aware controls, which will be further described below in the detailed description and the drawings. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used in defining the scope of the claimed subject matter.

[0007] Brief Description of the Drawings In this document, details of one or more aspects of radar-based gesture recognition using situation-aware gating and other situation-aware controls are described with reference to the following drawings. The same numbers are used throughout the drawings to refer to similar features and components.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6-1

Figure 6-2

Figure 6-3

Figure 6-4

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

DETAILED DESCRIPTION OF THE INVENTION

[0009] Detailed Description Overview This document describes a method and system for radar-based gesture recognition using context-aware gating and other context-aware controls. As an example, a user equipment (UE) (e.g., a computing device) includes a radar system for detecting user input among other uses. The UE receives sensor data from multiple sensors such as proximity sensors or motion sensors to develop the situation of the UE.

[0010] The situation defines the user activity, device characteristics, or operating environment of the UE. The situation can specify the orientation, acceleration, position, or proximity to an object. Location, temperature, brightness, pressure, and other environmental characteristics can also define the situation. The multiple sensors may include motion sensors such as an inertial measurement unit (IMU) for generating inertial data that defines the movement of the UE. The multiple sensors can include, among other examples, proximity sensors, light sensors, or temperature sensors. When the radar system operates in proximity mode (with gesture recognition enabled or disabled), the radar system is a proximity sensor. Particularly for UEs that rely on battery power, the UE may rely on sensors that provide accurate sensor data without consuming power as much as possible.

[0011] Based on the situation defined by the sensor data, the UE determines whether to prevent the radar system from recognizing radar-based gestures and / or whether to prevent recognized radar-based gestures from being used by components of the UE to perform functions. Gating the gesture recognition performed by the radar system prevents the waste of computing resources and power by the UE interpreting or performing functions (or even malfunctioning) in response to gesture recognition from unintended or non-user input.

[0012] Without gating, the UE may over-interpret radar-based gestures from radar data, thereby wasting computing resources by processing incorrect gestures or even malfunctioning in response thereto. By gating the output from the radar system based on the situation, the disclosed methods and systems enable the UE to save power, improve accuracy, enhance user satisfaction and usability, or reduce latency compared to other methods and systems for radar-based gesture recognition.

[0013] As an example, assume that sensor data obtained by a smartphone indicates that the user is holding the smartphone. The method and system enable the smartphone's radar system to recognize radar-based gestures in this situation. This is because there is a high likelihood that the user will input radar-based gestures to the smartphone while holding it. The sensor data then indicates that the user is also walking with the smartphone. The smartphone continues to recognize radar-based gestures using the radar system in this situation as well. This is because even while walking, the user may desire to intentionally gesture with the smartphone while holding it. The sensor data then indicates that although the user is still walking, they are no longer holding the smartphone, the smartphone is oriented away from the user, and / or the smartphone is blocked by an object (e.g., a backpack partition). Since the user will likely not interact with the smartphone while it is inside a backpack partition, for example, the method and system allow the smartphone to disable the radar system or at least adjust the radar system to prevent the radar system from being used to recognize radar-based gestures in this situation. When the smartphone recognizes a new situation, the smartphone re-evaluates whether to enable radar-based gesture recognition and enables radar-based gesture recognition by the radar system if the situation is appropriate for radar-based gesture recognition.

[0014] Finally, the user places the smartphone on a surface such as a desk, and the sensor data indicates that the user is not holding the smartphone and the smartphone is oriented with the screen facing up. If the proximity data indicates that the user is reaching for the smartphone, the smartphone selectively enables or prevents the radar system from recognizing radar-based gestures based on what the user will do next. If the smartphone detects a movement indicating that the user has picked up the smartphone after reaching for it, the smartphone uses the radar system to recognize radar-based gestures. If the smartphone does not detect a movement indicating that the user has picked up the smartphone after reaching for it (for example, if the user is holding a coffee cup next to the smartphone on the desk), the smartphone prevents gesture recognition using the radar system.

[0015] These are just a few examples of how the techniques and devices described can be used to gate radar-based gesture recognition. Other examples and realizations are described throughout this document. The document now turns to an exemplary operating environment, and then exemplary devices, methods, and systems are described.

[0016] Operating Environment FIG. 1 shows an exemplary environment 100 in which techniques for context-aware gating of radar-based gesture recognition and other context-aware controls can be implemented. The exemplary environment 100 includes a user equipment (UE) 102 (e.g., a smartphone) that includes or is associated with a radar system 104, a radar manager 106, a plurality of sensors 108, a motion manager 110, a state manager 112, an authentication system 114, and a display 116.

[0017] In an exemplary environment 100, the radar system 104 provides a radar field 118 by transmitting one or more radar signals or waveforms as described below with reference to FIGS. 7-9. The radar field 118 is a spatial volume from which the radar system 104 can detect reflections of radar signals and waveforms (e.g., radar signals and waveforms reflected from an object within the spatial volume, generally referred to herein as radar data). The radar system 104 also enables the UE 102 or other electronic devices to sense and analyze this radar data from reflections within the radar field 118 to recognize, for example, radar-based gestures (e.g., touch-independent gestures) performed by a user within the spatial volume. The radar field 118 may take on any of a variety of shapes and forms. For example, the radar field 118 may have a shape as described with reference to FIGS. 1 and 7. In other cases, the radar field 118 may take on a shape such as a radius extending from the radar system 104, a volume around the radar system 104 (e.g., a sphere, hemisphere, part of a sphere, beam, or cone), or a non-uniform shape (e.g., to account for interference from obstacles within the radar field 118). The radar field 118 may extend any of a variety of distances from the radar system 104, such as from a few inches to 12 feet (less than 1 / 3 meter to 4 meters). The radar field 118 may be predefined, selectable by the user, or otherwise determined (e.g., based on power requirements, remaining battery life, or another factor).

[0018] Reflections from the user 120 within the radar field 118 enable the radar system 104 to determine various information about the user 120, such as the position or pose of the user 120's body. This information may indicate various different non-verbal body language cues, body positions, or body poses, which can be recognized by the radar system 104 as touch-independent gestures performed by the user 120. These cues, positions, and poses may include the absolute position or distance of the user 120 relative to the UE 102, changes in the position or distance of the user 120 relative to the UE 102 (e.g., whether the user 120, or the user's hand, or an object held by the user 120 is moving closer to or farther away from the UE 102), the speed of the user 120 (e.g., hand, or non-user object) when moving towards or away from the UE 102, whether the user 120 is facing or turned away from the UE 102, whether the user 120 is leaning towards the UE 102, waving a hand, stretching a hand, or pointing at the UE 102, etc. These reflections can also be analyzed to determine or add confidence to the authentication of a human identity (e.g., the scattering center of the user's face) through the analysis of radar data. These reflections can be used by the UE 102 to define the situation (e.g., the operating environment of the UE 102) for situation-aware gating of radar-based gesture recognition and other situation-aware controls. These reflections can also be used to determine or add confidence to touch-independent gestures recognized by the radar system 104 when the user 120 provides input to the UE 102.

[0019] The radar manager 106 is configured to determine the intention of a user who intends to engage with, disengage from, or maintain engagement with the UE 102 based on radar data from the radar system 104. The user's intention can be estimated from various cues, positions, postures, and distances / speeds such as touch-independent gestures recognized by the radar system 104, for example, gestures of the arm or hand (e.g., extending the hand or arm towards the UE 102, swiping on the UE 102), eye gestures (e.g., eye movements looking at the UE 102), or head gestures (e.g., movements of the head or face oriented towards the UE 102). Regarding extending the hand or arm, the radar manager 106 determines whether the user is extending their hand or orienting their arm in a way that indicates an intention to touch or pick up the UE 102. Examples include extending the hand towards the volume button of a wirelessly attached speaker, extending the hand towards a wireless or wired mouse associated with a tablet computer, or extending the hand towards the UE 102 itself. This extension of the hand can be determined based on only the movement of the hand, based on the movement of the arm and hand, or based on bending and extending the arm in a manner that permits the hand of the arm to touch or grip the UE 102. including extending the hand towards the volume button of a wirelessly attached speaker, extending the hand towards a wireless or wired mouse associated with a tablet computer, or extending the hand towards the UE 102 itself.

[0020] The intention of the user who intends to participate can also be estimated based on the movement of the user's head or eyes to look at UE102 or, in some cases, the related peripheral devices of UE102, or to orient their face towards it. Regarding the eye movement of the user looking at UE102, the radar manager 106 determines that the user's eyes are looking in the direction of UE102 through, for example, tracking the user's eyes. Regarding the head movement (e.g., face orientation) of the user orienting their face towards UE102, the radar manager 106 determines that various points (e.g., scattering centers as described below) are currently oriented such that the user's face is facing towards UE102. Therefore, the user does not need to perform an action designed to control or activate UE102, such as activating (pressing) a button on UE102, or a touch-dependent gesture (e.g., on a touchpad or screen), or a touch-independent gesture (e.g., using the radar system 104), for the radar manager 106 to determine that the user intends to participate in (or intends to cancel or maintain participation in) UE102.

[0021] As described above, the radar manager 106 is also configured to determine the intention of a user who attempts to disengage from the UE 102. The radar manager 106 determines the intention of the user who attempts to disengage in the same way as the intention of the user who attempts to engage. However, the intention of the user who attempts to disengage is inferred from the radar data indicating the absence of touch-independent gestures, or the movement of the user's hand or arm away from the UE 102 (e.g., retracting), the eye movement away from the UE 102, or the movement of the head or face away from the UE 102 (e.g., a change in face orientation avoiding looking at the UE 102). Additional ways to determine the intention of the user who attempts to disengage include not only the reverse or cancellation of the above-described engagement, but also the radar data indicating that the user has walked away, moved away from their own body, or engaged with another unrelated object or device. For this reason, the radar manager 106 may determine the intention to disengage from the UE 102 based on determining the intention of the user who attempts to engage with some other object, device, or user equipment. For example, assume that the user is looking at a smartphone and interacting with it. Examples of the intention to engage that indicate the intention to disengage from the smartphone include the user looking at a TV screen instead of the smartphone, starting to talk to a person physically nearby, or reaching for another device, such as an e-book or media player, whose engagement is thought to replace the engagement with the smartphone.

[0022] The radar manager 106 is also configured to determine the intention of a user who attempts to maintain engagement with the UE 102. This maintenance of engagement can be either active or passive. For active engagement, the radar manager 106 may determine, based on radar data, that the user is interacting through touch-independent gestures or the like. The radar manager 106 may similarly or alternatively determine active engagement through non-radar data (e.g., with the assistance of other components of the UE 102). These non-radar data include indications that the user is inputting data into the UE 102 or a peripheral device, or is controlling the UE 102 or a peripheral device. Thus, through touch, typing, or voice data, the user is determined to be touching through touch screen input (e.g., tapping on a soft keyboard or performing a gesture), typing on a peripheral keyboard, or dictating voice input. For passive maintenance of engagement, the radar manager 106 determines, alone or with the assistance of other components of the UE 102, that the user is holding the UE 102 in a manner such that, for example, the user is facing the UE 102 with their face, looking at the display 116, or orienting the display of the UE 102 to be visible to the user or a third party, consuming content, or providing the UE 102 to others for consuming content. Other examples of passive engagement maintenance include the radar manager 106 determining that the user 120 is in the vicinity of the UE 102 (e.g., within a range of 2 meters, 1.5 meters, 1 meter, or 0.5 meter from the UE 102), including the presence of the user. Details of exemplary ways in which the radar manager 106 determines the intention of a user who attempts to engage, disengage, or maintain engagement are described below. It is determined that the user is touching through touch screen input (e.g., tapping on a soft keyboard or performing a gesture), typing on a peripheral keyboard, or dictating voice input. For passive maintenance of engagement, the radar manager 106 determines, alone or with the assistance of other components of the UE 102, that the user is holding the UE 102 in a manner such that, for example, the user is facing the UE 102 with their face, looking at the display 116, or orienting the display of the UE 102 to be visible to the user or a third party, consuming content, or providing the UE 102 to others for consuming content. Other examples of passive engagement maintenance include the radar manager 106 determining that the user 120 is in the vicinity of the UE 102 (e.g., within a range of 2 meters, 1.5 meters, 1 meter, or 0.5 meter from the UE 102), including the presence of the user. Details of exemplary ways in which the radar manager 106 determines the intention of a user who attempts to engage, disengage, or maintain engagement are described below.

[0023] Furthermore, the radar manager 106 may also use the radar data from the radar system 104 to determine gestures made by the user. These gestures can involve the user touching some surface such as a table, the display 116, or the sleeve of their shirt, or may involve touch-independent gestures. Touch-independent gestures can be performed in the air, three-dimensionally, and / or without the hand or finger touching the input device, but are not prevented from touching some object. These gestures are recognized or determined based on the radar data acquired by the radar system 104 and are then output to the application or other subscribers running on the UE 102, or can be used as an input for performing an operation, for example, to indicate involvement with the UE 102.

[0024] Exemplary gestures include those similar to sign language (e.g., ASL (American Sign Language)), which can be various and complex one-handed or two-handed gestures, or simple two-handed or one-handed gestures, such as swiping left, right, up, or down, raising or lowering the flat hand (e.g., raising or lowering the volume of the UE 102 or the volume of a TV or stereo controlled through the UE 102), or swiping forward or backward (e.g., from left to right or from right to left) to change music and video tracks, snooze an alarm, hang up a phone, or even play a game. These are just a few of the many exemplary gestures and functions that can be controlled by these gestures and enabled via the radar system 104 and the radar manager 106. Thus, although this document is directed in some aspects towards engagement and state management, it should not be misinterpreted to mean that nothing in this document can use aspects of engagement and state management to additionally or alternatively configure the radar system 104 and the radar manager 106 to perform gesture recognition.

[0025] The display 116 may include any suitable display device such as a touch screen, a liquid crystal display (LCD), a thin film transistor (TFT) LCD, an in-plane switching (IPS) LCD, a capacitive touch screen display, an organic light emitting diode (OLED) display, an active-matrix organic light-emitting diode (AMOLED) display, a super AMOLED display, and the like. As described, the display 116 can be powered at full color when power is supplied to the touch input, at reduced color when power is not supplied to the touch input, and at various levels of low color and low power (e.g., gray clock) or no power.

[0026] The plurality of sensors 108 can be any of a variety of sensor devices configured to generate sensor data indicative of the situation of the UE 102, or, in other words, the operating environment or ambient signature of the UE 102. The plurality of sensors 108 includes an inertial measurement unit (IMU) for measuring movement, where movement is defined herein to include specific forces, angular velocities, orientations, vibrations, accelerations, velocities, and positions, including pitch, roll, and yaw for each of three axes (e.g., X, Y, and Z). The IMU is merely an example of a sensor 108. Other examples of the plurality of sensors 108 for sensing movement include accelerometers, gyroscopes, and / or magnetometers. The plurality of sensors 108 can include proximity sensors, optical sensors, positioning sensors, compasses, temperature sensors, barometric pressure sensors, or other sensors that detect the presence or proximity of an object. The plurality of sensors 108 may include a proximity sensor such as a radar system 104 that operates in a proximity mode as opposed to a gesture recognition mode or another mode.

[0027] UE 102 may primarily rely on battery power, and thus, the plurality of sensors 108 may exclude high-power sensors such as cameras and instead may primarily include low-power sensors that provide accurate sensor data for developing an accurate situation. By avoiding the use of high-power sensors such as cameras to drive gating decisions and instead using sensor data from low-power sensors such as IMUs, UE 102 operates more efficiently and uses less power to make gating decisions compared to when a camera is used.

[0028] Motion manager 110 is configured to determine the motion of UE 102 based on inertial data or other sensor data obtained from sensors 108. The motion manager is configured to determine the motion of UE 102 to enable UE 102 to define a situation. Exemplary motions include UE 102 being lifted (e.g., being held in a hand), being oriented towards or away from user 120, and vibrations. Exemplary motions may include the cessation of physical contact of UE 102 by user 120, the placement of UE 102 on a non-biological object (e.g., a table, an automotive console, an arm of a bench, a pillow, a floor, a docking station), and the placement of UE 102 within an enclosed receptacle such as a pocket, a bag, or a handbag. Further exemplary motions include motions indicating that UE 102 is being held, motions indicating that UE 102 is being held by a person walking, riding a bicycle, riding in a vehicle, or moving in some other manner, motions indicating how UE 102 is being held (e.g., the carry orientation is landscape, portrait up, portrait down, or a combination thereof). Exemplary motions further include motions indicating that UE 102 is not being held or is being carried but not being held by a walking person.

[0029] These movements may indicate potential user engagement with, disengagement from, or maintenance of engagement with UE102. For example, the movement of UE102 may indicate that the user equipment is moving towards or oriented towards user 120, or is being moved / oriented away from user 120, that the user equipment is moving too quickly or changing its movement too quickly to be interacting for many possible types of user engagement, that the user equipment is being held by user 120 (through natural human movement, breathing, heartbeat), or that it is vibrating due to a mechanical source or non-user source (e.g., vehicle vibration, ambient sound shaking UE102, music vibrating UE102). Thus, orienting away may indicate potential disengagement from UE102, but may include a change in orientation of UE102 such that a previous orientation where user 120 was thought to be looking at display 116 is no longer the case. User 120 Typing or reading in one orientation and then flipping the phone over or turning sideways, or putting it in a pocket, etc. are just examples of movements that indicate orienting away and thus potential disengagement. Exemplary movements that may indicate maintenance of engagement include the user maintaining the hold or placement of UE102, or vibrations indicating that the user is maintaining their orientation with respect to UE102 that previously indicated or was consistent with engagement with UE102.

[0030] Radar system 104 relies on radar manager 106, movement manager 110, and sensor 108 to define the situation of UE102 used to drive the gating decisions made by radar system 104. Sensor data generated by sensor 108 is combined with the movement and user intent determined by movement manager 110 and radar manager 106 to help define the situation of UE102.

[0031] The movement determined by the movement manager 110 may indicate how the user 120 is interacting with the UE 102 or whether the user 120 is interacting with the UE 102. The acceleration or vibration detected by the movement manager 110 can correspond to similar vibrations and accelerations observed when the user 120 is walking or moving in some other manner with the UE 102, and thus the movement may indicate how the user is walking or moving. The change in movement determined by the movement manager 110 can indicate changes in the carried position and orientation as additional information about how the user 120 is interacting with the UE 102 or whether the user 120 is interacting with the UE 102. The pattern of movement or lack of movement inferred by the movement manager 110 may be similar to the movement typically observed when the user 120 is looking at or holding the UE 102 in a held situation. The movement and pattern of movement can indicate an enclosed situation where the UE 102 is in a pocket of the clothing the user 120 is wearing, or a backpack, or a briefcase, or an overhead storage bin on an airplane or train, or a vehicle console or glove box, or other storage enclosure.

[0032] The situation can be defined by other information beyond movement. For example, the proximity sensor or radar system 104 can detect whether the radar system 104 (or other part of the UE 102) is blocked by an object near the UE 102. Evidence of blockage may indicate that the UE 102 is in an enclosed situation, and the lack of blockage may indicate otherwise. Other sensors such as ambient light sensors, barometers, location sensors, optical sensors, infrared sensors, etc. can provide the UE 102 with signals that further define the operating environment or situation of the UE 102 to improve gesture recognition and other described techniques. Altitude, shadow, ambient sound, ambient temperature, etc. are further examples of signals that can be captured by the radar system 104 through the sensor 108 to enable the UE 102 to define the situation.

[0033] The state manager 112 manages the state of the UE 102, such as the power state, access state, and information state, and in some examples, manages the state based on the situations defined above. This management of the UE 102 and its components is performed based in part on the decisions made by the radar manager 106 and the motion manager 110, the sensor data from the sensor 108, and the situations defined therefrom. For example, the state manager 112 can change the display 116 of the UE 102 to power up in anticipation of receiving a touch input from the user 120 to enter a password, change the computer processor to perform the calculations used for authentication, or change the imaging system to perform image-based face authentication, or manage the power to the components of the authentication system 114 by changing the radar (e.g., radar system 104) or other components. The state manager 112 may instruct the radar manager 106 to put the radar system 104 in the proximity mode or the disabled mode when the radar-based gesture recognition is gated, and to put the radar system 104 in the enabled mode or the gesture recognition mode when the radar-based gesture recognition is not gated.

[0034] As described, this management of UE102 is based on the determinations by the radar manager 106 and the motion manager 110, which respectively determine an intention to engage with UE102, an intention to disengage from engaging with UE102, or an intention to maintain such engagement and the movement of UE102. The state manager 112 can do so based only on these determinations or also on other information that defines the situation of UE102, including the current state, the current engagement, the applications in execution, and the content indicated by these applications. By considering the situation, the state manager 112 can improve the accuracy, robustness, and speed of the overall determination that the user's intention is to engage with UE102, disengage from engaging with UE102, or maintain such engagement.

[0035] The "multiple determinations" (e.g., the determinations of the radar manager 106 and the motion manager 110) for defining the situation may be made simultaneously or stepwise as part of the state management of the UE 102, or only one of them may be used. For example, assume that the UE 102 is in a low-power state for the components used for authentication. The radar manager 106 may determine that the user 120 intends to authenticate with the UE 102 based on the movement towards the UE 102 or reaching out towards the UE 102. In some cases, this alone may be considered by the state manager 112 to be an insufficient situation for changing the UE 102 to a high-power state (e.g., for authenticating or interpreting radar-based gestures). Therefore, the state manager 112 can power up some of the authentication components to an intermediate state instead of a high-power state (e.g., the high-power state 504-1 in FIG. 5). For example, if the authentication system 114 uses an infrared sensor to perform face recognition, the state manager 112 can power these sensors and the display 116 up to a higher power, expecting to authenticate the user and indicate to the user that the UE 102 is "awake" and thus more responsive soon. As an additional step, the state manager 112 can wait until the motion manager 110 determines that the situation indicates that the user has moved, picked up, lifted, etc. the UE 102 before fully powering up the authentication components, here the infrared sensor. Optionally, the state manager 112 may enable authentication to be attempted by the components without further input from the user, thereby making the authentication seamless for the user 120.

[0036] However, in some cases, the state manager 112 determines to power up the UE 102 or prepare the state of the UE 102 in another way in response to both the inertial data and the radar data, for example, the radar manager 106 determines that the user intends to interact with the UE 102, and the motion manager 110 determines that the user has picked up the UE 102.

[0037] Thus, the state manager 112 can wait until a higher level of confidence is obtained that the user's intention is to interact with the UE 102 by picking up the UE 102, like the signal from the motion manager 110 that the user has just started touching the UE 102. In such a case, the state manager 112 may increase the power based only on the determination of the radar manager 116, but instead of waiting until the motion manager 110 indicates user contact to sufficiently power the display or authentication system 114 or its components, it may increase the power to an intermediate power level for these components. However, as described, the state manager 112 may change the state to a higher power level based only on the determination of the intention to interact based on the radar data, or may lower those levels based only on the determination of the intention to discontinue the interaction based on the radar data. One of the many exemplary ways in which the state manager 112 can manage the state of the UE 102 is shown in the exemplary environments 100-1, 100-2, and 100-3 of FIG. 1.

[0038]

[0039] ​In environment 100-1, assume that user 120 is authenticated and UE102 is in a high-level power state, access state, and information state. This authentication is shown to user 120 through display 116 which shows a high-color and high-luminance star symbol (shown as 122 in environment 100-1). In environment 100-1, user 120 places UE102 on the table. Placing this UE102 on the table causes sensor 108 to sense inertial data, which is then provided to motion manager 110. Based on this inertial data, motion manager 110 determines that although UE102 was moving, it is now stationary and placed. UE102 was in a moving situation but is now in a stationary situation. In this regard, motion manager 110 may pass this motion determination to radar system 104, radar manager 106, or state manager 112. In any of the three cases, this is a data point for determining whether to lower the state from high level to intermediate or low level. As described, lowering these states can save power, keep information secret and access secure, and still provide a seamless user experience for user 120. For example, based on the motion determination, radar system 104 forms a situation for managing its output. In environment 100-1 where user 120 is not holding UE102 but some slight movement is detected, radar system 104 can determine that environment 100-1 is an environment where UE120 is placed on a flat and stationary surface. Since the situation for UE120 is a stationary situation except for some slight movement, the situation meets the requirements for radar-based gesture recognition.

[0040] Continuing with this example, consider environment 100-2 where user 120 withdraws their hand from UE102. This withdrawal is sensed and analyzed by radar system 104 and radar manager 106 respectively. By doing so, radar manager 106 determines that user 120 intends to disengage from UE102. Based on this determination from radar manager 106 and the movement determination from movement manager 110, state manager 112 may lower one or more of the states of UE102. Here, this lowering is intended to correspond to the level of user 120's engagement with UE102. This lowering by state manager 112 is a reduction to an intermediate power level due to a reduction in the saturation and luminance of display 116, indicated by the star symbol of low saturation and luminance (shown at 124). Note that state manager 112 can lower the state of power, access, and / or information to a low level, but here, state manager 112 lowers the state to an intermediate level. This is because the intention to disengage from radar manager 106 indicates that although user 120 has withdrawn their arm, their body is still oriented towards UE102 and user 120 is still looking at UE102. This is an example of adapting the state to the user's engagement. Because the withdrawal indicates a certain level of disengagement, but in itself, it represents some level of continued engagement or some level of uncertainty in the determination of disengagement by radar manager 106. For example, the determination of withdrawal can be used as proximity information to define a second situation of UE102. Radar system 104 forms a second situation to manage its gesture recognition ability in environment 100-2. If the withdrawal is detected without movement towards UE102, radar system 104 can determine that environment 100-2 is an enclosed situation that does not meet the requirements for touch-independent gesture recognition. This is because the situation indicates that the device is currently out of reach of user 120's hand. This is to show some level of uncertainty in the determination of disengagement by radar manager 106. For example, the determination of withdrawal can be used as proximity information to define a second situation of UE102. Radar system 104 forms a second situation to manage its gesture recognition ability in environment 100-2. If the withdrawal is detected without movement towards UE102, radar system 104 can determine that environment 100-2 is an enclosed situation that does not meet the requirements for touch-independent gesture recognition. This is because the situation indicates that the device is currently out of reach of user 120's hand.

[0041] To conclude this example, consider environment 100-3. Here, user 120 is reading a book with UE102 placed on the table. User 120 moves their body away from UE102 at an angle and orientates towards their book, looking at the book instead of UE102. Based on this additional information about user 120's orientation, radar manager 106 determines that user 120 intends (and has perhaps already) disengaged from UE102. In this regard, radar manager 106 provides this additional determination of intent to disengage to state manager 112, which then reduces the state of UE102 to a lower level indicated with lower power usage on display 116 (showing only the time in low brightness and color at 126). Although not shown, state manager 112 also de-authenticates user 120 (e.g., locks UE102). This additional information about user 120's orientation, or the determination that user 120 intends to disengage from UE102, can be used as proximity information to define a third situation for UE102. Radar system 104 forms a third situation in response to detecting the intent to disengage and read the book. Radar system 104 can determine that environment 100-3 is not a situation that meets the requirements for radar-based gesture recognition. This is because the situation indicates that the device is currently outside the range of influence of user 120's proximity.

[0042] As shown in this example, the techniques described herein can manage the state of a user equipment to provide a seamless user experience related to authentication and radar-based gesture recognition. These techniques enable this with reduced power consumption and better privacy and security compared to other techniques. State management can achieve maintaining or raising levels of power, access, and information. As further shown, without gating, UE102 may over-interpret radar-based gestures from the radar data acquired by radar system 104, thereby wasting computational resources by processing and then discarding incorrect gestures. By gating the gesture recognition of radar system 104 based on the situation, the disclosed techniques and systems enable UE102 to save power, improve accuracy, or reduce the latency in interpreting and responding to radar-based inputs compared to other techniques and systems for radar-based gesture recognition.

[0043] More specifically, consider an example of authentication system 114 shown in FIG. 2. This is merely an example, because only a few examples are mentioned, other authentication systems controllable by state manager 112, such as password input via a touch-sensitive display, radar authentication using radar system 104, or a fingerprint reader, can be considered.

[0044] This example of authentication system 114 is illustrated showing the interior 200 of UE102 (shown as a smartphone). In the illustrated configuration, UE102 includes radar integrated circuit 202 of radar system 104, speaker 204, front camera 206, proximity sensor 208 and ambient light sensor 210 as examples of sensors 108. Further As an example, UE102 also includes a face recognition unlock sensor 212, which includes a near-infrared (NIR) projector 214 and a near-infrared (NIR) dot It includes a projector 216, and both of them project infrared light or near-infrared light to the user. The face recognition unlocking sensor 212 also includes two NIR cameras 218-1 and 218-2 positioned on both sides of the UE102. The NIR cameras 218-1 and 218-2 sense the infrared light and near-infrared light reflected by the user. This reflected near-infrared light can be used to determine facial features and, based on comparison with previously stored facial feature information, determine whether the user is genuine. The NIR projector illuminator 214, for example, "projects" NIR light into the environment, which provides an image as soon as it receives reflections from the user (and other objects). This image includes the user's face even when the ambient light is low or absent, and thus can be used to determine facial features. The NIR dot projector 216 provides NIR light reflections that can be analyzed to determine the depth of an object including the features of the user's face. Thus, a depth map (e.g., a spectral depth map) of the user may be created (e.g., in advance when setting up face authentication), and the current depth map may be determined and compared with the previously created and stored depth map. This depth map helps prevent the authentication of a photo or other 2D rendering of the user's face (rather than the actual face of a person).

[0045] This mapping of the user's facial features can be securely stored on the UE102 and, based on the user's preferences, can be made secure on the UE102 and prevented from being made available to external entities.

[0046] The authentication system 114 includes the face recognition unlocking sensor 212, but may also include other components such as the front camera 206, the proximity sensor 208, and the ambient light sensor 210, as well as a processor for analyzing data, a memory (which may also have multiple power states) for storing, caching, or buffering sensor data, etc.

[0047] The face authentication unlock sensor 212 senses IR (infrared) and NIR (near-infrared) data to perform face recognition, which is one way of authenticating the user and thus changing the access state (e.g., unlocking the UE 102) as described in the method explained below. To conserve power, the face authentication unlock sensor 212 operates in a low-power state when not in use (it can also simply be turned off). In particular, the NIR light projector 214 and the NIR dot projector 216 do not emit radiation when in the off state. However, a warm-up sequence associated with the transition from a low or no-power state to an intermediate and / or high-power state can be used for the NIR light projector 214 and the NIR dot projector 216. By powering up one or both of these components, the waiting time when authenticating the user can sometimes be reduced by more than 0.5 seconds. Considering that many users authenticate their devices dozens or even hundreds of times a day, this can save the user's time and improve the user experience. As described here, this time delay is reduced by the radar manager 106 determining that the user intends to interact with their device based on the radar data provided by the radar system 104. This is managed by the state manager 112. In practice, the method anticipates and detects the intention of the user attempting to interact and initiates the warm-up sequence. The method may or may not do so before the user touches the UE 102, but this is not essential. This enables the NIR light projector 214 and the NIR dot projector 216 to be sufficiently powered for use when authenticating the user, which reduces the time spent by the user waiting for face recognition to complete.

[0048] Before proceeding to other components of UE102, consider the scenario of the face recognition unlock sensor 212. This exemplary component of the authentication system 114 can authenticate a user using face recognition at an angle of only 10 degrees with respect to the plane of the display 116. Thus, the user does not need to pick up the phone and orient the sensor at an angle such as 70-110 degrees or 80-100 degrees to their face. Instead, the authentication system 114 is configured to use the face recognition unlock sensor 212 to authenticate the user even before the user picks up UE102. This is shown in FIG. 3, which depicts a user 120 in a state where the part of their face used for face recognition (e.g., their chin, nose, or cheekbones) can be positioned at an angle of only 10 degrees with respect to the plane 304 of the display 116. It is also shown that the user 120 can be authenticated while their face is more than 1 meter away from the face recognition unlock sensor 212, as indicated by the face distance 306. By doing so, the approach allows for a nearly seamless and rapid authentication even if UE102 is oriented upside down or at an odd angle.

[0049] More specifically, consider FIG. 4, which shows an exemplary implementation 400 of UE102 that can implement techniques for authentication management via an IMU and a radar (including a radar manager 106, a motion manager 110, and a state manager 112). UE102 in FIG. 4 is illustrated with various exemplary devices including UE102-1, a tablet 102-2, a laptop 102-3, a desktop computer 102-4, a computing watch 102-5, computing glasses 102-6, a gaming system 102-7, a home automation and control system 102-8, and a microwave oven 102-9. UE102 can also include other devices such as a television, an entertainment system, an audio system, an automobile, a drone, a trackpad, a drawing pad, a netbook, an e-reader, a home security system, and other household appliances. Note that UE102 can be wearable, or it can be mobile but non-wearable, or relatively non-mobile (e.g., a desktop and a device).

[0050] UE102 includes an inertial measurement unit 408 as an example of the sensor 108 described above. An exemplary overall horizontal dimension of UE102 can be, for example, about 8 centimeters by about 15 centimeters. An exemplary installation area of the radar system 104 can be even more restricted, such as about 4 millimeters by 6 millimeters including the antenna. The requirements for such a restricted installation area for the radar system 104 to accommodate many other desirable features of UE102 within such a space - restricted package, combined with power limitations and processing limitations, can result in a compromise in the accuracy and effectiveness of radar - based gesture recognition. However, at least some of these compromises can be overcome in view of the teachings herein.

[0051] UE102 also includes one or more computer processors 402 and one or more computer - readable media 404, and the computer - readable media 404 includes a memory medium and a storage medium. Applications and / or operating systems (not shown) implemented as computer - readable instructions on the computer - readable media 404 can be executed by the computer processor 402 to provide some or all of the functionality described herein, such as some or all of the functionality of the radar manager 106, the motion manager 110, and the state manager 112 (shown within the computer - readable media 404 but this is not essential).

[0052] UE102 may also include a network interface 406. UE102 can use the network interface 406 to communicate data through a wired network, a wireless network, or an optical network. By way of example and not limitation, the network interface 406 is a local area network Data may be communicated through a (local-area-network:LAN), wireless local-area-network (wireless local-area-network:WLAN), personal-area-network (personal-area-network:PAN), wide-area-network (wide-area-network :WAN), intranet, Internet, peer-to-peer network, point-to-point network, or mesh network.

[0053] In some scenarios, the radar system 104 is at least partially implemented in hardware. Various realizations of the radar system 104 may include a System-on-Chip (SoC), one or more Integrated Circuits (ICs), a processor configured to have or access processor instructions stored in memory, hardware with embedded firmware, a printed circuit board with various hardware components, or any combination thereof. The radar system 104 operates as a monostatic radar by transmitting and receiving its own radar signals. In some realizations, the radar system 104 may also cooperate with other radar systems 104 in the external environment to implement a bistatic radar, a multistatic radar, or a network radar. However, the constraints or limitations of the UE102 may affect the design of the radar system 104. The UE102 may have, for example, limitations on the power available to operate the radar, limitations on computing power, size constraints, layout constraints, an external housing that attenuates or distorts radar signals, etc. The radar system 104 includes several features that enable high radar functionality and performance to be achieved in the presence of these constraints, as further described below.

[0054] Before describing additional exemplary ways in which the state manager 112 can operate, consider FIG. 5, which shows many of the information states, power states, and access states that are managed by the state manager 112 and in which the UE 102 can operate.

[0055] FIG. 5 shows access states, information states, and power states in which the UE 102 can operate, each of which can be managed by the techniques described. These exemplary levels and types of device state 500 are shown at three levels of granularity for visual simplicity, but for access state 502, power state 504, and information state 506, many levels of each are contemplated. The access state 502 is shown at three exemplary levels of granularity: high access state 502-1, intermediate access state 502-2, and low access state 502-3. Similarly, the power state 504 is shown at three exemplary levels of granularity: high power state 504-1, intermediate power state 504-2, and low power state 504-3. Similarly, the information state 506 is shown at three exemplary levels of granularity: high information state 506-1, intermediate information state 506-2, and low information state 506-3.

[0056] More specifically, the access state 502 pertains to the access rights to the data, applications, and functions of the UE102 that are available to the user of the device. This access can be elevated and is sometimes referred to as the "unlocked" state for the UE102. This high access level may simply include the device's applications and functions, or it may also include access to various accounts such as bank accounts, social media accounts, etc. that are accessible through the UE102. Many computing devices such as the UE102 require authentication to provide high access such as the high access state 502-1. Various intermediate access levels (e.g., 502-2) can be permitted by the UE102, and such states allow the user to access only a part rather than all of the UE102's accounts, services, or components. Examples include permitting the user to take photos but not permitting access to previously imported photos. Another example includes permitting the user to go out on the phone but not permitting access to the contact list when making a call. These are just some of the many intermediate rights that the UE102 can permit, as indicated by the intermediate access state 502-2. Finally, the access state 502 may refrain from granting access, as shown as the low access state 502-3. In this case, the device may be turned on and may send notifications such as alarms to wake the user, but it may be configured not to permit access to the functions of the UE102 (or the UE102 may simply be turned off and thus not permit access). This includes permitting the user to go out on the phone but not permitting access to the contact list when making a call. These are just some of the many intermediate rights that the UE102 can permit, as indicated by the intermediate access state 502-2. Finally, the access state 502 may refrain from granting access, as shown as the low access state 502-3. In this case, the device may be turned on and may send notifications such as alarms to wake the user, but it may be configured not to permit access to the functions of the UE102 (or the UE102 may simply be turned off and thus not permit access).

[0057] The power state 504 is shown at three exemplary granularity levels: a high power state 504-1, an intermediate power state 504-2, and a low power state 504-3. The power state 504 pertains to the amount of power to one or more components of the UE 102, such as the radar system 104, the display 116, or other power-consuming components, e.g., a processor, a camera, a microphone, a voice assistant, a touch screen, a sensor, a radar, and components that are part of the authentication system 114 (which may include the aforementioned components listed in the same manner). In the context of powering up components and the power state 504, terms such as powering, powering up, increasing power, decreasing power, etc. refer to controlling a power-management integrated circuit (PMIC); managing power rails extending from the PMIC; opening and closing switches between the power rails, the PMIC, and one or more circuit components (e.g., the NIR components, cameras, displays, and radars described); and providing supply voltages to operate the components accurately and safely, which may include increasing, decreasing, or distributing the applied voltage or managing the current inflow. This may include controlling a power-management integrated circuit (PMIC); managing power rails extending from the PMIC; opening and closing switches between the power rails, the PMIC, and one or more circuit components (such as the NIR components, cameras, displays, and radars described); and providing supply voltages to operate the components accurately and safely, which may include increasing, decreasing, or distributing the applied voltage or managing the current inflow.

[0058] For the radar system 104, the power state 504 can be reduced by collecting radar data at different duty cycles (for example, lower frequencies may use less power and higher frequencies may use more power), turning off components when they are not active, or adjusting the power amplification level. By doing so, the radar system 104 may use power of about 90 mW in the high power state 504-1, 30 - 60 mW in the intermediate power state 504-2, or less than 30 mW in the low power state 504-3 (for example, the radar system 104 can operate at 2 - 20 mW while still providing some usable radar data such as the presence of a user). Each of these levels of power usage permits different resolutions and ranges. Additional details regarding the power management of the radar system 104 (and UE 102) are described with reference to FIG. 6-1.

[0059] In situations where the above states are changed, the state manager 112 may power up or power down various components of the UE 102 based on the decisions of the radar manager 106 and the motion manager 110.

[0060] For example, the state manager 112 can change the power of the authentication system 114 or the display 116 from a lower power state (for example, from the low power state 504-3 to the intermediate power state 504-2 or from either of these to the high power state 504-1). By doing so, the UE 102 may engage with or authenticate the user more quickly or easily. For this purpose, the state manager 112 may change the power state 504 to be higher or lower than the current power for the system of the UE 102 or for a particular power consuming entity associated with the UE 102. Exemplary components are further described as part of FIG. 2 above, the face recognition unlock sensor 212 and its components, the NIR projection illuminator 2 Powering up or down the 14 and NIR dot projector 216, as well as the NIR cameras 218-1 and 218-2, including reducing the power to these components, the display, the microphone, the touch input sensors, etc.

[0061] A third exemplary state of UE102 is information state 506, which is exemplified by a high information state 506-1, an intermediate information state 506-2, and a low information state 506-3. More specifically, information state 506 is related to the amount of information provided to a user, such as user 120 in FIG. 1. In a notification situation, high information state 506-1 generally provides the highest level of information, assuming that UE102 is unlocked or authenticated in some other way, or has user preferences for providing high-level information even without authentication. An example for high information state 506-1 includes showing the name, number, and even related images of the caller when a call is received. Similarly, when a text or email or other type of message is received, the content is automatically presented through display 116 or a voice speaker, peripheral device, etc. This represents a high level of engagement, but the user's preferences can determine what kind of engagement is required. Here, there is some correlation between the user's engagement and the amount of information provided, and thus it is assumed that the method can adjust the presented information according to that determination by judging the engagement. An example of information reduction, such as intermediate information state 506-2, includes presenting a ringtone when a call is received but not presenting the caller's name / identification information, indicating that a text message or email has been received but showing only the subject, or only the address, or only a part of the body content. Low information state 506-3 presents little or no information personally relevant to user 120, but may include information that is comprehensive, such as the current date, time, weather conditions, battery power status, or indicating that UE102 is on, or information that is widely considered to be common sense or non-confidential matters. Another example of low information state 506-3 is when a text message is received with an audible tone "ping" that simply indicates that a message has been received, or when a ringtone for a call is received but the caller's name, number, or other information is not received, including a blank or black screen.

[0062] FIG. 6-1 shows an exemplary implementation 600 of the radar system 104. In Example 600, the radar system 104 includes at least one of each of the components: a communication interface 602, an antenna array 604, a transceiver 606, a processor 608, and a system medium 610 (e.g., one or more computer-readable storage media). The processor 608 may be implemented as a digital signal processor, a controller, an application processor, other processors (e.g., the computer processor 402 of the UE 102), or some combination thereof. The system medium 610, which may be included within or separate from the computer-readable medium 404 of the UE 102, includes one or more of the modules: an attenuation reducer 614, a digital beamformer 616, an angle estimator 618, a power management module 620, or a gesture recognition module 621. These modules can compensate for or mitigate the effects of the integration of the radar system 104 within the UE 102, thereby enabling the radar system 104 to recognize small or complex gestures, distinguish different user orientations (e.g., "reach" (extending the hand)), continuously monitor the external environment, or achieve a target false alarm probability. With these features, the radar system 104 can be implemented within a variety of different devices such as the device shown in FIG. 4.

[0063] Using the communication interface 602, the radar system 104 can provide radar data to the radar manager 106. The communication interface 602 can be a wireless or wired interface based on whether the radar system 104 is implemented separate from or integrated within the UE 102. Depending on the application, the radar data can be raw or minimally processed data, in-phase and quadrature (I / Q) data, range-Doppler data and the like. The processed data may include target position information (such as range, azimuth angle, elevation angle), clutter map data, etc. Generally, the radar data includes information that can be used by the radar manager 106 to provide the state manager 112 with the intention of the user who intends to participate, intends to cancel participation, or intends to maintain participation.

[0064] The antenna array 604 includes at least one transmitting antenna element (not shown) and at least two receiving antenna elements (such as those shown in FIG. 7). In some cases, the antenna array 604 may include multiple transmitting antenna elements to implement a multiple-input multiple-output (MIMO) radar that can transmit multiple distinct waveforms (for example, different waveforms for each transmitting antenna element) at once. The use of multiple waveforms can improve the measurement accuracy of the radar system 104. For implementations including three or more receiving antenna elements, the receiving antenna elements can be positioned in a one-dimensional shape (such as a line) or a two-dimensional shape. The one-dimensional shape enables the radar system 104 to measure one angular dimension (for example, azimuth angle or elevation angle), while the two-dimensional shape enables two angular dimensions (for example, both azimuth angle and elevation angle) to be measured. An exemplary two-dimensional arrangement of the receiving antenna elements is further described with respect to FIG. 7.

[0065] FIG. 6-2 shows an exemplary transceiver 606 and a processor 608. The transceiver 606 includes a plurality of components that can be individually turned on or off via a power management module 620 according to the operating state of the radar system 104. Note that the power management module 620 may be remote, integrated, or under the control of the state manager 112, such as when the state manager 112 powers up or down components (e.g., authentication system 114) used to authenticate the user. The transceiver 606 includes an active component 622, a voltage-controlled oscillator (VCO) and voltage-controlled buffer 624, a multi plexer 626, an analog-to-digital converter (ADC) 628, a phase lock loop (PLL) 630, and a crystal oscillator 63 2, and is shown to include at least one of each of these components. When turned on, each of these components consumes power, even if the radar system 104 is not actively using these components to transmit or receive radar signals. The active component 622 may include, for example, an amplifier or filter coupled to a supply voltage. The VCO 624 generates a frequency-modulated radar signal based on a control voltage provided by the PLL 630. The crystal oscillator 632 generates a reference signal for signal generation, frequency conversion (e.g., upconversion or downconversion), or timing operations within the radar system 104. By turning these components on or off, the power management module 620 enables the radar system 104 to quickly switch between active and non-active operating states and conserve power during various non-active periods. These non-active periods may be on the order of microseconds (μs), milliseconds (ms), or seconds (s).

[0066] Processor 608 is shown to include a plurality of processors that consume different amounts of power, such as low-power processor 608-1 and high-power processor 608-2. As an example, low-power processor 608-1 may include a processor embedded within radar system 104, and the high-power processor may include computer processor 402 outside of radar system 104 or some other processor. The difference in power consumption may be due to different amounts of available memory or computing power. For example, low-power processor 608-1 may utilize less memory, perform less computation, or utilize simpler algorithms compared to high-power processor 608-2. Despite these limitations, low-power processor 608-1 can process data for less complex radar-based applications such as proximity detection or motion detection (based on radar data rather than inertial data). In contrast, high-power processor 608-2 may utilize large amounts of memory, perform large amounts of computation, or execute complex signal processing, tracking, or machine learning algorithms. High-power processor 608-2 can process data for radar-based applications that are in the spotlight, such as gesture recognition or face recognition (for authentication system 114), and can provide accurate and high-resolution data through angle ambiguity resolution or identification of multiple users and the characteristics of those users. For example, low-power processor 608-1 may utilize less memory, perform less computation, or utilize simpler algorithms compared to high-power processor 608-2. Despite these limitations, low-power processor 608-1 can process data for less complex radar-based applications such as proximity detection or motion detection (based on radar data rather than inertial data). In contrast, high-power processor 608-2 may utilize large amounts of memory, perform large amounts of computation, or execute complex signal processing, tracking, or machine learning algorithms. High-power processor 608-2 can process data for radar-based applications that are in the spotlight, such as gesture recognition or face recognition (for authentication system 114), and can provide accurate and high-resolution data through angle ambiguity resolution or identification of multiple users and the characteristics of those users.

[0067] To conserve power, the power management module 620 can control whether the low-power processor 608-1 or the high-power processor 608-2 is used to process radar data. In some cases, the low-power processor 608-1 can perform part of the analysis and pass the data to the high-power processor 608-2. Exemplary data can include a clutter map, raw radar data or minimally processed radar data (e.g., in-phase and quadrature data, or range-Doppler data), or digital beamforming data. The low-power processor 608-1 can also perform some low-level analysis to determine whether there is something in the environment that the high-power processor 608-2 should analyze. Thus, power can be conserved by restricting the operation of the high-power processor 608-2 while utilizing the high-power processor 608-2 for cases where high-fidelity or accurate radar data is required by radar-based applications. Other factors that can affect the power consumption within the radar system 104 are further described with respect to FIG. 6-1.

[0068] The gesture recognition model 621 interprets gestures, such as touch-independent gestures, from the radar data acquired by the radar system 104. The gestures can be two-dimensional gestures (e.g., performed near a surface when the radar system 104 outputs microwave radiation). The gestures can be three-dimensional gestures performed in the air.

[0069] Based on the radar data, the gesture recognition model 621 identifies cues, shapes, and signs made by the user using their body, including their fingers, hands, eyes, head, mouth, etc. The gesture recognition model 621 matches the user's movements to the matching shapes, signs, and movements of predefined gestures. In response to the determination that the radar data matches a particular gesture, the gesture recognition model 621 outputs the gesture indication to other components to perform functions, such as controlling an operating system or application function, e.g., authenticating user 120.

[0070] The gesture recognition model 621 may be a machine learning model, such as a neural network, trained to identify touch-independent gestures from radar data. For example, the gesture recognition model 621 may be trained using training data that includes radar data and samples of the corresponding parts of the gestures that match the radar data. Based on the training data, the gesture recognition model 621 determines the rules to apply to samples of the radar data received by the radar system 104 such that when similar radar data is received, the corresponding parts of the gesture are identified and used to construct a gesture prediction. When executing the rules, the gesture recognition model 621 can output the indication of the recognized gesture predicted from the radar data.

[0071] In some cases, the indication of the recognized gesture may be accompanied by a confidence level or a score. The confidence level indicates the degree of confidence applied to the gesture identified from the radar data 104. The gesture recognition model 621 may adjust the confidence in the identified gesture based on the situation. For example, the gesture recognition model 621 may apply a high confidence level when a gesture is detected in an environment where the user is not moving, while applying a low confidence level when a similar gesture is detected in an environment where the user is moving. The gesture recognition model 621 may apply a low confidence level when a gesture is detected in an environment where a large object is obstructing the radar system 104, while applying a high confidence level when a similar gesture is detected in an environment where the user is looking at the UE 102. An application or other component relying on the detected gesture may discard or process the gesture depending on the confidence level or score associated with the gesture. The techniques and systems described may apply a confidence level or a score to gate the gesture, such that the gesture is discarded and not used to perform a function.

[0072] These and other capabilities and configurations, and the ways in which the entities of FIGS. 1, 2, 4, and 6 - 9 act and interact, are described in more detail below. These entities may be further divided or combined. The environment 100 of FIG. 1 and the detailed illustrations of FIGS. 2 - 9 show some of the many possible environments and devices in which the techniques described can be employed. FIGS. 6 - 9 illustrate additional details and features of the radar system 104. In FIGS. 6 - 9, the radar system 104 is described in the context of the UE 102, but as noted above, the availability of the features and advantages of the systems and techniques described is not necessarily so limited, and other embodiments involving other types of electronic devices may also be within the scope of the present teachings.

[0073] FIG. 7 shows an exemplary arrangement 700 of the receive antenna elements 702. If the antenna array 604 includes, for example, at least four receive antenna elements 702, the receive antenna elements 702 can be arranged in a rectangular arrangement 704-1 as shown in the center of FIG. 7. Alternatively, if the antenna array 604 includes at least three receive antenna elements 702, a triangular arrangement 704-2 or an L-shaped arrangement 704-3 may be used.

[0074] Due to the size or layout constraints of the UE 102, the element spacing between the receive antenna elements 702, or the quantity of the receive antenna elements 702, may not be ideal for the angles when the radar system 104 monitors. In particular, the element spacing may introduce an angle ambiguity that makes it difficult for a conventional radar to estimate the angular position of a target. Conventional radars may therefore limit the field of view (e.g., the angles to be monitored) to avoid ambiguous zones with angle ambiguities and thereby reduce false detections. For example, a conventional radar may limit the field of view to an angle of about -45 degrees to 45 degrees to avoid the angle ambiguity resulting from using a wavelength of 8 millimeters (mm) and an element spacing of 6.5 mm (e.g., the element spacing is 90% of the wavelength). Thus, a conventional radar may not be able to detect targets beyond the 45-degree limit of the field of view. In contrast, the radar system 104 includes a digital beamformer 616 and an angle estimator 618, which resolve the angle ambiguity and enable the radar system 104 to monitor angles beyond the 45-degree limit, such as angles from about -90 degrees to 90 degrees, or angles up to about -180 degrees and 180 degrees. These angular ranges can be applied over one or more directions (e.g., azimuth and / or elevation). Thus, the radar system 104 can achieve a low false alarm probability for various different antenna array designs that include element spacings smaller than, larger than, or equal to half of the center wavelength of the radar signal.

[0075] Using the antenna array 604, the radar system 104 can form beams that are directed or undirected, wide or narrow, or shaped (such as a hemisphere, cube, fan, cone, or cylinder). As an example, one or more transmit antenna elements (not shown) may have an omnidirectional radiation pattern that is not directed, or may be capable of generating a wide beam such as the wide transmit beam 706. Any of these techniques enables the radar system 104 to illuminate a large spatial volume. However, to achieve target angle accuracy and angular resolution, the receive antenna elements 702 and the digital beamformer 616 can be used to generate thousands of narrow directed beams (such as 3000 beams, 7000 beams, or 9000 beams) such as the narrow receive beam 708. Thus, the radar system 104 can efficiently monitor the external environment and accurately determine the angle of arrival of reflections within the external environment.

[0076] Returning to FIG. 6-1, the transceiver 606 includes circuitry and logic for transmitting and receiving radar signals via the antenna array 604. The components of the transceiver 606 can include amplifiers, mixers, switches, analog / digital converters, filters, etc. for conditioning the radar signals. The transceiver 606 can also include logic for performing in-phase / quadrature (I / Q) operations such as modulation or demodulation. The transceiver 606 can be configured for continuous wave radar operation or pulsed radar operation. To generate the radar signals, various modulations can be used including linear frequency modulation, triangular frequency modulation, stepped frequency modulation, or phase modulation.

[0077] The transceiver 606 can generate radar signals within a range of frequencies (e.g., a frequency spectrum), such as 1 gigahertz (GHz) to 400 GHz, 4 GHz to 100 GHz, or 57 GHz to 63 GHz. The frequency spectrum can be divided into multiple sub - spectra having the same bandwidth or different bandwidths. The bandwidth can be on the order of 500 megahertz (MHz), 1 GHz, 2 GHz, etc. As an example, different frequency sub - spectra may include frequencies of about 57 GHz to 59 GHz, 59 GHz to 61 GHz, or 61 GHz to 63 GHz. Multiple frequency sub - spectra having the same bandwidth, whether adjacent or not, may also be selected for interference. The multiple frequency sub - spectra may be transmitted simultaneously or separated in time, using a single radar signal or multiple radar signals. Adjacent frequency sub - spectra enable the radar signal to have a wider bandwidth, while non - adjacent frequency sub - spectra can further emphasize the amplitude and phase differences that allow the angle estimator 618 to resolve angle ambiguities. The attenuation attenuator 614 or the angle estimator 618 may cause the transceiver 606 to utilize one or more frequency sub - spectra to improve the performance of the radar system 104, as further described with respect to FIGS. 8 and 9. Some embodiments of the technique are particularly advantageous, for example, when the UE 102 is a handheld smartphone, the radar signal is in the band of 57 GHz to 64 GHz, the peak equivalent isotropic radiated power (EIRP) is in the range of 10 dBm to 20 dBm (10 mW to 100 mW), and the average power spectral density is about 13 dBm / MHz. This provides a "bubble" of a suitable size for radar detection (e.g., with a range of at least 1 meter and often up to 2 meters or more) in the vicinity of the smartphone and the user, while suitably addressing radiation health and co - existence issues. Within the bubble, the methods described for authentication management via the IMU and radar provide particularly excellent time - saving conveniences while conserving power.

[0078] The power management module 620 regulates power usage to balance performance and power consumption. For example, the power management module 620 communicates with the radar manager 106 to cause the radar system 104 to collect data using predefined radar power states. Each predefined radar power state may be associated with a particular framing structure, a particular transmit power level, or particular hardware (e.g., the low power processor 608-1 or the high power processor 608-2 of FIG. 6-2). Adjusting one or more of these affects the power consumption of the radar system 104. However, reducing the power consumption affects performance such as gesture frame update rate and response delay, which are described below.

[0079] FIG. 6-3 illustrates an example relationship between power consumption, gesture frame update rate 634, and response delay. In graph 636, radar power states 638-1, 638-2, and 638-3 are associated with different levels of power consumption and different gesture frame update rates 634. The gesture frame update rate 634 represents how frequently the radar system 104 actively monitors the external environment by transmitting and receiving one or more radar signals. Generally speaking, power consumption is proportional to the gesture frame update rate 634. Thus, a higher gesture frame update rate 634 results in a greater amount of power being consumed by the radar system 104.

[0080] In graph 636, radar power state 638-1 utilizes the least amount of power, while radar power state 638-3 consumes the maximum amount of power. As an example, radar power state 638-1 consumes power on the order of a few milliwatts (mW) (e.g., about 2 mW to 4 mW), while radar power state 638-3 consumes power on the order of a dozen milliwatts (e.g., about 6 mW to 20 mW). Regarding the gesture frame update rate 634, radar power state 638-1 uses an update rate on the order of a few Hertz (e.g., about 1 Hz, or less than 5 Hz), while radar power state 638-3 uses a gesture frame update rate 634 on the order of dozens of Hertz (e.g., about 20 Hz, or greater than 10 Hz).

[0081] Graph 640 shows the relationship between the response delay and the gesture frame update rate 634 for different radar power states 638-1 to 638-3. Generally speaking, the response delay is inversely proportional to both the gesture frame update rate 634 and the power consumption. In particular, the response delay decreases exponentially while the gesture frame update rate 634 is increasing. The response delay associated with radar power state 638-1 may be on the order of hundreds of milliseconds (ms) (e.g., 1000 ms, or greater than 200 ms), while the response delay associated with radar power state 638-3 may be on the order of a few milliseconds (e.g., 50 ms, or less than 100 ms). For radar power state 638-2, the power consumption, the gesture frame update rate 634, and the response delay are between the values of radar power state 638-1 and radar power state 638-3. For example, the power consumption of radar power state 638-2 is about 5 mW, the gesture frame update rate is about 8 Hz, and the response delay is about 100 ms to 200 ms.

[0082] Instead of operating in either radar power state 638-1 or radar power state 638-3, the power management module 620 dynamically switches between radar power states 638-1, 638-2, and 638-3 (and sub-states between each of these radar power states 638) such that response latency and power consumption are both managed based on activity in the environment. As an example, the power management module 620 activates radar power state 638-1 to monitor the external environment or detect an approaching user. Thereafter, if the radar system 104 determines that the user may be showing an intention to engage, or may have started to engage, or may have started to make gestures, the power management module 620 activates radar power state 638-3. Different triggers may cause the power management module 620 to switch between different radar power states 638-1 to 638-3. Exemplary triggers include movement or lack of movement, the appearance or disappearance of the user, the user entering or exiting a specified area (e.g., an area defined by range, azimuth, or elevation), a change in the speed of movement associated with the user, an intention to engage (e.g., “reach”) determined by the radar manager 106 (although some intentions to engage require additional power such as face feature tracking), or a change in the reflected signal strength (e.g., due to a change in the radar cross-section). In general, triggers indicating that the user is less likely to interact with the UE 102 or preferring to collect data using a longer response latency may activate radar power state 638-1 to conserve power.

[0083] Generally, the power management module 620 determines when and how power can be saved and gradually adjusts the power consumption to enable the radar system 104 to operate within the power limit of the UE 102. In some cases, the power management module 620 may monitor the amount of remaining available power and adjust the operation of the radar system 104 accordingly (e.g., due to a low battery). For example, when the amount of remaining power is low, the power management module 620 may continue to operate in the radar power state 638-1 instead of switching to either the radar power state 638-2 or 638-3.

[0084] Each of the power states 638-1 to 638-3 may be associated with a specific framing structure. The framing structure specifies the configuration, scheduling, and signal characteristics associated with the transmission and reception of radar signals. Generally, the framing structure is set up so that appropriate radar data can be collected based on the external environment. The framing structure can be customized to facilitate the collection of different types of radar data for different applications (e.g., proximity detection, feature recognition, or gesture recognition). During inactive times throughout each level of the framing structure, the power management module 620 can turn off components within the transceiver 606 of FIG. 6-2 to save power. An exemplary framing structure is further described with respect to FIG. 6-4.

[0085] FIG. 6-4 shows an exemplary framing structure 642. In the illustrated configuration, the framing structure 642 includes three different types of frames. At the upper level, the framing structure 642 includes a sequence of gesture frames 644 that can be in an active state or an inactive state. Generally speaking, the active state consumes a greater amount of power compared to the inactive state. At the intermediate level, the framing structure 642 similarly includes feature frames (F F) It includes the sequence of 646. Different types of feature frames include the pulse mode feature frame 648 (shown at the lower left of FIG. 6-4) and the burst mode feature frame 650 (shown at the lower right of FIG. 6-4). At a lower level, the framing structure 642 can similarly be in an active state or an inactive state, a radar frame (radar frame: R F) It includes the sequence of 652.

[0086] The radar system 104 transmits and receives radar signals in the active radar frame (RF) 652. In some cases, the radar frame 652 is analyzed individually for basic radar operations such as search and tracking, clutter map generation, determination of the user's location, etc. The radar data collected in each active radar frame 652 can be stored in a buffer after the completion of the radar frame 652 or provided directly to the processor 608 in FIG. 6-1.

[0087] The radar system 104 analyzes radar data across multiple radar frames 652 (e.g., across a group of radar frames 652 associated with the active feature frame 646) to identify specific features associated with one or more gestures. Exemplary types of features include specific types of movements, movements associated with specific appendages (e.g., hands or each finger), and features associated with different parts of a gesture. To recognize a gesture made by the user 120 in the active gesture frame 644, the radar system 104 analyzes the radar data associated with one or more active feature frames 646.

[0088] Depending on the type of gesture, the duration of the gesture frame 644 may be on the order of milliseconds or seconds (e.g., about 10 ms to 10 s). After the active gesture frame 644 occurs, the radar system 104 becomes inactive as indicated by the inactive gesture frames 644-3 and 644-4. The duration of the inactive gesture frame 644 is characterized by the deep sleep time 654, which may be on the order of tens of milliseconds or more (e.g., greater than 50 ms). In an exemplary implementation, the radar system 104 can turn off all components within the transceiver 606 to conserve power during the deep sleep time 654.

[0089] In the illustrated framing structure 642, each gesture frame 644 includes K feature frames 646, where K is a positive integer. When the gesture frame 644 is in an inactive state, all of the feature frames 646 associated with that gesture frame 644 are also in an inactive state. In contrast, an active gesture frame 644 includes J active feature frames 646 and K-J inactive feature frames 646, where J is a positive integer that is less than or equal to K. The number of feature frames 646 may be based on the complexity of the gesture and may include several to 100 feature frames 646 (e.g., K may be equal to 2, 10, 30, 60, or 100). The duration of each feature frame 646 may be on the order of milliseconds (e.g., about 1 ms to 50 ms).

[0090] To conserve power, active feature frames 646-1 to 646-J occur prior to non-active feature frames 646-(J + 1) to 646-K. The duration of non-active feature frames 646-(J + 1) to 646-K is characterized by the sleep time 656. Thus, non-active feature frames 646-(J + 1) to 646-K are continuously executed so that the radar system 104 can be in a powered-down state for a longer duration compared to other techniques that alternate non-active feature frames 646-(J + 1) to 646-K with active feature frames 646-1 to 646-J. Generally speaking, increasing the duration of the sleep time 656 enables the radar system 104 to turn off components within the transceiver 606 that require a longer startup time.

[0091] Each feature frame 646 includes L radar frames 652, where L is a positive integer that may or may not be equal to J or K. In some implementations, the number of radar frames 652 may vary for different feature frames 646 and may include several frames or hundreds of frames (e.g., L may be equal to 5, 15, 30, 100, or 500). The duration of the radar frame 652 may be on the order of tens or thousands of microseconds (e.g., about 30 μs to 5 ms). The radar frames 652 within a particular feature frame 646 can be customized for a predetermined detection range, range resolution, or Doppler sensitivity, which facilitates the detection of specific features and gestures. For example, the radar frame 652 may utilize a particular type of modulation, bandwidth, frequency, transmit power, or timing. When a feature frame 646 is in a non-active state, all of the radar frames 652 associated with that feature frame 646 are also in a non-active state.

[0092] The pulse mode feature frame 648 and the burst mode feature frame 650 include different sequences of radar frames 652. Generally speaking, the radar frames 652 within the active pulse mode feature frame 648 transmit pulses that are temporally separated by a predetermined amount. In contrast, the radar frames 652 within the active burst mode feature frame 650 transmit pulses continuously over a portion of the burst mode feature frame 650 (e.g., the pulses are not separated by a predetermined amount of time).

[0093] Within each active pulse mode feature frame 648, the sequence of radar frames 652 alternates between an active state and a non - active state. Each active radar frame 652 transmits a radar signal (e.g., a chirp) illustrated by a triangle. The duration of the radar signal is characterized by an active time 658. During the active time 658, components within the transceiver 606 are powered on. During a short idle time 660 that includes the remaining time within the active radar frame 652 and the duration of the next non - active radar frame 652, the radar system 104 conserves power by turning off components within the transceiver 606 that have startup times within the duration of the short idle time 660.

[0094] The active burst mode characteristic frame 650 includes M active radar frames 652 and L - M non - active radar frames 652, where M is a positive integer less than or equal to L. To conserve power, the active radar frames 652 - 1 to 652 - M occur prior to the non - active radar frames 652-(M + 1) to 652 - L. The duration of the non - active radar frames 652-(M + 1) to 652 - L is characterized by a long idle time 662. By grouping together the non - active radar frames 652-(M + 1) to 652 - L, the radar system 104 can be in a power - down state for a longer duration compared to the short idle time 660 that occurs during the pulse mode characteristic frame 648. Additionally, the power management module 620 can turn off additional components within the transceiver 606 that have an activation time longer than the short idle time 660 and shorter than the long idle time 662.

[0095] Each active radar frame 652 within the active burst mode characteristic frame 650 transmits a portion of the radar signal. In this example, the active radar frames 652 - 1 to 652 - M alternately repeat transmitting a portion of the radar signal with increasing frequency and transmitting a portion of the radar signal with decreasing frequency.

[0096] The framing structure 642 enables power to be conserved through adjustable duty cycles within each frame type. The first duty cycle 664 is based on the number (J) of active characteristic frames 646 relative to the total number (K) of characteristic frames 646. The second duty cycle 665 is based on the number (e.g., L / 2 or M) of active radar frames 652 relative to the total number (L) of radar frames 652. The third duty cycle 668 is based on the duration of the radar signal relative to the duration of the radar frame 652.

[0097] Consider an exemplary framing structure 642 for a power state 638-1 that consumes about 2 mW of power and has a gesture frame update rate 634 of about 1 Hz to 4 Hz. In this example, the framing structure 642 includes a gesture frame 644 having a duration of about 250 ms to 1 second. The gesture frame 644 includes 31 pulse mode feature frames 648 (e.g., L is equal to 31). One of the 31 pulse mode feature frames 648 is in an active state. This results in a duty cycle 664 that is equal to about 3.2%. The duration of each pulse mode feature frame 648 is about 8 ms to 32 ms. Each pulse mode feature frame 648 is composed of 8 radar frames 652. Within the active pulse mode feature frame 648, all 8 radar frames 652 are in an active state. This results in a duty cycle 665 that is equal to 100%. The duration of each radar frame 652 is about 1 ms to 4 ms. The active time 658 within each active radar frame 652 is about 32 μs to 128 μs. As a result, the resulting duty cycle 668 is about 3.2%. This exemplary framing structure 642 has been found to produce good performance results. These good performance results relate to good gesture recognition and presence detection and also produce good power efficiency results in the application scenario of a handheld smartphone in a low power state (e.g., low power state 504-3).

[0098] Based on the framing structure 642, the power management module 620 can determine the time when the radar system 104 is not actively collecting radar data. Based on this inactive period, the power management module 620 can adjust the operating state of the radar system 104 and turn off one or more components of the transceiver 606 to save power, as further described below.

[0099] As described, power management module 620 can conserve power by turning off one or more components within transceiver 606 (e.g., voltage controlled oscillator, multiplexer, analog / digital converter, phase locked loop, or crystal oscillator) during inactive periods. These inactive periods occur when radar system 104 is not actively transmitting or receiving radar signals that can be on the order of microseconds (μs), milliseconds (ms), or seconds (s). Also, power management module 620 can modify the transmit power of the radar signal by adjusting the amount of amplification provided by the signal amplifier. Additionally, power management module 620 can control the use of different hardware components within radar system 104 to conserve power. If processor 608 includes, for example, a lower power processor and a higher power processor (e.g., processors having different amounts of memory and computational capabilities), power management module 620 can utilize the lower power processor for low level analysis (e.g., detecting movement, determining the location of a user, or monitoring the environment) and switch to utilizing the higher power processor for when high fidelity or accurate radar data is required by radar manager 106 (e.g., to implement high power state 504-1 of authentication system 114 for authenticating a user using radar data).

[0100] In addition to the internal power saving techniques described above, the power management module 620 can also save power within the UE 102 by activating or deactivating other external components or sensors within the UE 102, either alone or at the command of the authentication system 114. These external components may include speakers, camera sensors, global positioning systems, wireless communication transceivers, displays, gyroscopes, or accelerometers. Since the radar system 104 can monitor the environment using a small amount of power, the power management module 620 can appropriately turn these external components on or off based on where the user is located or what the user is doing. In this way, the UE 102 can use an auto-shutdown timer or save power seamlessly in response to the user without the user physically touching or verbally controlling the UE 102.

[0101] FIG. 8 shows additional details of an exemplary implementation 800 of the radar system 104 within the UE 102. In example 800, the antenna array 604 is positioned under the external housing of the UE 102, such as a glass cover or an external case. Depending on its material properties, the external housing may act as an attenuator 802, which attenuates or distorts the radar signals transmitted and received by the radar system 104. The attenuator 802 may include different types of glass or plastic, some of which may be found within the display screen, external housing, or other components of the UE 102 and may have a dielectric constant (e.g., relative dielectric constant) of about 4 to 10. Thus, the attenuator 802 is impermeable or semi-permeable to the radar signal 806, which can result in a portion of the transmitted or received radar signal 806 being reflected (as shown by the reflected portion 804). For conventional radars, the attenuator 802 can reduce the monitorable effective range, prevent small targets from being detected, or reduce the overall accuracy.

[0102] Assuming that the transmit power of radar system 104 is limited and that redesign of the external housing is not desirable, one or more attenuation-dependent characteristics of radar signal 806 (e.g., frequency sub-spectrum 808 or steering angle 810), or attenuation-dependent characteristics of attenuator 802 (e.g., distance 812 between attenuator 802 and radar system 104, or thickness 814 of attenuator 802), are adjusted to mitigate the effects of attenuator 802. Some of these characteristics can be set during manufacturing or adjusted by attenuation mitigator 614 during operation of radar system 104. Attenuation mitigator 614 can cause, for example, transceiver 606 to transmit radar signal 806 using a selected frequency sub-spectrum 808 or steering angle 810, cause the platform to move the radar system 104 closer to or farther from attenuator 802 to change distance 812, or prompt the user to apply another attenuator to increase thickness 814 of attenuator 802.

[0103] Based on predetermined characteristics of attenuator 802 (e.g., characteristics stored in computer-readable medium 404 of UE 102 or within system medium 610), or by processing the return of radar signal 806 to measure one or more characteristics of attenuator 802, appropriate adjustments can be made by attenuation mitigator 614. Even if some of the attenuation-dependent characteristics are fixed or constrained, attenuation mitigator 614 can take into account these limitations, balance each parameter, and achieve the target radar performance. As a result, attenuation mitigator 614 enables radar system 104 to achieve increased accuracy and a greater effective range for detecting and tracking a user located on the opposite side of attenuator 802. These techniques provide an alternative to increasing transmit power, which increases the power consumption of radar system 104, or to changing the material characteristics of attenuator 802, which can be difficult and expensive once the device has started production.

[0104] FIG. 9 shows an exemplary scheme 900 implemented by the radar system 104. The portions of the scheme 900 may be performed by the processor 608, the computer processor 402, or other hardware circuits. The scheme 900 is customizable to support different types of electronic devices and radar-based applications (e.g., the radar manager 106), and also enables the radar system 104 to achieve the target angle accuracy despite design constraints.

[0105] The transceiver 606 generates the raw data 902 based on the individual responses of the receiving antenna elements 702 to the received radar signals. The received radar signals may be associated with one or more frequency sub-spectra 904 selected by the angle estimator 618 to facilitate the resolution of angle ambiguities. The frequency sub-spectra 904 may be selected, for example , to reduce the amount of sidelobes, or to reduce the amplitude of the sidelobes (e.g., reduce the amplitude by 0.5 dB, 1 dB, or more). The amount of frequency sub-spectra may be determined based on the target angle accuracy or computational limitations of the radar system 104.

[0106] The raw data 902 includes digital information (e.g., in-phase and quadrature data) for a period, different frequencies, and a plurality of channels respectively associated with the receiving antenna elements 702. A Fast-Fourier Transform (FFT) 906 is performed on the raw data 902 to generate the preprocessed data 908. The preprocessed data 908 includes digital information for different ranges (e.g., range bins) and for the plurality of channels over the period. A Doppler filtering process 910 is performed on the preprocessed data 908 to generate the range-Doppler data 912. The Doppler filtering process 910 may include another FFT that generates amplitude and phase information for a plurality of range bins, a plurality of Doppler frequencies, and for the plurality of channels. Based on the range-Doppler data 912, the digital beamformer 616 generates the beamforming data 914. The beamforming data 914 includes digital information for a set of azimuth angles and / or elevation angles that represent the field of view in which different steering angles or beams are formed by the digital beamformer 616. Although not shown, alternatively, the digital beamformer 616 may generate the beamforming data 914 based on the preprocessed data 908, and the Doppler filtering process 910 may generate the range-Doppler data 912 based on the beamforming data 914. To reduce the computational load, the digital beamformer 616 may process a part of the range-Doppler data 912 or the preprocessed data 908 based on the range, time, or Doppler frequency interval of interest.

[0107] The digital beamformer 616 can be implemented using a single-look beamformer 916, a multi-look interferometer 918, or a multi-look beamformer 920. Generally, the single-look beamformer 916 can be used for deterministic objects (e.g., a point-source target having a single phase center). For non-deterministic targets (e.g., a target having multiple phase centers), the multi-look interferometer 918 or the multi-look beamformer 920 is used to improve the accuracy compared to the single-look beamformer 916. A human is an example of a non-deterministic target and has multiple phase centers 922 that can vary based on different aspect angles as shown at 924-1 and 924-2. Variations in the constructive or destructive interference generated by the multiple phase centers 922 can make it difficult for conventional radar systems to accurately determine the angular position. However, the multi-look interferometer 918 or the multi-look beamformer 920 performs interference averaging to enhance the accuracy of the beamforming data 914. The multi-look interferometer 918 performs two-channel interference averaging and generates phase information that can be used to accurately determine the angular information. On the other hand, the multi-look beamformer 920 uses linear or non-linear beamformers such as Fourier, Capon, multiple signal classification (MUSIC), or is the minimum variance distortion less response (MVDR) and performs interference averaging of two or more channels. The accuracy improvement provided through the multi-look beamformer 920 or the multi-look interferometer 918 enables the radar system 104 to recognize small gestures or distinguish multiple parts of a user (e.g., facial features).

[0108] The angle estimator 618 analyzes the beamforming data 914 to estimate one or more angular positions. The angle estimator 618 may utilize signal processing techniques, pattern matching techniques, or machine learning. The angle estimator 618 also resolves angle ambiguities that may arise from the design of the radar system 104 or the field of view monitored by the radar system 104. Exemplary angle ambiguities are shown within the amplitude plot 926 (e.g., amplitude response).

[0109] The amplitude plot 926 shows the amplitude differences that can occur for different angular positions of the target and for different steering angles 810. A first amplitude response 928-1 (illustrated by a solid line) is shown for a target positioned at a first angular position 930-1. Similarly, a second amplitude response 928-2 (illustrated by a dotted line) is shown for a target positioned at a second angular position 930-2. In this example, the difference is considered to span an angle from -180 degrees to 180 degrees.

[0110] As shown in the amplitude plot 926, there are ambiguous zones for two angular positions 930-1 and 930-2. The first amplitude response 928-1 has the highest peak at the first angular position 930-1 and a lower peak at the second angular position 930-2. The highest peak corresponds to the actual position of the target, while the lower peak ambiguates the first angular position 930-1. This is because it may be within some threshold that a conventional radar cannot confidently determine whether the target is at the first angular position 930-1 or the second angular position 930-2. In contrast, the second amplitude response 928-2 has a lower peak at the second angular position 930-2 and a higher peak at the first angular position 930-1. In this case, the lower peak corresponds to the location of the target.

[0111] Conventional radars may be limited to using the highest peak amplitude to determine the angular position. Instead, the angle estimator 618 analyzes the subtle differences in the shapes of the amplitude responses 928-1 and 928-2. The characteristics of the shape can include, for example, roll-off, the width of the peak or null, the angular position of the peak or null, the height or depth of the peak and null, the shape of the sidelobe, the symmetry within the amplitude response 928-1 or 928-2, or the lack of symmetry within the amplitude response 928-1 or 928-2. Similar shape characteristics may be analyzed in the phase response, which can provide additional information for resolving angular ambiguities. The angle estimator 618 thus maps a unique angular signature or pattern to the angular position.

[0112] The angle estimator 618 may include an algorithm or a set of tools that can be selected according to the type of the UE 102 (e.g., computing power or power constraint) or the target angle resolution for the radar manager 106. In some implementations, the angle estimator 618 may include a neural network 932, a convolutional neural network (CNN) 934, or a long short-term memory (LSTM) network 936. The neural network 932 may have different depths or numbers of hidden layers (e.g., 3 hidden layers, 5 hidden layers, or 10 hidden layers), and may include different numbers of connections (e.g., the neural network 932 may include a fully connected neural network or a partially connected neural network). In some cases, the CNN 934 may be used to increase the calculation speed of the angle estimator 618. The LSTM network 936 may be used to enable the angle estimator 618 to track the target. Using a machine learning approach, the angle estimator 618 employs a non-linear function and analyzes the shape of the amplitude response 928-1 or 928-2 to generate angle probability data 938, which indicates the likelihood that a user or a part of the user is within an angle bin. The angle estimator 618 may provide angle probability data 938 for a small number of angle bins, such as two angle bins, to provide the probability that the target is on the left or right side of the UE 102, or may provide angle probability data 938 for thousands of angle bins (e.g., to provide angle probability data 938 for continuous angle measurements).

[0113] Based on the angle probability data 938, the tracker module 940 generates angle position data 942, which identifies the angular position of the target. The tracker module 940 may determine the angular position of the target based on the angular bin having the highest probability in the angle probability data 938 or based on prediction information (e.g., previously measured angular position information). The tracker module 940 may also track one or more moving targets to enable the radar system 104 to distinguish or identify the target with confidence. Other data, including range, Doppler, velocity, or acceleration, may also be used to determine the angular position. In some cases, the tracker module 940 may include an alpha-beta tracker, a Kalman filter, a multiple hypothesis tracker (MHT), etc.

[0114] The quantizer module 944 acquires the angle position data 942 and quantizes the data to generate quantized angle position data 946. The quantization may be performed based on the target angular resolution for the radar manager 106. In some cases, fewer quantization levels may be used such that the quantized angle position data 946 indicates whether the target is on the right or left side of the UE 102 or identifies the 90-degree quadrant in which the target is located. This may be sufficient for some radar-based applications such as user proximity detection. In other cases, more quantization levels may be used such that the quantized angle position data 946 indicates the angular position of the target within an accuracy of, for example, fractions of a degree, one degree, five degrees, etc. This resolution may be used for higher-resolution radar-based applications such as gesture recognition or in the realization examples of the attentional or interaction states as described herein. In some realization examples, the digital beamformer 616, the angle estimator 618, the tracker module 940, and the quantizer module 944 are all realized in a single machine learning module.

[0115] Embodiments are included where radar is used to determine the intention of a user to attempt to engage, disengage, or maintain engagement, and further include embodiments where radar is used to detect user actions classified as indicators of a user's intention to attempt to engage with or interact with an electronic device (any of these embodiments may alternatively be achieved using a device-mounted camera provided in most modern smartphones). Among the advantages of the described embodiments, the advantage is that although the power consumption of the radar system is substantially less than that of the camera system, the resulting justification can often be better with the radar system than with the camera system. For example, using the radar system described above, an average power in the range of a few milliwatts to tens of milliwatts (e.g., 10 mW, 20 mW, 30 mW, or 40 mW), including the processing power for processing radar vector data to make a determination, can achieve the desired user intention detection. At these low power levels, it would be readily acceptable to always enable the radar system 104. Thus, for example, if a smartphone radar system is always enabled, the desired enjoyable and seamless experience described herein can still be provided to a user sitting in a room away from their smartphone for an extended period of time.

[0116] In contrast, the optical cameras provided in most current smartphones typically operate at a power of several hundred milliwatts (e.g., a power one order of magnitude higher than 40 mW, i.e., 400 mW). At such power rates, the optical cameras would be disadvantageous. This is because they significantly reduce the battery life of most current smartphones to such an extent that, if not prohibitively, it is highly unrealistic to keep the optical cameras always on. An additional advantage of the radar system is that (for many typical embodiments where the radar chip generally faces outward in the same direction as the selfie camera) with the screen facing up and lying flat on the table Even when present, the field of view can be made quite large so as to easily detect a user walking from any direction, and furthermore, due to its Doppler processing ability, it can be very effective in detecting even relatively subtle movements of a moving body from various directions (especially at an operating frequency near 60 GHz).

[0117] In addition, the radar system can operate in an environment where the performance of the camera system is degraded or limited. For example, in an environment with lower light levels, the camera system may have a reduced ability to detect shape or movement. In contrast, the radar system functions well in low light levels, just as it does in full light. The radar system can also detect the presence and gestures through some obstacles. For example, when a smartphone is inside a jacket or a pocket of trousers, the camera system may not be able to detect the user or the gesture. However, the radar system can still detect an object in the radar field even through a fabric that blocks the camera system. A further advantage of using a radar system over the built-in video camera system of a smartphone is privacy. Because the user can have the advantage of the enjoyable and seamless experience described here and at the same time does not have to worry about having a video camera that is taking pictures of themselves for such purposes.

[0118] The entities in FIGS. 1, 2, 4, and 6 - 9 may be further divided, combined, or used with other sensors or components. Thus, different realizations of the UE 102 having different configurations of the radar system 104 and the sensor 108 can be used to implement radar-based gesture recognition using situation-aware gating and other situation-aware controls. The exemplary operating environment 100 of FIG. 1 and the detailed illustrations of FIGS. 2 - 9 merely show some of the many possible environments and devices that can employ the described techniques.

[0119] Exemplary Method This section shows exemplary methods that may operate separately, or in whole or in part together. Various exemplary methods are described, each of which is presented in subsections for readability, but the titles of these subsections are not intended to limit the interoperability of each of these methods with others.

[0120] Authentication Management FIG. 10 shows an exemplary method 1000 for managing authentication via an IMU and a radar, and is an example of managing the power state for a user device. Method 1000 is shown as a set of blocks that identify the operations to be performed, but is not necessarily limited to the order or combination shown for performing the operations by each block. Also, in order to provide a wide range of additional and / or alternative methods, any one or more of the operations may be repeated, combined, rearranged, or linked. In the following portions of the description, reference may be made to the exemplary operating environment 100 of FIG. 1, or to entities or processes as detailed in other drawings, and such references are made for purposes of illustration only. The techniques are not limited to execution by one entity or multiple entities operating on one device.

[0121] At 1002, based on radar data and by the user device, an intention of the user to engage is determined. The intention to engage indicates that the user intends to engage with the user device. As described above, the intention to engage may be indicated, by way of example only, by determining that the user 120 is reaching towards the UE 102, looking at the UE 102, or leaning or orienting their body towards the UE 102. towards it.

[0122] At 1004, instead of or in addition to determining the intention to engage through radar data, the movement of the user equipment is determined based on inertial data. This movement may indicate that user 120 picks up UE102, touches UE102, and other movements as described above.

[0123] At 1006, in response to the determination of the intention to engage and, optionally, the determination of the movement of the user equipment, the power state of the power consumption component of the authentication system is changed. The power state of the power consumption component is changed from the first power state to the second power state, and the second power state consumes more power than the first power state. This change may be based only on the intention to engage determined using radar data, or may also be through the movement determined through inertial data. Further, the power state of the power consumption component or other power supply components may be further increased based on the determination of the movement. As described above, this determination of the movement may confirm the intention of user 120 to engage, may also provide the intention to engage, or may otherwise add speed and / or robustness to the determination to add power, resources, etc. to the authentication system. Note that, in some cases, even if it is determined that the user does not intend to engage, the components of the authentication system remain powered. In such cases, the technique acts to perform the authentication process in response to the determination that the intention to engage has been made. In such cases, even if power is not conserved for that process, the waiting time is reduced. However, the technique can refrain from using resources not associated with the authentication system, thereby conserving power in other ways.

[0124] The power state in which the power consumption components of the authentication system are changed may or may not be sufficient for the authentication system to perform an authentication process on the user. In some cases, the second power state of the power consumption component is not the high power state 504-1. In such a case, the second power state is the intermediate power state 504-2 as described above. This intermediate power state 504-2 is, in some cases, sufficient for the performance of power consumption components such as a camera that can still provide sensor data for authentication without fully powering up (for example, capturing an image of the user where there is sufficient light rather than in the dark). Another example is a display 116 that can be powered to accept touch input for a password without powering the display brightness to full power. In another case, it includes a radar system 104 where full power is not required to provide sufficiently accurate face features to the authentication system 114 when the user's face is in a fairly close range from the radar system 104.

[0125] In some cases, powering up a component can be an intermediate step such as a warm-up sequence that can prepare the component by giving it additional time or simply reduce the waiting time. In such a case, for example, if it is determined that there is an intention to cancel participation by moving the UE 102 (e.g., into a pocket) by the user 120 before the component is ready to authenticate, thereby preventing authentication, the state manager 112 may determine not to proceed to high power. In some cases, powering is an intermediate step where, in response to the determination that the user 120 has moved the UE 102 as indicated at 1004, the component is then sufficiently powered up to a power level sufficient to perform the authentication process. This warm-up sequence powers the component to an intermediate power state 504-2, and then, after some short period, the component is sufficiently powered up (e.g., to a high power state 504-1) for use in the authentication process. In such a case, the component is at high power (or nearly high power) during the post-warm-up sequence that follows the warm-up sequence. For components that consume a significant amount of power and are left on when not needed, such as some infrared (IR) or near-infrared (NIR) sensors, the intermediate power state during which the warm-up sequence is performed can save a significant amount of power or reduce the significant waiting time that may degrade the user experience. During the post-warm-up sequence following the warm-up sequence, the component is at high power (or nearly high power). For components that consume a significant amount of power and are left on when not needed, such as some infrared (IR) or near-infrared (NIR) sensors, the intermediate power state during which the warm-up sequence is performed can save a significant amount of power or reduce the significant waiting time that may degrade the user experience.

[0126] Exemplary power-consuming components of the authentication system have been described above and include, for example, the face authentication unlock sensor 212 of the authentication system 114 in FIG. 1, the touch screen of the display 116, the radar system 104, and the processor 608 (e.g., the high-power processor 608-2). For specific details of many potential power-consuming components of a face recognition system for authentication, refer to FIG. 2 and its description.

[0127] At 1008, an authentication process is performed by an authentication system. In doing so, the authentication system 114 uses a power consumption component in an altered power state, such as a second power state or a higher third power state. The authentication process is effective to authenticate the user or to indicate that the user should not be authenticated and access to the UE 102 should not be permitted. As described, the authentication process can be via face recognition, fingerprint reading, password or other authentication information input through a touch or voice interface (e.g., the touch screen data input component of the display 112). The authentication process compares the identifying features of the user or authentication information to some secure storage of equivalent features or authentication information to determine that the user's identity is genuine and thus access to the UE 102 should be permitted. This may be as simple as comparing a six-digit password entered through the touch screen of the display, or may require greater computational and system complexity, such as determining face features based on sensor data received from the power consumption component and comparing the determined face features to a face feature library. Although not essential, this face feature library can be stored locally from the UE 102 and can be created during face feature initialization by the UE 102 using the authentication system 114. Further, this library can be securely stored in the UE 102, such as in the form of being embedded on a secure chip integrated with the UE 102. This is one way to maintain the privacy of the user 120.

[0128] Throughout this disclosure, examples are described where a computing system (e.g., UE102, a client device, a server device, a computer, or other types of computing systems) analyzes information associated with a user, such as face features in operation 1008 just described, including radar data, inertial data, and face recognition sensor data. However, the computing system may be configured to use such information only after receiving explicit permission from the user of the computing system for the computing system to use the data. For example, when UE102 analyzes sensor data about face features to authenticate user 102, an individual user may be given the opportunity to provide input to control whether the program or feature of UE102 can collect and use the data. An individual user may have some control over which programs can or cannot be performed using the sensor data. Additionally, the collected information may be pre-processed in one or more ways such that personally identifiable information is removed before it is transferred, stored, or used in other ways. For example, before UE102 shares sensor data with another device (e.g., to train a model run on another device), UE102 may pre-process the sensor data to ensure that any user identification information or device identification information embedded in the data is removed. Thus, the user has control over whether information is collected about the user and the user's device, and how such information can be used by the computing device and / or remote computing system if collected. and the user's device, and how such information can be used by the computing device and / or remote computing system if collected.

[0129] Returning to method 1000, at 1010, instead of or in addition to, the power state of the display is changed in response to a determination that the user equipment has moved or is moving. This change can be to power up the touch input receiving ability of the display or simply to change the visual representation of the display. An example includes adding luminosity to display 116 so that when the user touches UE102, UE102 recognizes the user's intention and thus the user can see that it is probably preparing to engage with user 120. Similarly, UE102 may do so in response to the intention to engage as determined at 1002.

[0130] In some cases, the authentication process is executed without success over some period or repetition (e.g., some preset number of times or period). In such cases, method 1000 can continue by re-executing the authentication process in response to the determination of movement at 1004 as shown at 1012, or the process can continue. This alternative is shown using some of the dashed arrows in FIG. 10.

[0131] At 1014, in response to the successful authentication process of the user at 1008 (or re - execution at 1012), the user is authenticated and the access state of UE102 is changed. This change can raise the access of UE102 from a non, low, or intermediate access state to a high access state, and in such a case, UE102 is "unlocked". However, this high access state (e.g., high access state 502 - 1 in FIG. 5) is not essential. Multiple levels of authentication can secure access, power, or information for the next authentication. Examples include authenticating the user for only some, not all, uses of the applications and / or accounts of UE102 (such as an account for purchasing music, a bank account, etc.), and requiring additional authentication for those secured access accounts and applications. For example, in addition to the high access state 502 - 1, the state manager 112 can place UE102 in a high information state 506 - 1. Examples of this change to the information state include presenting the last - engaged application or web page, which is included in the last - engaged part, such as playing the location where user 120 last engaged with or was authenticated on UE102, the fourth page of a 10 - page article on a web page, or in the middle of a song or video. The state manager 112 may change these states quickly and seamlessly in response to the authentication of user 120.

[0132] As an example, consider an embodiment of the application of method 1000 to scenario 1100 shown in FIG. 11. Scenario 1100 includes five parts, and each part follows the previous part in the order of the passage of time. In the first part of scenario 1100, shown as scenario part 1100-1, user 1102 has not looked at or touched smartphone 1104, or otherwise engaged with smartphone 1104. Here, smartphone 1104 is assumed to be in a low access state 501-3, a low power state 504-3, and a low information state 506-3 (for example, smartphone 1104 appears to be switched off, but has enough power to determine the intention to engage). This scenario part 1100-1 is assumed to be the state prior to the operation of the method at 1002 in FIG. 10. The second part is shown at 1100-2, during which user 1102 is looking at smartphone 1104 while facing it, but has not touched smartphone 1104. At this point, the method, at operation 1002, based on the radar data, determines that user 1102 intends to engage with smartphone 1104. This intention to engage is determined without using the movement of reaching out a hand, but rather is based on user 1102 looking at smartphone 1104 and orienting their body towards smartphone 1104. The method makes this determination at operation 1002 through radar manager 106, which passes the determination to state manager 112. Subsequently, state manager 112, at operation 1006, changes the power state of the power consumption component (face recognition unlock sensor 212) of authentication system 114. Note that this is done well before the user reaches out a hand to or picks up smartphone 1104, reducing the waiting time and getting the authentication system 114 ready to authenticate the user.

[0133] Also, assume that over the next 0.5 seconds, while the power consumption component is powering up, user 1102 moves closer to smartphone 1104 and extends a hand (the extended hand is indicated by hand 1106) towards smartphone 1104. This is shown in the third portion 1100-3. At this point, authentication system 114 performs the authentication process (operation 1008), but assume that the authentication process does not succeed after several repetitions and / or over a certain period. The technique may abort the attempt to authenticate user 1102, thereby saving power. However, here, as shown in portion 1100-4, user 1102 touches smartphone 1104. This is operation 1004 and is determined to be the movement of smartphone 1104 through the inertial data sensed by sensor 108 in FIG. 1. This determination of the movement is passed to state manager 112. Based on this movement, state manager 112 causes authentication system 114 to continue attempting to authenticate user 1102, as shown by operation 1012 of method 1000. Further, at operation 1010, similarly based on the movement, state manager 112 illuminates display 1108 of smartphone 1104. This illumination or power-up of display 1108 can occur in scenario portions 1100-2, 1100-3, or 1100-4, but here it is shown in response to determining that user 1102 has touched smartphone 1104 (shown at 1110, along with time and notification information). By doing so, user 1102 is given feedback that smartphone 1104 recognizes that user 1102 intends to be involved.

[0134] As described, the state manager 112 causes the authentication system 114 to continue the authentication process and authenticates the user 1102 through these continued attempts. This is shown in portion 1100-5, resulting in the smartphone 1104 being in different states such as a high access state 502-1, a high power state 504-1, and a high information state 506-1, where the high access state 502-1 is indicated by the display 1108 presenting an unlock icon 1112. These state levels can be automatically increased by the state manager 112 to provide a seamless user experience for the user 1102.

[0135] In this exemplary scenario 1100, the inertial data provided by the sensor 108 causes the state manager 112 to confirm, with a higher level of confidence, that the user 1102 intends to interact with the smartphone 1104 and thus that the user 1102 desires to be authenticated, thereby justifying additional power. This is just one exemplary scenario showing how inertial data from the IMU and radar data from the radar system can be used to authenticate the user quickly, easily, and with reduced power consumption.

[0136] Lowering the high-level state FIG. 12 shows an exemplary method 1200 for lowering the high-level state via an IMU and a radar. The method 1200 is shown as a set of blocks that identify the actions to be performed, although the order or grouping by which each block performs the action is not indicated It is not necessarily limited to the combination. Also, to provide a wide range of additional and / or alternative methods, including other methods (e.g., methods 1000, 1400, 1700, and 1800) described in this document, any one or more of the operations may be repeated, combined, rearranged, or linked. In parts of the following description, reference may be made to the exemplary operating environment 100 of FIG. 1, or an entity or process as detailed in other drawings, and these references are made for illustrative purposes only. The techniques are not limited to execution by one entity or multiple entities operating on one device.

[0137] Optionally, at 1202 and prior to operation 1204 or 1206, it is determined that the inactive period has expired. In contrast to any other conventional technique that relies solely on the expiration of the period, method 1200 may use or refrain from using the inactive period to reduce the high-level state for the user equipment. Although this inactive timer is not essential, the use of the timer, even a short timer, may save power in some cases. More specifically, when the last user action on the user equipment is received, e.g., the last touch on the touch screen or button, the last voice command, or the last gesture input is received by the user equipment, the inactive timer starts. Note that some conventional techniques use only the timer, so conventional timers often last for several minutes (e.g., 1 minute, 3 minutes, 5 minutes, or 10 minutes), while method 1200 can use relatively short periods such as 0.5 seconds, 1 second, 3 seconds, 5 seconds, 10 seconds, or 20 seconds. By doing so, the possibility that the user equipment discloses information and enables inappropriate access is very low, while the use of a short inactive period can operate to save some amount of power by refraining from performing operations 1204 and / or 1206 during the inactive period.

[0138] At 1204, movement is determined during a high-level state of a user device with which the user is interacting or has recently interacted. The movement manager 110 determines this movement based on inertial data received from a sensor 108 integrated with the UE 102. As shown using the dashed arrow, this operation is optional and may respond to operations 1206 and / or 1202 (not shown). This determined movement may be one or more of the various movements described above, such movements indicating that the user 120 has picked up the UE 102, is walking with the UE 102, has placed the UE 102, has put it in a pocket or enclosure, or is simply near or touching the UE 102. In some cases, the movement manager 110 determines whether the movement is sufficient or not to change the state of the UE 102, and thus passes the determination to the state manager 112. Examples include those described above that do not exceed a threshold movement, those caused by ambient vibrations, and those that are not a sufficient change to an ongoing movement while moving. For this reason, the movement manager 110 may determine that the UE 102 is moving because the user 120 is walking with the UE 102, but the movement may be determined not to be a sufficient change to indicate that the user 120 may disengage from the UE 102. Another way to look at this is that the movement may be based not only on the current movement of the UE 102 but also on a change. Exemplary changes include moving and then not moving, for example, the user walking with the UE 102 and then placing it on a table. The inertial data from the sensor 108 may not capture the user 120 placing the UE 102 on the table, but the determination that the inertial data shows little or no movement if there was a previous movement (the user 120 walking with the UE 102) may still be determined as a movement at operation 1204 based on this previous movement.

[0139] More specifically, the technique can adapt the state of the user device to the user's engagement. For this reason, in some cases, due to the user being highly engaged with the user device, the The user equipment is in a high-level state. For example, in method 1200, prior to operation 1204 or 1206, it may be determined that the user is interacting with the user equipment. This determination of user involvement may be based on prior radar data indicating the intention of the user attempting to engage, or may be based on voice or touch input from the user, commands or inputs received from the user through voice or touch sensors, a successful authentication process, etc.

[0140] At 1206, an intention to disengage is determined based on the radar data and by the user equipment. The radar manager 106 receives radar data from the radar system 104 and uses this radar data to determine whether the user intends to disengage from the UE 102. This intention to disengage includes various types as described above, such as the user 120 pulling their hand away from the UE 102, a change in face orientation with respect to the UE 102, the user 120 turning their face away from the UE 102, or orienting their back towards the UE 102.

[0141] As indicated using the dashed arrow, this operation 1206 is optional and may respond to operation 1204 (and / or 1202, not shown). In these cases, the state manager 112 or the radar manager 106 acts to conserve power by refraining from determining the intention of the user 120 attempting to disengage until movement is determined, and for the determination of movement at 1204, by doing the opposite. By doing so, power can be conserved. For this reason, the power management module 620 may be directed by a technique to keep the radar system 104 at reduced power until movement is determined at 1204. Once movement is determined, the state manager 112 powers up the radar system 104 to the power management module 620 in preparation for determining whether the user 120 is acting in a manner indicating an intention to disengage.

[0142] At 1208, in response to a determination of an intention to deactivate movement and / or participation, the high-level state of the user equipment is reduced to an intermediate or low-level state. More specifically, refer to the exemplary high-level state 1208-1, which can be one or more states related to access, power, or information, such as those shown in FIG. 5 (high access state 502-1, high power state 504-1, or high information state 506-1). The state manager 112 determines to reduce one or more of the states of the UE 102 in response to a determination of movement, or an intention to deactivate participation, or both. This is shown in FIG. 12 using arrows indicating a reduction from the high level 1208-1 to the intermediate level 1208-2 or the low level 1208-3. These are only two of the various granularities of power, access, and information. As shown in FIG. 5, the intermediate level 1208-2 and the low level 1208-3 each include the intermediate access state 502-2, the intermediate power state 504-2, and the intermediate information state 506-2 described above. The low level 1208-3 is exemplified by three low states: the low access state 502-3, the low power state 504-3, and the low information state 506-3. These states have been described in detail above. Note that any one, two, or all three of these states can be reduced by the state manager 112 to the same level or different levels during operation 1208. For this reason, the state manager 112 may reduce the high access state 502-1 to an intermediate or low state while keeping the power state and the information state at a high level or a mixed level. Similarly, the state manager 112 may keep the UE 102 in the high access state 502-1 (e.g., an "unlocked" state) while reducing the power state 504 to the low power state 504-3.

[0143] As an example, consider the application of method 1200 to scenario 1300 shown in FIG. 13. Scenario 1300 includes three parts, and each part follows the previous part in the order of the passage of time. Prior to the first part of scenario 1300, user 1302 actively participates with smartphone 1304, and smartphone 1304 is in a high-level state, namely, a high power state, high access Assume that it is in the sessile state and the high-information state. In the first part shown in scenario part 1300-1, user 1302 walks to the table and places smartphone 1304 on the table. In operation 1204, sensor 108 receives inertial data regarding the contact of smartphone 1304 with the table or the lack of inertial data if the inertial data indicated movement (based on user 1302 walking with smartphone 1304) before it was placed on the table. Based on any or both of these inertial data, movement manager 110 determines the movement of smartphone 1304 and passes this determination to radar manager 106 and / or state manager 112.

[0144] Assume that radar manager 106 provides or has already provided radar field 118 (not shown for visual simplicity, see for example FIG. 1) in immediate response to the movement data and thus receives radar data indicating the position of user 1302's body, etc. Based on this radar data, in scenario part 1300-1, radar manager 106 determines, in operation 1206 regarding the placement of the body, arms, and hands, that user 1302 does not intend to disengage over the first iteration (and perhaps multiple other iterations). This is due to user 1302 having a body orientation towards smartphone 1304 and the user's hands and arms being oriented towards smartphone 1304. For this reason, high-information state 1306-1 is not changed.

[0145] However, in scenario part 1300-2, it is assumed that approximately 2 seconds later, user 1302 picks up their coffee cup and starts walking away while turning their back to smartphone 1304. At this point, based on the fact that the body orientation of user 1302 is partially facing away from smartphone 1304 and the arms and hands of user 1302 are oriented towards the coffee cup rather than smartphone 1304, radar manager 106 determines that user 1302 intends to disengage from smartphone 1304. Radar manager 106 passes this determination to state manager 112.

[0146] In operation 1208, in response to receiving the motion determination and the determination of the intention to disengage, state manager 112 lowers the information state of smartphone 1304 from the high information state 1306-1 shown in scenario part 1300-1 to the intermediate information state 1306-2. These exemplary information states are shown by the fact that the information displayed in scenario part 1300-1 indicates content consisting of two text messages and a time. As soon as user 1302 changes their body orientation and picks up the coffee cup, the information state is lowered to the intermediate information state 1306-2, which is indicated by the reduced information about the time and the text message (the name of the sender is shown, but the context is not shown). This intermediate amount of information may be useful to user 1302. This is because user 1302 may change their mind about their involvement or may want to look at smartphone 1304 again to see if a new notification, such as a text from another person, has arrived.

[0147] In addition to, or instead of, showing the intermediate information state 1306-2, and as part of operation 1208, state manager 112 may progress to a low level either immediately or after first reaching an intermediate state. Here, assume that in response to additional determination by radar manager 106 indicating that user 1302 intends to disengage or a higher confidence level thereof (e.g., here, shown with high confidence as user 1302 is currently several meters away and has their back fully turned towards smartphone 1304), state manager 112 further reduces the information state to low information state 1306-3, shown as scenario portion 1300-3 presenting only the current time.

[0148] This example shows a change to the information state, but access and power may be changed as well or instead. This is partially shown using unlock icon 1310 shown in scenario portion 1300-1 indicating high level access (e.g., high level access 502-1 in FIG. 5). In scenario portion 1300-2, after state manager 112 receives an intention to disengage from movement data, state manager 112 reduces access to a low level, which is shown to the user using lock icon 1312. Further, the power state may be changed, such as by reducing the brightness of the display of smartphone 1304 in scenario portion 1300-2 and / or 1300-3 (not shown).

[0149] Maintaining an authenticated state FIG. 14 shows an exemplary method 1400 for maintaining an authenticated state. Method 1400 is shown as a set of blocks that identify the actions to be performed, but is not necessarily limited to the order or combination shown for performing the actions by each block. Also, one or more of the actions may be repeated, combined, rearranged, or linked to provide a wide variety of additional and / or alternative methods, including the other methods described in this document (e.g., methods 1000, 1200, 1700, and 1800). In the following portions of the description, reference may be made to the exemplary operating environment 100 of FIG. 1, or to entities or processes as detailed in other drawings, and such references are made for illustrative purposes only. The techniques are not limited to execution by one entity or multiple entities operating on one device.

[0150] Prior to describing method 1400, note that any of the methods described above may be combined, in whole or in part, with method 1400. For example, consider the execution of method 1000 of FIG. 10. This method 1000 describes an example of authentication management that results in the authentication of a user. In response to this authentication, the user device becomes authenticated. This state is described in more detail above. Thus, method 1000 (or some other aspect of user authentication) is performed prior to method 1400.

[0151] At 1402, during the authenticated state of the user equipment, a potential disengagement by the user of the user equipment is determined. This determination of potential disengagement by the user may include determining the intention of the user attempting to disengage, as described above, and other determinations described below. Also, as described above, the authenticated state permits access by the user to one or more of the data, applications, functions, accounts, or components of the user equipment. Examples of the authenticated state include the high access state 502-1 and the intermediate access state 502-2 described above with respect to FIG. 5. Any of these access states may be permitted by UE102 when in the authenticated state (often based on user preferences or operating system default settings), but the authenticated state assumes a previous authentication by the user. However, preferences or settings selected by the user may permit high or intermediate access of UE102 without authentication. Thus, while the authenticated state may include the access permitted by the high access and intermediate access states described above, high and intermediate access are not necessarily in the authenticated state.

[0152] As shown in FIG. 14, the determination of potential disengagement is optional and may be made in response to (or by performing) other aspects described herein, such as operation 1404 or operation 1406, and by determining the intention to disengage at operation 1206 of method 1200. At 1404, expiration of an inactive period is determined. As described above, this inactive period may start when the last user action is received and the active engagement with the user equipment ends (or when last received), or when the last intention to engage is determined. For example, when the user last touches a touch-sensitive display or button, when the last received voice command is spoken, or when the last determined touch-independent gesture (e.g., a gesture determined using the radar system 104 described above) is performed, an inactive timer (e.g., a period) starts.

[0153] At 1406, based on the inertial data of an inertial measurement unit (IMU) integrated with the user equipment, the movement of the user equipment is determined. Exemplary movements and inertial data are described above, such as the inertial data received from sensor 108 in FIG. 1. Thus, one way the method can determine that the movement potentially disengages, for example, when the user places UE 102 in a locker, bag, or pocket (although placing it in a bag or pocket may later be determined to be a passive engagement as described below).

[0154] At 1408, based on the radar data, passive engagement of the user with the user equipment is determined. This determination of passive engagement may respond to the determination of potential disengagement at 1402 (shown using a dashed arrow), or it may be independent of, or consistent with, that determination. Performing operation 1408 in response to the determination of potential disengagement can, in some cases, save power or reduce latency. For example, method 1400 may power up components of the radar system 104 (see also FIGS. 6-1 and 6-2) in response to the determination of potential disengagement. This can save power as described above, or provide additional time for the radar system 104 to determine whether the user is passively engaged with the radar system 104.

[0155] In the situation of FIG. 1, the radar manager 106 determines that the user 120 is passively involved with the UE 102. This passive involvement can be determined by the radar manager 106 in a plurality of ways that can be exclusive or overlapping with each other. For example, the radar manager 106 can determine that the user is passively involved based on radar data indicating that the user 120's hand holds the user equipment 102 in an orientation in which the display 116 of the user equipment 102 is maintained. Thus, if the user 120 holds the UE 102 stably (or stably enough to view the content or show the content to others), the user 120 is passively involved. Other examples of determining passive involvement have been described above and include the user 120 looking at the UE 102 or orienting their body towards the UE 102.

[0156] Furthermore, the radar manager 106 can determine passive involvement based on radar data indicating the presence of the user 120, such as the user 120 being within 2 meters of the UE 102. Other distances such as 1.5 meters, 1 meter, and even 0.5 meters can be used as well or instead. In practice, the radar manager 106 can determine that the user 120 is passively involved by the user 120 generally being within reach of the UE 102. The radar manager 106 may explicitly determine that the user 120 is passively involved by indicating so, or may simply pass information indicating the distance from the UE 102 to the state manager 112. The state manager 112 then determines passive involvement based on the proximity of the user 120 and, in some cases, situations such as the presence (or absence) of others, whether the user 120 is in a vehicle (car, bus, train), and whether the user is facing a desk. For example, a user sitting at home may have a greater permitted distance than a user sitting in a crowded coffee shop or on a train.

[0157] At 1410, in response to a determination of passive user engagement with the user equipment, the authenticated state is maintained. This maintenance of the authenticated state can continue until another potential disengagement is determined or over a period of time, after which the method 1400 can be performed again. An example of the authenticated state is the high access state 502-1 in FIG. 5. In many cases, this authenticated state is an unlocked state for UE102, but in some other cases, the authenticated state permits a portion of access to UE102, such as the intermediate access state 502-2 described above, without full access. Maintaining the authenticated state for UE102 does not necessarily require other states to be maintained. For example, if the user 120 is within 2 meters of UE102 but it is not known whether the user is looking at or oriented towards UE102, the state manager 112 can lower the power state or information state of UE102, for example, from the high power state 504-1 and high information state 506-1 described in FIG. 5 to an intermediate or low power state or information state. However, if the passive engagement includes the user looking at UE102, the power state or information state can also be maintained, for example, to continue presenting content to the user 120 through the display 116.

[0158]

[0159] ​Optionally, method 1400 may proceed to operation 1412, where the presence or intent to engage of a non - user is determined based on radar data. This radar data may be the same radar data as that on which passive engagement was based, or may be later - received radar data, such as radar data from radar system 104 received seconds or minutes after the radar data on which passive engagement was based. Thus, at 1412, radar manager 106 determines that a non - user is present or intends to engage with UE 102. Accordingly, if a non - user reaches for UE 102 or looks at the display 116 of UE 102, radar manager 106 may determine this presence or intent and pass it to state manager 112.

[0160] At 1414, in response to the determination that a non - user is present or intends to engage with the user equipment, the maintenance of the authenticated state is aborted. Thus, if a non - user approaches, reaches for, or looks at the display 116 of UE 102, state manager 112 aborts maintaining the authenticated state (or actively de - authenticates UE 102). Along with this abort, state manager 112 may also degrade other states, such as an information state that is effective in reducing or eliminating the information presented to the non - user. For example, assume an authenticated user is reading a private email on a subway train. If a person sitting behind the user looks at the display, perhaps to read the private email, state manager 112 can lock UE 102 and abort the display of the private email. This can be performed quickly and seamlessly, further enhancing the user's privacy.

[0161] At 1416, optionally, after stopping maintaining the authenticated state, the method can return to the authenticated state in response to a determination that the non - user is no longer present or no longer intends to participate. Continuing with the above example, when a non - user of the subway train looks away from the display 116 of the UE102, the state manager 112 may re - authenticate the user 120 through the authentication process or simply by switching to the authenticated state without re - authenticating. Thus, the user 120 can easily return to the previous state as soon as the condition that caused the de - authentication is aborted. Although some authentication processes such as the systems and processes described herein are fast and power - efficient, not performing the authentication process can be even faster and more power - efficient. As soon as returning to the authenticated state, the state manager 112 can return the information state to the previous level with content that is consistent with the content last presented to the user 120. In this example, when the non - user looks away, the display 116 presents the private email at the same location as was last presented to the user 120 by the UE102. By doing so, seamless management of authentication and improved information privacy are provided to the user. Note that selections by the user 120, such as a user selection to de - authenticate, can override the operation of the technique. In some cases, the user 120 simply turns off the UE102, which is permitted by the method described herein. When the non - user looks away, the display 116 presents the private email at the same location as was last presented to the user 120 by the UE102. By doing so, seamless management of authentication and improved information privacy are provided to the user. Note that selections by the user 120, such as a user selection to de - authenticate, can override the operation of the technique. In some cases, the user 120 simply turns off the UE102, which is permitted by the method described herein.

[0162] Consider another example shown through scenario 1500 in FIG. 15. Scenario 1500 includes four parts. In the first part 1500 - 1, assume that the user 1502 is authenticated against the smartphone 1504 through authentication information or face feature analysis, etc., and thus the smartphone 1504 is in the authenticated state 1506. This authenticated state 1506 enables the user 1502 to access the content of the smartphone 1504, which is shown by the user 1502 accessing the content of the smartphone 1504 by watching a TV program about a volcanic eruption.

[0163] Scenario 1500 is shown as branching along two different paths. In one path, if user 120 stops touching or providing input to smartphone 1504, here, when user 120 relaxes and watches a TV program, an inactivity timer starts. In another case, the inactivity timer may or may not start, and without its expiration, a potential disengagement will be judged. For this reason, in scenario part 1500-2, after 3 minutes of inactivity, the inactivity timer expires. Returning to FIG. 14, operation 1402 determines that a potential disengagement by the user has occurred due to the expiration of the inactivity period in operation 1404. For the second path shown in scenario part 1500-3, operation 1402 determines that a potential disengagement by the user has occurred by determining that movement of smartphone 1504 has occurred based on inertial data through performing operation 1406. The cause of this movement is that user 1502 places his foot on the edge of the table on which smartphone 1504 is placed.

[0164] In response to any of these determinations of potential disengagement, radar manager 106 determines, based on radar data, that user 1502 is passively engaged with smartphone 1504. This operation is performed at 1408. Here, assume that the presence of user 1502, or that user 1502 is looking at smartphone 1504, is determined. All of these indicate that user 1502 is passively engaged.

[0165] In response to this, at operation 1410, state manager 112 maintains the authenticated state. All of this can be done seamlessly and without user 1502 noticing that it has been done. As shown in scenario part 1500-4, smartphone 1504 simply continues to present the TV program through either path.

[0166] Consider another scenario 1600 of FIG. 16 that can follow scenario 1500 or be an alternative independent scenario. Scenario 1600 includes three scenario parts. In the first scenario part 1600-1, user 1502 is watching a TV program about a volcano marked with content 1602 of smartphone 1504, similar to what is shown in FIG. 15. During this presentation of the program, smartphone 1504 is in an authenticated state, such as the authenticated state 1506 described in FIG. 15.

[0167] However, in scenario part 1600-2, non-user 1604 sits on the bench with user 1502. Since this non-user 1604 is a colleague of user 1502, user 1502 starts talking to non-user 1604 while facing them. These actions of user 1502, such as facing, talking, or both, can be considered potential disengagement. If it is considered potential disengagement by user 1502, state manager 112 reduces the state of smartphone 1504, for example, by reducing the access state or information state as described in FIGS. 5 and 12 (e.g., operations 1206 and 1208 of method 1200). However, assume that radar manager 106 determines the presence of non-user 1604 through operation 1412 of method 1400 and based on radar data. Based on this presence of non-user 1604, state manager 112 stops maintaining the authenticated state 1506 after state manager 112 previously acted to maintain the authenticated state of smartphone 1504 (e.g., through operation 1410 shown in FIG. 15). For this reason, state manager 112 can reduce smartphone 1504 to an unauthenticated state 1604 as shown in the enlarged view of scenario part 1600-2. This change is shown to user 1502 through lock icon 1606 and by stopping the presentation of content 1602.

[0168] However, assume that radar manager 106 determines the presence of non-user 1604 through operation 1412 of method 1400 and based on radar data. Based on this presence of non-user 1604, state manager 112 stops maintaining the authenticated state 1506 after state manager 112 previously acted to maintain the authenticated state of smartphone 1504 (e.g., through operation 1410 shown in FIG. 15). For this reason, state manager 112 can reduce smartphone 1504 to an unauthenticated state 1604 as shown in the enlarged view of scenario part 1600-2. This change is shown to user 1502 through lock icon 1606 and by stopping the presentation of content 1602.

[0169] In scenario portion 1600-3, non-user 1604 leaves, and user 1502 looks at smartphone 1504 again. Radar manager 106 determines that non-user 1604 is no longer present and indicates this determination to state manager 112, and state manager 112 then returns smartphone 1504 to an authenticated state 1506. Note that state manager 112 may also require a determination that user 1502 intends to interact with smartphone 1504, or may simply return to the authenticated state based on the fact that non-user 1604 has left the vicinity of smartphone 1504. Also, the techniques described in this document can seamlessly return the user to the location where the user interrupted work, thereby providing an excellent user experience. This is shown in FIG. 16, where state manager 112 returns smartphone 1504 to the same or nearly the same point as was last presented to user 1502 for the same television program. For some embodiments, the technique can instruct whether smartphone 1504 returns to an authenticated state in response to a determination in step 1416 that no non-user is present or intends to interact in a setup screen or similar device configuration screen, or whether smartphone 1504 remains in an unauthenticated state until a more stringent authentication process (e.g., step 1006 described above) that uses power-consuming components of the authentication system is performed. In other words, the technique can provide a user-selectable setting through a setup configuration or similar device configuration that leaves smartphone 1504 unauthenticated even if the trace of the non-user is no longer present once the trace was present.

[0170] Gesture Recognition Management FIG. 17 illustrates an exemplary method 1700 for radar-based gesture recognition using situation-aware gating and other situation-aware controls. Method 1700 is shown as a set of blocks that identify the operations to be performed, but are not necessarily limited to the order or combination shown for performing the operations by each block. Also, any one or more of the operations may be repeated, combined, rearranged, or linked to provide a wide variety of additional and / or alternative methods, such as, for example, methods 1000, 1200, 1400, and 1800. In parts of the following description, reference may be made to the exemplary operating environment 100 of FIG. 1, or to entities or processes as detailed in other drawings, and such references are made for illustrative purposes only. The techniques are not limited to execution by one entity or multiple entities operating on one device.

[0171] In operation 1702, sensor data from a plurality of sensors 108 is received. For example, a proximity sensor 208 (e.g., a radar system 104 configured as a proximity sensor by operating in a proximity mode) generates sensor data indicative of proximity to an object. The IMU 408 can generate sensor data indicative of movement, and other sensors 108 can generate other sensor data used to define the situation. are radar systems 104) generates sensor data indicating proximity to an object. The IMU 408 can generate sensor data indicating movement, and other sensors 108 can generate other sensor data used to define the situation.

[0172] The radar system 104 may be operable in a low-power proximity mode (e.g., low-power state 504-3) to generate sensor data of sufficient resolution and quality for detecting proximity. The radar system 104 may also be operable in a high-power gesture recognition mode (e.g., high-power state 504-1) to generate improved sensor data compared to the sensor data generated in the proximity mode. In the gesture recognition mode, the sensor data generated by the radar system 104 is of higher resolution or better quality compared to the proximity mode. This is because the sensor data is used for more complex gesture recognition tasks. The sensor data received from multiple sensors in operation 1702 may indicate gross movement and proximity, while the sensor data collected from the radar system 104 for performing radar-based gesture recognition may indicate more precise movement, proximity, or occlusion.

[0173] The sensor data may indicate proximity as a binary measurement of an object or as a variable measurement that further specifies the proximity to the object. Proximity can indicate whether the radar system 104 or other parts of the UE102 are blocked by an object (meaning that the user 120 relative to the UE102, or vice versa, is blocked by the object), where a larger amount of presence indicates occlusion and a smaller amount of presence indicates little or no occlusion. The sensor data can define movement that is effective for determining the position, speed, acceleration, velocity, rotation, orientation, or other movement or positioning characteristics of the UE102.

[0174] In operation 1704, the situation of the UE is determined. The sensor data obtained in operation 1702 indicates the operating environment of the UE 102 when the user 120 interacts with the UE 102. The sensor data may include a pattern or signature indicating whether the movement is intentional or not. The UE 102 may include, or may access, a machine learning activity classifier trained using machine learning to recognize a pattern or signature in the sensor data corresponding to a specific user activity or device situation. The machine learning activity classifier outputs notifications to applications and other subscribers that use activity recognition for other tasks. The acceleration or vibration identified in the movement data corresponds to similar vibrations and accelerations that are recorded as sensor data by the IMU 408 or other sensors of the sensor 108 when the user 120 is walking with the UE 102 or moving in other ways.

[0175] The recognized activity or movement may indicate different situations. Examples of situations include a walking situation, a cycling situation, a driving situation, a riding situation, or other activity situations corresponding to the recognized activity. Movements typically include position and orientation, as well as movements or lack of movements, that are observed when the user 120 is looking at or holding the UE 102. Lack of movement may indicate a seated situation, a stationary situation, an unused situation, or an enclosed situation. Opposite movements may correspond to opposite activities. For example, one movement may indicate that the user is picking up the UE 102, and the opposite or different movement may indicate that the user is placing down the UE 102.

[0176] In operation 1706, it is determined whether the situation meets the requirements for radar-based gesture recognition. If the sensor data from the sensor 108 (e.g., proximity sensor 208 and IMU 408) is consistent over time with the sensor data that the UE 102 is expected to detect when radar-based gesture recognition is typically received, the UE 102 determines that the situation is It may be determined that the requirements are met. If the situation does not meet the requirements for radar-based gesture recognition, UE102 may determine that the opposite is true and prevent or otherwise discard the gesture recognized by radar system 104.

[0177] For example, UE102 can make the current situation of UE102 a condition for radar-based gesture recognition. When the user 120 holds UE102 while walking, the situation meets the requirements for radar-based gesture recognition, but when the user 120 does not hold UE102 while walking, the situation does not meet the requirements. However, carrying UE102 in a pocket or backpack while walking is not a situation that meets the requirements for radar-based gesture recognition (except when gesture detection through intervening materials is permitted).

[0178] The enclosed situation is when sensor data indicates that UE102 is positioned in a pocket of the clothes the user 120 is wearing, a backpack, a briefcase, or a partition of a suitcase, a storage bin of an airplane, a taxi, a car, a ship, a bus, or a train, a vehicle console or glove box, or other housing. When sensor data indicates that the user 120 is holding UE102, the holding situation or carrying situation is identified. The stationary situation is evident from sensor data indicating that UE102 is not being held, not moving, or hardly moving relative to the surface on which UE102 is placed. The moving situation indicates that UE102 is moving, for example, when the user 120 is walking, driving, cycling, or moving in some other way with UE102, regardless of whether UE102 is being held or enclosed.

[0179] The situation meets the requirements for radar-based gesture recognition, based in part on whether the UE 102 is moving, being held, or being carried. For example, being held and moving is a situation where the radar system 104 can recognize radar-based gestures, while being carried and moving may not be a situation where the radar system 104 can recognize radar-based gestures.

[0180] Whether the situation is met can further depend on the orientation, specifically the carrying orientation. When the user 120 is walking while holding the UE 102, the situation may still not meet the requirements for radar-based gesture recognition if the user is not holding the UE 120 in a particular way. For example, while the user 120 is walking, holding the UE 102 in a landscape orientation and / or a portrait-down orientation (e.g., the touch screen of the UE 102 points near the ground) may not meet the requirements for radar-based gesture recognition. This is because in this situation, the user 120 probably does not want to interact with the UE 102. Conversely, while the user 120 is walking, holding the UE 102 in a different orientation (e.g., a portrait-up orientation where the touch screen of the UE 102 points at the sky or towards the user 120's face) may meet the requirements for radar-based gesture recognition. This is because the user 120 is probably looking at the touch screen of the UE 102 while walking.

[0181] UE102 can condition radar-based gesture recognition on whether or not user 120 is holding UE102 and how UE102 is being held. For example, if user 120 is not holding UE102 while cycling or driving, for example, if UE102 is fixed to a mounting bracket on a bike frame or mounted in an automotive vent or dashboard and in a stationary situation during cycling or driving, the situation meets the requirements for radar-based gesture recognition. However, a similar cycling or driving situation where the user is holding UE102 may not meet the requirements for radar-based gesture recognition.

[0182] Radar-based gesture recognition can be conditioned by UE102 based on occlusion from or proximity to an object. For example, in response to detecting proximity to an object while UE102 is already enclosed or in a stationary situation, radar system 104 enables gesture recognition model 621. Conversely, in response to detecting occlusion by an object while UE102 is already enclosed or in a stationary situation, radar system 104 may disable gesture recognition model 621 in this case. For example, placing UE102 on a flat surface with the screen facing up (touch screen facing up) may be a stationary situation where proximity to or non-occlusion by an object is detected and thus gesture recognition is enabled. Placing UE102 on a flat surface with the screen facing down (touch screen facing down) is a reverse stationary situation where occlusion is detected and thus gesture recognition is gated.

[0183] Significant movement can condition the gesture recognition model 621. If the UE 102 is in a situation of significant movement, where the UE 102 experiences frequent or strong movement, or a change in movement, the situation may not be very suitable for radar-based gesture detection. For example, when the user 120 who carries the UE 102 in their hand runs at a brisk pace, the radar system 104 gates the gesture recognition model 621 to ensure that the radar system 104 does not accidentally trigger an event conditioned by a gesture.

[0184] The radar system 104 can apply different sensitivity levels for different types of gestures or for different types of situations. A situation with significant movement may trigger gating for most radar-based gestures, while a situation with less movement may trigger gating for only some of the radar-based gestures. As an example, the radar system 104 may recognize a gesture of coarse adjustment (e.g., the whole hand) in a high-vibration manufacturing situation, but the same situation may not be suitable for radar-based gestures of specific fine adjustments (e.g., each finger) where the UE 102 or the user 120 is moving unstably. Instead of attempting to recognize radar-based gestures of fine adjustments in a high-vibration situation, the radar system 104 gates the gesture recognition features for gestures of fine adjustments while continuing to recognize gestures of coarse adjustment in the same situation. Since the radar system 104 applies different sensitivity levels to gestures of fine adjustments, they are triggered more easily than gestures of coarse adjustment. The radar system 104 applies different sensitivity levels to gestures of coarse adjustment so that they are not triggered as easily as gestures of fine adjustments.

[0185] In a marine situation, user 120 interacts with UE102 as a passenger on a ship. User 120 may hold UE102, or UE102 may be a computing device built into the ship. The ship moves with the ocean waves. In a stormy environment, radar system 104 recognizes that it may be difficult to recognize certain radar-based gestures when UE102 is undergoing large changes in pitch or orientation, and can gate the radar-based gestures without risking outputting false judgments. When the stormy environment calms down and the pitch and orientation fluctuations subside, radar system 104 automatically stops gating and enables the radar-based gestures that were gated during the storm.

[0186] Radar system 104 may gate all radar-based gestures for a particular situation, or may gate only certain types of radar-based gestures. For example, for a gesture of reaching out a hand to grasp, specifically for the gesture of reaching out a hand to try to pick up UE102, radar system 104 can reduce false judgments to face recognition system 114 by recognizing from sensor data when user 120 reaches out a hand to UE102 and then picks up UE102 to trigger face recognition system 114. This is in contrast to triggering face recognition system 114 in response to only recognizing the reaching out of a hand. If sensor data indicates that UE102 is in a quiet environment or a noisy environment, or an environment with intermittent communication signals, radar system 104 can gate radar-based gestures for making a phone call. When radar system 104 determines that UE102 is near a workplace or a laptop computer on a desk where user 120 likely wants to use gesture recognition model 621 to make a phone call, radar system 104 automatically ungates and enables the radar-based gestures for making a phone call.

[0187] The UE 102 can use whether the situation indicates that the user 120 is holding the UE 102, whether the situation indicates that the user 120 is walking, or both, as conditions for radar-based gesture recognition. If the UE 102 determines that the user 120 is holding the UE 102 and the user 120 is walking, the situation meets the requirements for touch-independent gesture recognition. If the UE 102 determines that the user 120 is not holding the UE 102 and the user 120 is walking, the situation does not meet the requirements for touch-independent gesture recognition. If the sensor data from the proximity sensor 208 and the IMU 408 is consistent over time with the sensor data that the UE 102 expects to detect when the user 120 is walking and holding the UE 102, the UE 102 may determine that the situation meets the requirements. If the situation does not meet the requirements, the UE 102 may determine the opposite to be true and discard the gesture recognized by the radar system 104.

[0188] In the situation of reaching out and grasping, when the user 120 reaches out to the UE 102 while the UE 102 is placed on the table with the screen facing up. The user 120 may be reaching out to the UE 102 to grasp it. The user 120 may also be reaching out to grasp something on the other side of the UE 102. Determining in 1706 that the situation does not meet the requirements for radar-based gesture recognition may be in response to determining that after an object approaches the UE 102, the user 120 has not picked up the UE 102 by hand. If the user does not grasp and pick up the UE 102 after reaching out (for example, the user 120 approaches), the UE 102 gates the output from the gesture recognition model 621 (for example, to prevent the authentication algorithm from performing face authentication), and prevents the subscriber (for example, an application, component, system service) from obtaining gesture cues. Determining in 1706 that the situation meets the requirements for radar-based gesture recognition may be in response to determining that after an object approaches the UE 102, the user 120 has picked up the UE 102 by hand. If the user grasps and picks up the UE 102 after reaching out, the UE 102 enables the output from the gesture recognition model 621 (for example, to enable the authentication algorithm to perform face authentication), and enables the subscriber to obtain gesture cues. Using situation-aware gating and other situation-aware controls in this way reduces misjudgment when reaching out and grasping.

[0189] Other sensors among the sensors 108, such as an ambient light sensor, a barometer, a position sensor, an optical sensor, an infrared sensor, etc., provide signals to the UE 102 to further define the situation of the UE 102 and can improve gesture recognition and other described techniques. Determining in 1706 that the situation meets the requirements for radar-based gesture recognition is the position information, time, atmospheric pressure, ambient light, ambient sound, and other sensors for defining the situation for gating or not gating the radar system 104 It may respond to information. For example, while detecting noisy and frequent ambient noise, identifying UE102 as being near a movie theater location and in low light conditions is not a suitable situation for radar-based gesture recognition. On the other hand, while detecting low light conditions and noisy ambient noise, identifying UE102 as being near a railway station is a suitable situation for radar-based (e.g., touch-independent) gesture recognition.

[0190] If the situation in operation 1706 does not meet the requirements for radar-based gesture recognition using the radar system, the radar data acquired by the radar system is gated, and the method proceeds to B (described below in the description of FIG. 18). If the situation in operation 1706 meets the requirements for radar-based gesture recognition using the radar system, in operation 1708, the radar data acquired by the radar system is input into a model that determines radar-based gestures from the input radar data.

[0191] In 1708, by inputting the radar data acquired by the radar system 104 into the gesture recognition model 621, the gesture recognition model 621 will perform a gesture recognition method. The radar system 104 may operate in a high-power gesture recognition mode to acquire radar data having sufficient resolution, frequency, detail, and quality for radar-based (e.g., touch-independent) gesture recognition. The radar system 104 may further operate in other modes including a proximity mode or a standby mode. When multiple mode operations are supported, the radar system 104 can continue to operate in one or more modes even if different modes are disabled. For example, disabling radar-based gesture recognition may not affect the radar-based collision avoidance operation performed by the radar system 104. Some examples of the radar system 104 may not be multimode, and thus, disabling radar-based gesture recognition may disable the entire radar system 104.

[0192] In addition to being situation-aware, the gesture recognition model 621 may adjust the gating sensitivity based on the identity of the subscriber. A subscriber can be an application, service, or component that receives the output from the gesture recognition model 621. For example, the gesture recognition model 621 provides an interface through which an application or component of the UE102 (such as the authentication system 114, an operating system function or service, an application, a driver) can register with the gesture recognition model 621 and be assigned an identity. The subscriber indicates the gating sensitivity to be applied for different situations. The subscriber may indicate the type of gesture for which gating is to be applied or the type of radar-based gesture. For example, the operating system may provide access to functions through widgets on the lock screen user interface of the UE102. The widget may recognize radar-based gestures and may subscribe to the gesture recognition output from the gesture recognition model 621. In some situations, the output from the gesture recognition model 621 is gated to prevent the gesture cues from being used by the subscriber. In other situations, the output from the gesture recognition model 621 is permitted and the gesture cues are sent to the subscriber. In some situations, the output from the gesture recognition model 621 can be gated for one subscriber but not for another. For example, the same gesture recognition used by a widget subscriber in a particular situation may not be usable by a different subscriber who chooses to gate the gesture for that situation. For example, a face authentication application may not be able to use gesture information under certain conditions, while a widget on the lock screen can use gesture information. The gesture information may not be usable, while the widget on the lock screen can use the gesture information.

[0193] In operation 1406, the gesture recognition model 621 selects a gating sensitivity based on the identity of the subscriber. The gesture recognition model 621 determines whether the situation meets the requirements for radar-based gesture recognition using the radar system 104 based on the gating sensitivity associated with the identity of the subscriber.

[0194] In 1710, an operation is performed in response to the model determining a radar-based (e.g., touch-independent) gesture. The output from the gesture recognition model 621 indicates the gesture recognized from the radar data and can output the gesture indication to the subscriber.

[0195] The UE 102 may provide user interface feedback on the gating state of the radar system 104. The UE 102 can output an audible or visual display to the user, such as an audible or visual alert (e.g., "You are moving the device too much and the radar cannot sense your gesture"), control the lighting elements of the UE 102 to provide tactile feedback, or provide some other user interface feedback. The UE 102 may output an indication of the gating state as "gating" or "non-gating" to indicate whether the UE 102 is gating the output from the gesture recognition model 621. The indication of the gating state can indicate the reason for the gating (e.g., provide an indication of the situation characteristics or environmental characteristics that require gating). The indication of the gating state can indicate the level of gating (see, for example, FIG. 18 for exemplary levels of gating, including soft gating, hard gating, and non-gating).

[0196] UE102 can change the user interface and provide user interface feedback in other ways. For example, when UE102 is in use, the display is on, and UE102 is operating in a high-power state, the user interface feedback output from UE102 may depend only on sensor data from a motion sensor or other non-radar sensor. When the display is off or UE102 is in a lower power state, it may be prohibited to always operate the motion sensor or other non-radar-based sensors in an active state. UE102 may refrain from monitoring the motion sensor or other non-radar sensors, except in situations that meet the requirements for touch-independent gesture recognition. Thus, the user interface feedback is conditional on whether the gesture recognition model 621 can determine radar-based gestures.

[0197] For example, UE102 can provide a "gesture" user interface feedback element when soft or hard gating the radar system 104 and / or when the gating is stopped and radar-based gesture recognition resumes. A gesture user interface feedback element is an element perceptible to the user, such as a visual element that appears in the active area of the display. The gesture feedback element may also be (or include) a light element not on the display, a tactile element (such as a vibration element), and / or an audio element (such as a sound perceptible to the user), and may be presented on the display or along its edge, and may have any of various shapes, sizes, colors, and other visual parameters or characteristics. Examples of other visual parameters or characteristics include brightness, color, contrast, shape, saturation, or opacity.

[0198] Gesture Recognition Gating FIG. 18 illustrates an exemplary method 1800 for radar-based gesture recognition using situation-aware gating and other situation-aware controls. Method 1800 is shown as a set of blocks that specify the operations to be performed, but are not necessarily limited to the order or combination shown for performing the operations by each block. Also, one or more of the operations may be repeated, combined, rearranged, or linked to provide a wide variety of additional and / or alternative methods, such as, for example, methods 1000, 1200, 1400, and 1700. In parts of the following description, reference may be made to the exemplary operating environment 100 of FIG. 1, or to entities or processes as detailed in other drawings, and such references are made for illustrative purposes only. The techniques are not limited to execution by one entity or multiple entities operating on one device.

[0199] There are two general scenarios for performing operations 1802, 1804, and 1806 to employ gating of a radar-based detection system. One scenario is when the radar system 104 is covered or blocked by an object. The radar system 104 can be blocked or covered when the UE 102 is face down on a surface, or when it is in a pocket, handbag, bag, or other enclosure. The other scenario is when the UE 102 is experiencing significant movement. For example, if the user 120 who is carrying the UE 102 in their hand starts running, the UE 102 should not misinterpret touch-independent gestures using the radar system 104.

[0200] In 1802, it is determined whether to hard gate the radar-based gesture recognition by the radar system 104. A situation indicating whether the radar system is blocked by an object (e.g., from the user) is determined. The UE 102 selects from multiple levels of gating including hard gating and soft gating based on the situation, and then gates the gesture recognition model 621 accordingly.

[0201] As used herein, the term "soft gating" refers to an operation that prevents the radar-based gesture marks from being output to the subscriber by the radar system 104. Different from the hard gating where the radar system 104 operates in the low power mode or the intermediate power mode, soft gating occurs regardless of the power level of the radar system 104. Soft gating may occur by invalidating the output from the gesture recognition model 621, and in other cases, soft gating results from invalidating the input to the gesture recognition model 621. The gesture recognition model 621 may continue to recognize the radar-based gestures during soft gating. However, during soft gating, the radar system 104 does not share the recognized gestures with the subscriber (e.g., an application, a thread, an activity, a user interface object). The subscriber does not receive the marks of the recognized gestures for use when performing higher-level functions.

[0202] During soft gating, the gesture recognition model 621 is protected from the radar data collected by the radar system 104, and during other times of soft gating, although the radar-based gesture determination is made by the gesture recognition model 621 in any case, it is used internally by the radar system 104 for some other purpose (e.g., system services or stealth functions). During soft gating, UE102 may still perform lower-level support functions based on the gestures recognized by the gesture recognition model 621, but the support functions may be transparent to the subscribers and users of UE102. The support functions include learning, understanding, and acting based on gestures even during gating situations to minimize potential latency from soft-gating the future gesture recognition model 621.

[0203] Soft gating is contrasted with the term "hard gating". Hard gating, as used herein, refers to the operation that triggers the radar system 104 to function in a state where the radar system 104 does not recognize gestures from radar data. During hard gating, the gesture recognition model 621 is disabled. During hard gating, the radar system 104 can be used for other tasks other than gesture recognition. Depending on whether UE102 requires the radar system 104 for any other capabilities, other parts of the radar system 104 may or may not be similarly disabled during hard gating situations. For this reason, the radar system 104 may continue to perform other functions unrelated to radar-based gesture recognition, such as obstacle avoidance, but when hard gated, the gesture recognition model 621 of the radar system 104 does not output the signs of the recognized gestures, thereby providing some power consumption savings compared to soft-gating the radar system 104 or not gating at all. In addition to providing power consumption savings, hard gating is particularly useful for improving the user experience by preventing the subscribers of the radar system 104 from performing higher-level functions in response to incorrect or unintended inputs.

[0204] When UE102 soft gates the radar system 104, UE102 may have an improved latency in recovering from the non-active gesture recognition state. In hard gating, the increased latency in recovering from the non-active gesture recognition state (e.g., when the gesture recognition feature of the radar system 104 can be powered off) is offset by the power saved from not performing complex gesture recognition functions or splitting high-level functions from incorrect inputs. That is, hard gating the radar system 104 prevents unnecessary power consumption for UE102 to interpret gestures from radar data in situations where UE102 will probably not receive input from the user, but UE102 may be slower than when it is soft gated so that the radar system 104 transitions to the normal operating mode when gating is no longer necessary.

[0205] At 1804, UE102 performs hard gating by setting the radar system 104 to the intermediate power mode or the low power mode so that the gesture recognition model 621 cannot use the data output or other data for determining touch-independent gestures. When the radar system 104 is blocked, at 1804, the output from the radar system 104 is hard gated by disabling the gesture recognition model 621.

[0206] At 1802, for example when the radar system 104 is not blocked, at 1806, the output from the radar system is soft gated. UE102 soft gates the radar system 104 by refraining from inputting the radar data acquired by the radar system 104 into the gesture recognition model 621. Alternatively, UE102 soft gates the radar system 104 by preventing the gesture recognition model 621 from outputting signs of recognized gestures.

[0207] During the next execution of operations 1700 and 1800, radar system 104 can transition between non-gating, soft gating, and hard gating depending on the situation. For example, after soft gating or hard gating the radar system 104, the method of FIG. 18 returns to "A" and the start of operation 1700 of FIG. 17. After soft gating the radar system, if the situation indicates at 1706, 1802 that the radar system 104 is blocked by proximity to an object, the radar system 104 is hard gated.

[0208] Gating sensitivity FIG. 19 shows a decision tree implementing the methods of FIGS. 17 and 18. The portion of scheme 1900 may be performed by processor 608, computer processor 402, or other hardware circuitry. Scheme 1900 may be customized to support different types of electronic devices and radar-based applications.

[0209] When UE102 performs operation 1702, sensor data reception 1902 occurs. From operation 1702, UE102 uses the sensor data to perform motion detection 1904 and a proximity detection algorithm on the sensor data, and unfolds a situation during operation 1704 that includes determining whether the radar system 104 is blocked or whether an increase in the speed of motion is determined.

[0210] The sensitivities of motion detection 1904 and proximity detection 1906 can be selected to balance motion and gating behavior while being able to reject device motion. Two common scenarios illustrate the need for sensitivity adjustment. The first scenario is when the UE 102 is carried on the user's body side and is swaying beyond the user's body while the user 120 is walking slowly. Without continuous gating, the first scenario can cause significant false triggers. The second scenario is when the user 120 picks up the UE 102 to act on it. When picking up the UE 102 (from the body side, table, pocket, etc.) to interact, subsequent detection of natural body movements should trigger a rapid gating response from the UE 102. A lower gating sensitivity is required so that the response is fast enough for the user and causes no interference delay. During operation 1706, a gating decision 1908 is made, which results in one of three gating modes: off 1910, soft gating 1912, or hard gating 1914.

[0211] Gating state machine Figure 20 shows a state diagram for a state machine 2000 that implements the methods of FIGS. 17 and 18. The state machine 2000 is a gating state machine and may be executed as part of the radar system 104.

[0212] The state machine 2000 includes a plurality of states 2002, 2004, and 2006, each linked by respective situation-aware transition functions 2008-1 to 2008-6 (collectively "function 2008"). Each of the functions 2008 receives at least a portion of the sensor data or a derivative thereof as a variable input. For ease of explanation, the state machine 2000 includes only three states 2002, 2004, and 2006. In other examples, four or more states are used by the state machine 2000. The state machine 2000 transitions between states 2004 and 2006 based on the function 2008.

[0213] The state machine 2000 includes a non-gating state 2002 in which radar-based gesture recognition using the radar system 104 is enabled. In the soft-gating state 200 4, although radar-based gesture recognition using the radar system 104 is enabled, the results of the radar-based gesture recognition are not provided to the applications executed on the UE 102 and other subscribers. For the hard-gating state 2006, the radar-based gesture recognition functionality of the radar system 104 is disabled, but other functions of the radar system 104 may remain enabled (for example, when gesture recognition is disabled, the radar system 104 can execute a collision avoidance function during hard-gating).

[0214] Each of the functions 2008 calculates a respective situation score indicating the compatibility between the current situation and each of the states 2002, 2004, and 2006. For example, the function 2008-3 calculates a situation score indicating whether the state machine 2000 should transition to the soft-gating state 2004 based on the sensor data defining the current situation. The function 2008-6 calculates a situation score indicating whether the state machine 2000 should transition to the hard-gating state 2006 based on the sensor data defining the current situation. The state machine 2000 transitions from the non-gating state 2000 to either the soft-gating state 2004 or the hard-gating state 2006 based on which of the two states 2004 or 2006 has a situation score that meets the transition threshold. If each of the functions 2008-3 and 2008-6 calculates a situation score that meets the transition threshold for transitioning the state machine 2000 to the next state, the state machine 2000 may transition to the next state with the highest situation score.

[0215] When in the non-gating state 2002, the state machine 2000 of the radar system 104 receives sensor data from the sensor 108. Functions 2008-3 and 2008-6 receive the sensor data as input and calculate a situation score indicating whether the sensor data meets the requirements to enter the soft gating state 2004 or the hard gating state 2006 respectively. Function 2008-3 corresponds to the "No" result from operation 1802 in FIG. 18. Function 2008-6 corresponds to the "Yes" result from operation 1802 in FIG. 18. If neither of the situation scores from functions 2008-3 and 2008-6 meets their respective transition thresholds, the state machine 2000 remains in the non-gating state 2002.

[0216] In a situation where the sensor data indicates that the user 120 is holding the UE102 and looking at the UE102, the state machine 2000 keeps the radar system 104 in the gesture recognition mode operating in the non-gating state 2002. When the user 120 looks away from the UE102 without lowering the UE102 or maintaining the UE102 substantially stably in order to talk to someone else, function 2008-3 may calculate a situation score that exceeds the respective transition thresholds for transitioning to the soft gating state 2004. Since it may be desired that the UE102 remains continuously ready to resume the detection of radar-based user input, when the user temporarily disengages from the UE102, when the user 120 returns their line of sight to the UE102, the UE102 can quickly return to the non-gating state 2002, and the soft gating thereby enhances the user experience of using the UE102. The state machine 2000 transitions to the soft gating state 2004 and continues to enable radar-based gesture recognition using the radar system 104, but the state machine 2000 prevents the radar system 104 from outputting the results of gesture recognition to the application executed on the UE102.

[0217] In slightly different situations where sensor data indicates that User 120 has lowered UE 102 to the side of their body or otherwise not maintained UE 102 substantially stably in order to talk to someone else and has looked away from UE 102, start again from the non-gating state 2002. Function 2008-6 may calculate a situation score that exceeds the respective transition threshold for transitioning to the hard gating state 2006. The radar system 104 can continue to perform other radar operations for UE 102, but in the hard gating state, the radar-based gesture recognition function of the radar system 104 is disabled. Hard gating thereby promotes power savings and places the radar state 104 in a state where gesture recognition is disabled when gesture recognition is probably not needed.

[0218] After transitioning to the soft gating state 2004, updated sensor data is received from the sensors 108 and the radar system 104. The state machine 2000 calculates the respective situation scores using functions 2008-1 and 2008-4. Function 2008-1 corresponds to the "yes" result from operation 1706 in FIG. 17. Function 2008-4 corresponds to the "yes" result from operation 1802 in FIG. 18. If the situation score of function 2008-1 exceeds the transition threshold for transitioning to the non-gating state 2002, the state machine 2000 transitions to the non-gating state 2002. If the situation score of function 2008-4 exceeds the transition threshold for transitioning to the hard gating state 2006, the state machine 2000 transitions to the hard gating state 2006. If the situation scores of both function 2008-1 and 2008-1 exceed their respective transition thresholds, the state machine 2000 may transition to the state 2002 or 2006 associated with the situation score that is higher than the situation scores of the other functions. Assume that the situation score of function 2008-4 exceeds the situation score of function 2008-1 and the transition threshold associated with the transition from the soft gating state 2004 to the hard gating state 2006.

[0219] After transitioning to the hard-gating state 2006, updated sensor data is received from the sensor 108 and the radar system 104. The state machine 2000 calculates respective situation scores using functions 2008-2 and 2008-5. Function 2008-2 corresponds to the "no" result from operation 1802 in FIG. 18. Function 2008-5 corresponds to the "yes" result from operation 1706 in FIG. 17. If the situation score of function 2008-5 exceeds the transition threshold for transitioning to the non-gating state 2002, the state machine 2000 transitions to the non-gating state 2002. If the situation score of function 2008-2 exceeds the transition threshold for transitioning to the soft-gating state 2004, the state machine 2000 transitions to the soft-gating state 2004. If the situation scores of both functions 2008-2 and 2008-5 exceed their respective transition thresholds, the state machine 2000 may transition to the state 2002 or 2004 associated with the situation score higher than the situation scores of the other functions.

[0220] The state machine 2000 can be machine-learned or driven based on inferences made by a machine learning model. The machine learning model is trained to predict a suitable gating state for the radar system 104 based on sensor data or other inputs that define the current situation. For example, function 2008 can be a machine learning rule or application or machine learning model for calculating a situation score for the current situation. In other words, each of the functions 2008 can be a machine learning model, or an instance of a machine learning model, trained to predict the next radar state, or the situation score that makes the current situation equal to the next radar state.

[0221] Other situation-aware controls As described in great detail above, the radar system 104 relies on the situation of the UE 102 and the recognition of the location and position of the user 120 to gate or not to gate the radar system 104. These same techniques that apply to situation-aware gating may apply to other situation-aware controls that rely on the radar system 104 and radar functionality.

[0222] The UE 102 can also use other sensors among the radar system 104 and the sensor 108 to predict the intention of the user 120 who is about to be involved. The situation of the UE 102 may be with respect to the user 120, indicating the distance from the UE 102 to the user 120, whether the user 120 is moving towards or away from the UE 102, whether the user 120 is reaching out to the UE 102, and indicating the posture or orientation of the user 120 with respect to the UE 102.

[0223] The radar system 104 reconfigures how gesture recognition, proximity detection, and other radar functions are performed in order to adapt each radar function to best suit the current situation. For example, when the user 120 and the UE 102 are in a medium-sized room, the distances and sensitivities programmed into the function 2008 for transitioning between the different states 2002, 2004, and 2006 of the state machine 2000 may not be appropriate in some situations. When the user 120 and the UE 102 are in a smaller room, inside a vehicle, or even in a medium-sized room with a different number of people than originally predicted, the function 2008 for transitioning between the different states 2002, 2004, and 2006 changes or adapts to suit the new situation. In other words, the state machine 2000 may include transition functions such as the function 2008, and they dynamically change the criteria based on changes in the situation. Similarly, the results of the function 2008 may change accordingly. Based on inputs from available signals, sensors, or other data, the state machine 2000 can adjust the parameters to match the function 2008, thereby adjusting the functionality of the UE 102. As described above, the function 2008 may be a machine learning model, or a part of a machine learning model, trained to predict a confidence level or score that a particular state is suitable for the current situation. The following are some non-limiting examples of how the radar system 104 dynamically adapts the radar functions to best suit the current situation.

[0224] The state machine 2000 can put the radar system 104 into a sleep mode (e.g., soft gating, hard gating) based on the inertial data generated by the IMU of the sensor 108. The inertial data indicates that the UE 102 is moving in a way that degrades the accuracy or efficiency of the capabilities of the radar system 104 for performing not only radar-based gesture recognition but also other radar functions. The inertial data from the IMU may include X, Y, and Z axis movement information. The state machine 2000 combines three movements into floating-point values that are input into a function 20008 for the state machine 2000 to transition between states 2002, 2004, and 2006.

[0225] The state machine 2000 can also control the radar system 104 (or put the radar system 104 into a sleep mode) based on other non-IMU sensor data generated by the sensor 108, or other useful information generated by any other data source. For example, the UE 102 may include a calendar application, a clock application, location services, proximity services, communication services, financial services, or any other situational data source. A subscriber application running on the UE 102 may provide situational information to the radar system 104 in the same way that the subscriber application receives indications of gesture inputs recognized by the radar system 104.

[0226] All of these potential information sources can feed the state machine 2000 and the function 2008 to determine whether the radar system 104 should be put into a sleep or gated state. Additionally, the system can know which applications are running, which can further improve the situation awareness of the UE 102 and help the UE 102 make decisions regarding the sleep mode. decisions.

[0227] The situation awareness by the radar system 104 further enables the UE 102 to change the number of available radar states or modes depending on the situation. For example, in an automotive situation, the radar system 104 may be in a non-gating mode or a soft-gating mode. This is because in the automotive mode (when based on the power of the vehicle), maximum responsiveness independent of power consumption is a desirable characteristic of the UE 102. For example, only two states are required. This is because the radar system 104 assumes that the user 120 is only a few feet away (inside the vehicle), so it is not necessary to hard-gate when the user is not present or perhaps save power when not interacting with the UE 102.

[0228] The situation awareness by the radar system 104 relies on the dynamic function 2008 and further on the machine learning model to adjust the trigger parameters between the gating states and other radar modes of the radar system 104, such as the size of the awareness zone or recognition zone, the sensitivity to changes in distance or speed of reaching out a hand or other gestures. Other functionalities of the radar system 104 can be situation-based. Consider a user alone in a vehicle and a user in a subway or a crowded conference room. The radar system 104 can use different sensitivities, feedback, and features to determine radar-based gestures. This is because certain settings like these may be more effective in different situations.

[0229] The situation awareness for controlling the radar system 104 can be useful in other ways. For example, in response to detecting the UE 102 in an enclosed situation such as on a bicycle, the radar system 104 may automatically configure itself for the collision avoidance radar mode and disable gesture recognition.

[0230] The radar system 104 may be more effective when stable. If the sensor data from the sensor 108 indicates that the UE 102 is swaying or vibrating at an amplitude or frequency that is too high, the radar system 104 automatically disables radar-based gesture recognition and other radar functionality. This saves many unnecessary calculation and measurement cycles. This is because if the UE 102 is not stable and is swaying, the radar system 104 will probably not provide useful results.

[0231] The source of the situation information to the UE 102 may be remote to the UE 102. For example, the sensors or input components of a computerized wristwatch paired with the UE 102 can be an additional source of sensor information that supplements the sensor data collected from the sensor 108. In this case, the radar system 104 may gate the radar functionality or control it in other ways based on the sensor data from the communicatively coupled wristwatch. The sensor data may include heart rate information. If the user's heart rate exceeds a certain threshold for indicating exercise or intense physical movement, the radar system 104 may disable the radar-based gesture recognition or other features of the radar system 104. This is because during exercise, the user probably does not gesture with the UE 102.

[0232] The ambient light sensor from the sensor 108 captures sensor data indicating when the situation of the UE 102 is in a dim area. In such a situation, the radar system 104 operates under the assumption that it will make the radar system 104 more tolerant of noisy inputs to its interface, as it is expected that the user 120 will have difficulty interacting with the UE 102.

[0233] A proximity sensor from sensor 108, such as an optical proximity sensor, can switch off or trigger the radar system 104 to enter a state where gesture recognition is disabled when the radar system 104 is blocked. The wireless signal, power connection, network connection, and other connections to the UE 102 can provide additional situational information for controlling the radar system 104. In response to detecting a charging cable, docking station, or wireless charging system that powers the UE 102, the radar system 104 refrains from entering the hard-gating state 2006. This is because during charging, the UE 102 does not need to handle power consumption, and it is more likely that the user 120 desires a faster response speed from the radar system 104. In a related example, when connected to a wireless charging system, the radar system 104 may disable much of its capabilities so as not to interfere with the wireless charger. The radar system 104 may operate in an inactive mode so as not to interfere with communication signals and other signals transmitted or received by the UE 102.

[0234] The radar system 104 is operably coupled to one or more of the sensors 108 and can be triggered in response to an interrupt or information received directly from the sensors 108. For example, a near-field communication unit or NFC sensor can trigger the radar system 104 to enter a non-gating mode when NFC is processing a payment or other authentication gesture.

[0235] The radar system 104 can be switched on or off in cooperation with other input components. For example, the user 120 may provide an input to the touch screen of the UE102, and while the input is being detected on the touch screen, the radar system 104 may disable gesture recognition. In other cases, the radar system 104, while remaining switched on, enhances touch screen functionality by sending information about the recognized gestures to an input decoder that simultaneously processes touch screen data and radar data to infer the user's intent. In this way, the radar system 104 and the touch screen can recognize typing on a soft keyboard or other inputs to the GUI even when the user 120 is wearing gloves while providing a touch input (which may interfere with some presence - sensitive screens).

[0236] The radar system 104 can control the radar function based on other situational information including temperature, humidity, pressure, etc. The radar system 104 may use a certain setting to account for performance variations that may occur due to fluctuations in weather conditions. Using voice or acoustic information, the radar system 104 can control the radar function and activate or deactivate features based on voice commands.

[0237] Example Examples are provided in the following paragraphs.

[0238] Example 1 A method for situation-aware control of radar-based gesture recognition, the method comprising: receiving sensor data from a plurality of sensors of a user device; determining a situation of the user device based on the sensor data; determining whether the situation meets requirements for radar-based gesture recognition; and gating a radar system to prevent the radar system from outputting indications of radar-based gestures to an application subscriber of the user device in response to a determination that the situation does not meet the requirements for radar-based gesture recognition.

[0239] Example 2 The step of gating the radar system includes hard-gating the radar system by triggering the radar system to function in a state where the radar system does not recognize gestures from radar data, according to the method described in Example 1.

[0240] Example 3 The step of hard-gating the radar system further includes, in response to a determination that the situation indicates that the radar system is blocked by an object, according to the method described in Example 2.

[0241] Example 4 The step of gating the radar system includes soft-gating the radar system by triggering the radar system to function in a state where the radar system does not output indications of radar-based gestures, according to the method described in Example 1.

[0242] Example 5 The step of soft-gating the radar system further includes, in response to a determination that the situation indicates that the radar system is not blocked by an object, according to the method described in Example 4.

[0243] Example 6 The step of soft-gating the radar system to prevent the radar system from outputting radar-based gesture indications to an application subscriber of a user device is the method described in Example 4 that does not prohibit the radar system from recognizing radar-based gestures from radar data.

[0244] Example 7 After soft-gating the radar system, the step of determining that the situation indicates that the radar system is blocked by an object, and in response to the determination that the situation indicates that the radar system is blocked by an object, triggering the radar system so that it functions in a state where it does not recognize gestures from the radar data, thereby further including the step of hard-gating the radar system, is the method described in Example 4.

[0245] Example 8 The situation is the first situation, the sensor data is the first sensor data, and the method further includes the step of receiving second sensor data from a plurality of sensors, the step of determining a second situation of the user device based on the second sensor data, the step of determining whether the second situation meets the requirements for radar-based gesture recognition, and in response to the determination that the second situation meets the requirements for radar-based gesture recognition, inputting the radar data acquired by the radar system into a model that determines radar-based gestures from the input radar data, and the step of performing an operation in response to the model determining a radar-based gesture, where the operation is associated with the determined radar-based gesture, is the method described in Example 1.

[0246] Example 9 The step of inputting the radar data acquired by the radar system into a model for radar-based gesture recognition is gating the radar system a step of waiting, and a step of setting the radar system to an active state for radar-based gesture recognition, the method according to Example 8.

[0247] Example 10 The radar system is configured as a proximity sensor for generating at least a part of sensor data, the method according to any one of Examples 1 to 9.

[0248] Example 11 The step of determining whether the situation meets the requirements for radar-based gesture recognition using the radar system includes the step of determining whether the situation indicates that the user is holding the user device or whether the situation indicates that the user is walking, the method according to any one of Examples 1 to 10.

[0249] Example 12 In response to the determination that the user is not holding the user device and the user is walking, or in response to the determination that the radar system is blocked by an object, the method further includes the step of determining that the situation does not meet the requirements for radar-based gesture recognition, the method according to Example 11.

[0250] Example 13 The method according to Example 11 further includes the step of determining based on whether the sensor data indicates a specific movement, whether the user is holding the user device, how the user is holding the user device, or whether the user is walking.

[0251] Example 14 Determining the identity of an application subscriber of radar-based gesture recognition, and selecting a gating sensitivity for determining whether the situation meets the requirements for radar-based gesture recognition based on the identity of the subscriber, wherein the step of determining whether the situation meets the requirements for radar-based gesture recognition using a radar system is based on the gating sensitivity associated with the identity of the subscriber, the method according to any one of Examples 1 to 13.

[0252] Example 15 The gating sensitivity is specific to the type of radar-based gesture preselected by one of the application subscribers, the method according to Example 14.

[0253] Example 16 Further including the step of changing the state of the user equipment in response to the model determining a radar-based gesture, wherein the state of the user equipment includes an access state, a power state, or an information state, the method according to any one of Examples 1 to 15.

[0254] Example 17 The step of determining whether the situation meets the requirements for radar-based gesture recognition includes executing a state machine including a plurality of states linked by respective situation-aware transition functions that receive at least a portion of the sensor data as variable inputs, the method according to any one of Examples 1 to 16.

[0255] Example 18 The state machine enables radar-based gesture recognition using a radar system a non-gating state in which radar-based gesture recognition using a radar system is enabled, a soft-gating state in which the result of radar-based gesture recognition is not provided to the applications executed on the user device and other subscribers even though radar-based gesture recognition is enabled, and a hard-gating state in which radar-based gesture recognition is disabled, the method according to Example 17

[0256] Example 19 The plurality of sensors includes an inertial measurement unit, the method according to any one of Examples 1 to 18

[0257] Example 20 The plurality of sensors excludes a camera sensor, the method according to any one of Examples 1 to 19

[0258] Example 21 The plurality of sensors includes a proximity sensor, an ambient light sensor, a microphone, or a barometer, the method according to any one of Examples 1 to 20

[0259] Example 22 The proximity sensor is an optical proximity sensor, the method according to Example 21

[0260] Example 23 The situation is the first situation, the sensor data is the first sensor data, and the method further includes receiving second sensor data from a plurality of sensors, determining a second situation of the user device based on the second sensor data, determining whether the second situation meets the requirements for radar-based gesture recognition, and outputting an indication of a radar-based gesture to an application subscriber of the user device in response to the determination that the second situation meets the requirements for radar-based gesture recognition, the method according to Example 1

[0261] Example 24 In further response to a determination that the second situation meets the requirements for radar-based gesture recognition, a method according to Example 23 further includes the steps of inputting radar data acquired by a radar system into a model that determines radar-based gestures from the input radar data, and outputting an indication of the radar-based gesture from the model to an application subscriber.

[0262] Example 25 The method according to any one of Examples 1, 23, and 24, wherein the plurality of sensors includes an inertial measurement unit.

[0263] Example 26 The method according to any one of Examples 1 and 23 to 25, wherein the plurality of sensors excludes a camera sensor.

[0264] Example 27 The method according to any one of Examples 1 and 23 to 28, wherein the plurality of sensors includes a proximity sensor, an ambient light sensor, a microphone, or a barometer.

[0265] Example 28 The step of determining the situation of the user equipment includes determining whether the user is holding the user equipment, in what orientation the user is holding the user equipment, or whether the user is walking. The method according to any one of Examples 1 and 23 to 28, wherein the step of determining whether the situation meets the requirements for radar-based gesture recognition includes determining whether the situation meets the requirements for radar-based gesture recognition based on whether the user is holding the user equipment, in what orientation the user is holding the user equipment, or whether the user is walking.

[0266] Example 29 The step of determining the situation of the user equipment includes the step of determining whether the radar system is blocked by an object, and the step of determining whether the situation meets the requirements for radar-based gesture recognition includes the step of determining whether the situation meets the requirements for radar-based gesture recognition based on whether the radar system is blocked by an object, according to any one of Examples 1 and 23 to 28.

[0267] Example 30 The step of determining the situation of the user equipment includes the step of determining whether the ambient light determined based on the sensor data indicates low light conditions, and the step of determining whether the situation meets the requirements for radar-based gesture recognition includes the step of determining whether the situation meets the requirements for radar-based gesture recognition based on whether the ambient light determined based on the sensor data indicates low light conditions, according to any one of Examples 1 and 23 to 29.

[0268] Example 31 The step of determining the situation of the user equipment includes the step of determining whether a wired or wireless charging system is supplying power to the user equipment, and the step of determining whether the situation meets the requirements for radar-based gesture recognition includes the step of determining whether the situation meets the requirements for radar-based gesture recognition based on whether a wired or wireless charging system is supplying power to the user equipment, according to any one of Examples 1 and 23 to 30.

[0269] Example 32 The step of determining the situation of the user equipment includes the step of determining the wireless signal received by the user equipment, and the method further includes the step of controlling the radar system to prevent interference with the wireless signal received by the user equipment, according to any one of Examples 1 and 23 to 31.

[0270] Example 33 The step of determining the status of the user equipment includes the step of determining the variation of weather conditions, and the method further includes the step of controlling the radar system to consider the weather conditions while recognizing radar-based gestures, the method according to any one of Examples 1 and 23 to 32.

[0271] Example 34 The step of determining the status of the user equipment includes the step of determining whether an input is detected by the presence-sensing display, and the step of determining whether the situation meets the requirements for radar-based gesture recognition includes the step of determining whether the situation meets the requirements for radar-based gesture recognition based on whether an input is detected by the presence-sensing display, the method according to any one of Examples 1 and 23 to 33.

[0272] Example 35 A system including means for performing the method according to any one of Examples 1 to 34.

[0273] Example 36 An apparatus configured to perform the method according to any one of Examples 1 to 34.

[0274] Conclusion Although techniques for radar-based gesture recognition using situation-aware gating and other situation-aware controls and implementation examples of apparatuses enabling such radar-based gesture recognition have been described in terms specific to the features and / or methods, it should be understood that the subject matter of the claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as exemplary implementation examples enabling radar-based gesture recognition using situation-aware gating and other situation-aware controls.

Claims

1. A method for situation-aware control of radar-based gesture recognition, the method comprising: receiving sensor data from a plurality of sensors of a user's user equipment; determining the situation of the user equipment based on the sensor data; determining whether the situation meets the requirements for radar-based gesture recognition; gating the radar system to prevent the radar system from outputting data indicating a radar-based gesture to the user equipment in response to a determination that the situation does not meet the requirements for radar-based gesture recognition, the step of gating the radar system including at least one of soft gating or hard gating based on the situation of the user equipment.

2. The method according to claim 1, wherein the step of gating the radar system includes the step of hard gating the radar system by triggering the radar system to function in a state where the radar system does not recognize gestures from radar data.

3. The method according to claim 2, wherein the step of hard gating the radar system further responds to a determination that the situation indicates that the radar system is blocked by an object.

4. The method according to claim 1, wherein the step of gating the radar system includes the step of soft gating the radar system by triggering the radar system to function in a state where the radar system does not output data indicating the radar-based gesture.

5. The step of soft-gating the radar system further comprises the method according to claim 4, in response to a determination by the situation that the radar system is not blocked by an object.

6. The step of soft-gating the radar system to prevent the radar system from outputting the data indicating the radar-based gesture to the user equipment does not prohibit the radar system from recognizing the radar-based gesture from the radar data, according to the method of claim 4.

7. After soft-gating the radar system, the step of determining that the situation indicates that the radar system is blocked by an object; In response to the determination that the situation indicates that the radar system is blocked by the object, hard-gating the radar system by triggering the radar system to function in a state where the radar system does not recognize gestures from radar data, further comprising the method of claim 4.

8. The situation is a first situation, the sensor data is first sensor data, and the method further comprises: Receiving second sensor data from the plurality of sensors; Determining a second situation of the user equipment based on the second sensor data; Determining whether the second situation meets the requirements for radar-based gesture recognition; In response to the determination that the second situation meets the requirements for radar-based gesture recognition, inputting the radar data acquired by the radar system into a model for determining a radar-based gesture from the input radar data; The method according to claim 1, further comprising the step of performing an operation in response to the model determining a radar-based gesture, wherein the operation is associated with the determined radar-based gesture. **Claim 9** The method according to claim 8, wherein the step of inputting the radar data obtained by the radar system into the model for radar-based gesture recognition includes the step of refraining from gating the radar system and the step of setting the radar system to an active state for radar-based gesture recognition. **Claim 10** The method according to any one of claims 1 to 9, wherein the radar system is configured as a proximity sensor for generating at least a part of the sensor data. **Claim 11** The method according to claim 1, further comprising the step of determining that the situation does not meet the requirements for radar-based gesture recognition in response to a determination that the user is not holding the user device, or in response to a determination that the user is walking, or in response to a determination that the radar system is blocked by an object. **Claim 12** The step of determining the identity of the user for the radar-based gesture recognition, The method according to any one of claims 1 to 11, further comprising the step of selecting a gating sensitivity for determining whether the situation meets the requirements for radar-based gesture recognition based on the identity of the user, wherein the step of determining whether the situation meets the requirements for radar-based gesture recognition using the radar system is based on the gating sensitivity associated with the identity of the user. **Claim 13** The method according to claim 12, wherein the gating sensitivity is specific to the type of radar-based gesture preselected by the user.

14. A system comprising means for performing the method according to any one of claims 1 to 13.

15. An apparatus configured to perform the method according to any one of claims 1 to 13.

16. A program for causing a computer to execute the method according to any one of claims 1 to 13.

Images