SMART PHONE-BASED RADAR SYSTEM FOR DETERMINING USER INTEnt IN LOW POWER

Abstract

The application discloses a smartphone-based radar system that determines user intent in a lower power mode. This document describes techniques and systems for implementing a smartphone-based radar system for determining user intent in a lower power mode. These techniques and systems use a radar field (110) to enable a smartphone to accurately determine the presence or absence of a user (112) and further determine the user's intent to interact with the smartphone (716, 718, 720, 722). Using these techniques, a smartphone can describe a user's non-verbal communication cues to determine and maintain perception of a user in the environment of the smartphone, and only respond to direct interaction once the user has indicated an intention to interact, which saves battery power. The smartphone may determine the intent of the user by identifying various cues from the user, such as changes in position, changes in gesture, or the like with respect to the smartphone, or by a display action, such as a gesture.

Description

Description of the case

[0001] This application is a divisional application of Chinese invention patent application No. 201980017941.7, with a filing date of September 27, 2019. Technical Field

[0002] The present disclosure relates to a smartphone-based radar system that determines user intent in a lower power mode. Background Art

[0003] Smartphones have become an almost necessary accessory for both business and personal life. Applications on smartphones and other electronic devices provide an ever-increasing variety of productivity, entertainment, and communication features that users interact with on a regular basis. These devices are our almost constant companions at work, play, and home. We communicate with them via voice and touch, and treat them like virtual assistants as we use them to schedule meetings and events, consume digital media, and share presentations and documents. However, in everyday life, communication is more than the words we use. Much of how we communicate and express our intentions to communicate is based on the way we perceive and manage interpersonal space. People often use changes in spatial relationships (e.g., interpersonal distance or orientation) as an implicit form of communication. For example, we maintain a certain distance from others based on intimacy, we face people when dealing with them, we move closer to objects of interest, and we stand or sit relative to others based on the task at hand.

[0004] With the help of machine learning techniques, smartphone applications and smartphones themselves have become more familiar with our routines and preferences and are able to provide restaurant recommendations, suggest movies, and otherwise communicate with us independently. However, for all the computing power and artificial intelligence of smartphones, smartphones are still reactive communicators. That is, no matter how "smart" smartphones are and no matter how much we talk to them like they are people, smartphones still rely on being activated, and users usually have to first occupy the smartphone and make it aware of the user's intention to use the smartphone before actually interacting with the application. Therefore, it may be inconvenient and frustrating to utilize the powerful and interesting features of our applications, and we may not realize the full potential of our electronic devices and applications. Summary of the invention

[0005] This document describes techniques and systems that implement a smartphone-based radar system for determining user intent in a lower power mode. These techniques and systems use radar fields to enable a smartphone to accurately determine the presence or absence of a user (112) and further determine the user's intent to interact with the smartphone. Using these techniques, the smartphone can interpret the user's non-verbal communication cues to determine and maintain awareness of the user in the smartphone's environment, and only respond to direct interactions once the user has indicated an intent to interact, which conserves battery power. The smartphone can determine the user's intent by identifying various cues from the user such as changes in position relative to the smartphone, changes in posture, etc., or through displayed actions such as gestures.

[0006] The following aspects include a smart phone that includes a radar system, one or more computer processors, and one or more computer-readable media. The radar system is implemented at least in part in hardware and provides a radar field. The radar system also senses reflections from objects in the radar field and analyzes reflections from objects in the radar field. The radar system provides radar data based on the analysis of the reflections. The one or more computer-readable media include stored instructions that can be executed by one or more computer processors to implement a persistent radar-based interaction manager. The persistent radar-based interaction manager maintains the radar system in an idle mode that requires no more than about 30 milliwatts (mW) of power. The persistent radar-based interaction manager also determines the presence of an object within a perception zone of the smart phone based on a first subset of the radar data. In response to determining the presence of the object within the perception zone, the persistent radar-based interaction manager causes the radar system to enter an attention mode that requires no more than about 60 mW of power. In response to entering the attention mode, and based on a second subset of the radar data, the persistent radar-based interaction manager determines an intent level of the object, and based on the intent level, determines whether the object intends to interact with the smart phone. In response to determining that the intent level indicates that the subject does not intend to interact with the smartphone, the persistent radar-based interaction manager causes the radar system to exit the attention mode and enter an idle mode. In response to determining that the intent level indicates that the user intends to interact with the smartphone, the persistent radar-based interaction manager causes the radar system to exit the attention mode and enter an interaction mode that requires no more than approximately 90 mW of power.

[0007] The following aspects also include a method implemented in an electronic device including a radar system. The method includes providing a radar field by the radar system and sensing reflections from objects in the radar field by the radar system. The method also includes analyzing the reflections from the objects in the radar field and providing radar data based on the analysis of the reflections. The method additionally includes maintaining the radar system in an idle mode that requires no more than about 30 milliwatts (mW) of power. The method also includes: determining the presence of an object within a perception zone of the electronic device based on a first subset of the radar data; and in response to determining the presence of the object within the perception zone, causing the radar system to enter an attention mode that requires no more than about 60 mW of power. The method additionally includes: determining an intent level of the object in response to entering the attention mode and based on a second subset of the radar data; and determining whether the object intends to interact with the electronic device based on the intent level. The method further includes, in response to determining that the intent level indicates that the object does not intend to interact with the electronic device, causing the radar system to exit the attention mode and enter the idle mode. The method also includes, in response to determining that the intent level indicates that the user intends to interact with the electronic device, causing the radar system to exit the attention mode and enter the interaction mode that requires no more than about 90 mW of power.

[0008] The following aspects also include a method implemented in an electronic device including a radar system. The method includes providing a radar field by the radar system and sensing, by the radar system, reflections from objects in the radar field. The method also includes analyzing the reflections from the objects in the radar field and providing radar data based on the analysis of the reflections. The method additionally includes maintaining the radar system in an idle mode that requires no more than approximately 30 milliwatts (mW) of power. The method also includes: determining, based on a first subset of the radar data, the presence of an object within a perception zone of the electronic device; and in response to determining the presence of the object within the perception zone, causing the radar system to exit a lower power mode and enter an interactive mode that requires no more than approximately 90 mW of power.

[0009] The following aspects also include a system, the system includes an electronic device, the electronic device includes or is associated with a first device. The first device is a device for providing a radar field, sensing reflections from objects in the radar field, analyzing reflections from objects in the radar field, and providing radar data based on the analysis of the reflections. The system also includes a second device. The second device is a device for maintaining the first device in an idle mode that requires no more than about 30 milliwatts (mW) of power. The second device is also a device for determining the presence of an object within a perception zone of the electronic device based on a first subset of radar data and causing the first device to enter an attention mode that requires no more than about 60 mW of power in response to determining the presence of the object within the perception zone. The second device is also a device for determining the intent level of the object in response to entering the attention mode and based on the second subset of radar data. The second device is also a device for determining whether the object intends to interact with the electronic device based on the intent level and causing the first device to exit the attention mode and enter the idle mode in response to determining that the intent level indicates that the object does not intend to interact with the electronic device. The second device is also a device for causing the first device to exit the attention mode and enter the interaction mode that requires no more than about 90 mW of power in response to determining that the intent level indicates that the user intends to interact with the electronic device.

[0010] Other aspects include a method that can be implemented in an electronic device including a radar system. The method includes providing a radar field by the radar system and sensing reflections from objects in the radar field by the radar system. The method also includes analyzing the reflections from the objects in the radar field and providing radar data based on the analysis of the reflections. The method also includes: maintaining the radar system in a first power mode; and determining, based on a first subset of the radar data and at a first degree of certainty, a first intent of the object to interact with the electronic device. The method additionally includes, in response to determining the first intent of the object to interact with the electronic device, causing the radar system to enter a second power mode. In response to entering the second power mode and based on a second subset of the radar data, the method also includes determining, at a second degree of certainty, a second intent of the object to interact with the electronic device. The method further includes, in response to determining the second intent of the object to interact with the electronic device, causing the radar system to enter a third power mode. In response to entering the third power mode and based on a third subset of the radar data, the method also includes determining, at a third degree of certainty, a third intent of the object to interact with the electronic device.

[0011] This Summary is provided to introduce simplified concepts related to a smartphone-based radar system for determining user intent in a low power mode, which will be further described in the following Detailed Description and Figures. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Details of one or more aspects of a smartphone-based radar system for determining user intent in a lower power mode are described in this document with reference to the following figures. In all figures, the same numbers are used to reference similar features and components:

[0013] Figure 1 An example environment is illustrated in which techniques for enabling a smartphone-based radar system for determining user intent in a lowered power mode may be implemented.

[0014] Figure 2 Diagram Figure 1 Example embodiments of a smartphone that includes a radar system and can implement a smartphone-based radar system for determining user intent in a lower power mode.

[0015] Figure 3 Diagram Figure 2 Example implementation of a radar system.

[0016] Figure 4 Diagram Figure 3 Example arrangement of receiving antenna elements of a radar system.

[0017] Figure 5 Diagram Figure 2 Additional details of an example implementation of a radar system.

[0018] Figure 6 The diagram can be Figure 2 Example scenario for radar system implementation.

[0019] Figure 7-11 An example method of enabling a smartphone-based radar system for determining user intent in a lowered power mode is described.

[0020] Figure 12-18 The diagram can be realized Figures 7 to 11 Additional details of the method are provided for example implementations of electronic devices.

[0021] Fig.19 Illustrate various components of an example computing system that may be implemented as a reference Figures 1 to 18 Any type of client, server, and / or electronic device described may implement, or in which techniques for enabling a smartphone-based radar system for determining user intent in a lower power mode may be implemented. DETAILED DESCRIPTION Overview

[0022] This document describes the technology and system of implementing a radar system based on a smartphone for determining the user's intention in a lower power mode. As noted, it may be difficult for an electronic device to determine the user's intention to interact with the device, which can limit its responsiveness. Although the device can receive direct voice or touch input and act on these inputs, humans usually use a mixture of verbal communication and non-verbal communication to communicate intentions, and current smartphones understand the verbal part at most. Therefore, due to limitations in recognizing non-verbal communication, users may not be able to realize the full potential of their smartphone features. The described technology and system use a radar system to determine the user's intention to interact or not interact with an electronic device. The user's intention can be discerned by determining the user's position, the user's orientation relative to the device, the user's posture, or other factors. These technologies increase the degree of refinement, increase the compliance with facts, or both increase the degree of refinement and increase the compliance with facts. Because the electronic device can determine the user's intention while saving battery power, the interaction can be more convenient and less frustrating because the electronic device can anticipate the user's intention and enter the appropriate mode or state to interact and receive input.

[0023] Additionally, these techniques and systems can enable a smartphone to receive gesture input via a natural three-dimensional (3D) gesture interaction language that can be used to extend the interaction space of the smartphone beyond the screen, which provides accessibility in more situations. 3D gestures (e.g., gestures that include one or more movements in any direction within a 3D space illuminated by a radar field), such as sweeping or reaching, can be used to interact with a smartphone without touching it. Using 3D gestures, a user can browse recipes without touching the screen when their hands are dirty, take a selfie from the right distance and angle without stretching or awkwardly pressing buttons, or maintain focus on the current task by implementing quick gestures that can be used to reduce interruptions by suppressing alarms, phone calls, or notifications. Thus, the described techniques and systems can improve the quality and effectiveness of the user experience using convenient and natural 3D gestures, thereby improving the efficiency, workflow, and enjoyment of the user.

[0024] Consider a smartphone or other electronic device including a radar system and an interaction manager that can be used to provide a radar field near the electronic device and determine when people or other objects enter the radar field. For example, a user can place a smartphone on a workbench or table while the user performs other tasks. The interaction manager and the radar system can operate in a persistent lower power mode, which allows the radar system to detect the presence of a user within a specified distance. When no user is present, the interaction manager can also put the smartphone in a lower power state. The interaction manager can also determine whether the user intends to interact with the smartphone. If the interaction manager determines that the user does not intend to interact with the smartphone, the interaction manager can maintain the interaction manager and the radar system in a persistent lower power mode and maintain the smartphone in a lower power state until the interaction manager determines that the user intends to interact with the smartphone. In this example, the interaction manager can determine whether the user intends to interact by observing the user's body posture and position, such as whether the user is looking at the smartphone, turning to the smartphone, or reaching for the smartphone. Example power saving modes can include turning off display elements, hiding message previews, hiding notifications, etc. When the interaction manager determines that the user intends to interact with the smartphone, the radar system and smartphone can resume full operation to enable the user to access all features and functions of the smartphone.

[0025] For example, when the radar system is in a lower power mode, the smartphone can also be in a lower power state by turning off or otherwise reducing the power consumption of various functions such as the display, touch screen, microphone, voice assistant, etc. When the radar system detects a user in the area and determines that the user intends to interact with the smartphone, the interaction manager can automatically restore the smartphone to a fully operational state. The smartphone can be presented in a ready-to-use state (e.g., the user can be automatically authenticated via a radar-based authentication system or by another method), or the smartphone can be presented as ready to be authenticated by the user (e.g., ready to receive a password, gesture, fingerprint, or other authentication). In this way, the interaction manager can save power without reducing the functionality of the user. In addition, the user can enjoy an improved experience with the smartphone because the interaction manager anticipates the user's intention and can present the smartphone in a customizable state that allows the user to start interacting with the smartphone without having to pick up or touch the smartphone and then wait for the smartphone to be ready.

[0026] Some conventional electronic devices can use cameras or proximity sensors (e.g., capacitive sensors) to determine the location of the user and adjust various functions of the electronic device based on the proximity of the user. For example, unless the user is within a predetermined distance, the electronic device can provide additional privacy or aesthetic value by turning off the display. However, conventional electronic devices are generally unable to distinguish between the mere presence of a user (or a non-user who happens to be near the electronic device) and a user who intends to interact with the electronic device. Therefore, conventional electronic devices use more power and potentially provide less privacy or aesthetic value because the electronic device may be turned on when a non-user is nearby or when a user is nearby but not interested in the interaction. For example, a user may place an electronic device on a table while talking to a guest. A previously set reminder to make an appointment with a doctor can be displayed in a manner that allows the guest to see the reminder. Even a quick-response user may experience the frustration or embarrassment of having a private reminder displayed at an inappropriate time. Continuous annoying or inconvenient interaction with a smart phone can reduce the efficiency and quality of the user experience, or even reduce the likelihood that the user will use some features.

[0027] In contrast, the described systems and techniques can improve efficiency and usability in several areas while preserving battery life. For example, in the above example, the smartphone can determine (e.g., by determining that the user is away from the smartphone) that the user is not paying attention to the smartphone and hide the reminder. In this way, the described techniques and systems allow efficient and natural interaction with smartphones or other electronic devices. Users can enjoy the advantages and convenience provided by smartphones, while the interaction manager and radar system provide additional flexibility and enhanced functionality without excessive power consumption. This can improve efficiency and reduce user frustration by reducing the need for users to adjust reminders, alarms, and notifications for different social settings while maintaining privacy and reducing interruptions, which can improve the quality of user experience. In addition, the power consumption of the radar system and the electronic device itself can be much lower than some conventional technologies that can use an always-on camera (or other sensor or combination of sensors) to control some display features. These are just a few examples of how the described techniques and devices can be used to implement a smartphone-based radar system for determining user intent in a lower power mode. Other examples and implementations thereof are described throughout this document. This document now turns to an example environment, after which example systems, devices, methods, and components are described.

[0028] In some examples described herein, power, angle or another measurable amount is defined as an approximate specified value. In these cases, the power, angle or other measurable amount is located within a maximum of 5% near the specified value. In addition or as an alternative, in some examples, when power, angle or another measurable amount is defined as an approximate specified value, the mean value of power, angle or another measurable amount is located within a maximum of 5% near the specified value. In some of these examples, the extreme value of power, angle or other measurable amount can differ from the specified value by more than 5%. Operating Environment

[0029] Figure 1 An example environment 100 is illustrated in which techniques for enabling a smartphone-based radar system for determining user intent in a lower power mode may be implemented. The example environment 100 includes a smartphone 102 that includes a radar system 104 or a persistent radar-based interaction manager 106 (interaction manager 106) and optionally includes or is associated with one or more non-radar sensors 108 (non-radar sensors 108). The non-radar sensor 108 may be any of a variety of devices, such as an audio sensor (e.g., a microphone), a touch input sensor (e.g., a touch screen), or an image capture device (e.g., a camera or camcorder).

[0030] In the example environment 100, the radar system 104 provides a radar field 110 by transmitting one or more radar signals or waveforms, as described below with reference to Figures 3 to 6 As described. The radar field 110 is a volume of space from which the radar system 104 can detect reflections of radar signals and waveforms (e.g., radar signals and waveforms reflected from objects in the volume of space). The radar system 104 also enables the smart phone 102 or another electronic device to sense and analyze reflections from objects in the radar field 110. Some embodiments of the radar system 104 are particularly advantageous when applied in the context of a smart phone (such as the smart phone 102): for which there are problems such as low power requirements, processing efficiency requirements, limitations on the spacing and layout of antenna units, and other problems, and some embodiments of the radar system 104 are even more advantageous in the specific context of the following smart phone: the smart phone requires radar detection of fine gestures. Although these embodiments are particularly advantageous in the context of smart phones that require fine radar detection of gestures, it should be understood that the applicability of the features and advantages of the present invention is not necessarily limited thereto, and other embodiments involving other types of electronic devices may also be within the scope of the present teachings.

[0031] The object may be any of a variety of objects that radar system 104 may sense and analyze radar reflections from, such as wood, plastic, metal, fabric, a human body or a human body part (e.g., a foot, hand, or finger of a user of smartphone 102). Figure 1 , the object is a user of smartphone 102 (user 112). Based on the analysis of the reflections, radar system 104 may provide radar data including various types of information associated with radar field 110 and the reflections from user 112, as shown in FIG. Figures 3 to 6 As described (eg, radar system 104 may communicate radar data to other entities, such as interaction manager 106 ).

[0032] It should be noted that radar data may be provided continuously or periodically over time based on reflections sensed and analyzed from objects in radar field 110. The location of an object may change over time (e.g., an object may move within radar field 110), and therefore, radar data may change over time corresponding to the changed location, reflections, and analysis. Because radar data may change over time, radar system 104 may provide radar data including one or more subsets of radar data corresponding to different time periods. For example, radar system 104 may provide a first subset of radar data corresponding to a first time period, a second subset of radar data corresponding to a second time period, and so on.

[0033] The interaction manager 106 can be used to interact with or control various components (e.g., modules, managers, systems, or interfaces) of the smartphone 102. For example, the interaction manager 106 can be used to maintain the radar system 104 in an idle mode. The idle mode is a persistent, lower-power radar mode that allows the radar system 104 to scan the environment outside the smartphone 102 and determine the presence of the user 112. The term "persistent" with respect to the interaction manager 106 and the idle mode of the radar system 104 means that no user interaction is required to maintain the radar system 104 in the idle mode or to activate the interaction manager 106. In some embodiments, the "persistent" state can be suspended or turned off (e.g., by the user 112). In other embodiments, the "persistent" state can be scheduled or otherwise managed according to one or more parameters of the smartphone 102 (or other electronic device). For example, even if the smartphone 102 is operated at night and during the day, the user 112 can schedule the "persistent" state so that it only operates during the day. In another example, user 112 may coordinate the “persistent” state with a power saving mode of smartphone 102 .

[0034] In idle mode, the interaction manager 106 can determine the presence of the user 112 without verbal, touch, or other input from the user. For example, when in idle mode, the interaction manager 106 can use one or more subsets of radar data provided by the radar system 104 (as described herein) to determine the presence of the user 112 or other objects that may be within the perception zone 114 of the smart phone 102. In this way, the interaction manager 106 can provide seamless power management without requiring explicit interaction input.

[0035] In some embodiments, the idle mode requires no more than about 30 milliwatts (mW) of power. In other embodiments, the idle mode may require a different amount of power, such as about 2 mW or about 8 mW. In addition, when the interaction manager 106 maintains the radar system 104 in the idle mode, the interaction manager 106 may also maintain the smartphone 102 in a lower power state (e.g., a sleep mode or other power saving mode). In this way, by determining whether the user 112 (or another person) is near the smartphone 102, the interaction manager can help save battery power by reducing power consumption when no user is near the smartphone 102.

[0036] Perception zone 114 is an area around radar system 104 within which interaction manager 106 can accurately determine the presence of user 112. Perception zone 114 can take any of a variety of shapes and forms. For example, perception zone 114 can be approximately contiguous with radar field 110 (e.g., as shown in FIG. 1 ). Figure 3 and Figure 4 In other cases, perception zone 114 may take on a shape such as a radius extending from radar system 104, a volume (e.g., a sphere, hemisphere, partial sphere, beam, or cone) around radar system 104, or a non-uniform shape (e.g., to accommodate interference from obstacles in the perception zone). The perception zone may extend any of a variety of distances from radar system 104, such as three, seven, ten, or fourteen feet (or one, two, three, or four meters), and may coincide with the range of radar field 110. In other cases, such as Figure 1 , the perception zone 114 may be less than the maximum range of the radar field 110. The perception zone 114 may be predefined, user selectable, or determined via other methods (e.g., based on power requirements, remaining battery life, or other factors). In some embodiments, when the interaction manager 106 determines the presence of the user 112 (or another object) within the perception zone 114, the interaction manager 106 may cause the radar system 104 to exit the idle mode and enter the interactive mode, as described in detail below.

[0037] Alternatively, or in other embodiments, interaction manager 106 may cause radar system 104 to enter an attention mode when interaction manager 106 determines the presence of user 112 (or another object) in perception zone 114. Attention mode is a radar mode that allows radar system 104 to provide additional information about objects within perception zone 114. For example, while in attention mode, radar system 104 may provide other radar data (including one or more other subsets of radar data as described herein) that may be used to determine a level of intent of user 112.

[0038] In some embodiments, the power required for the attention mode does not exceed about 60 mW. In other embodiments, the attention mode may require a different amount of power, such as between about 8 mW and about 55 mW or between about 2 mW and about 20 mW. When the interaction manager 106 maintains the radar system 104 in the attention mode, the interaction manager 106 may also maintain the smartphone 102 in a lower power state that may be used with the idle mode, or the interaction manager 106 may cause the smartphone to exit the lower power state and enter another state (e.g., awake mode, active mode, etc.).

[0039] The interaction manager 106 (or another module or entity) may use the level of intent to determine whether the user 112 intends to communicate or interact with the smartphone 102. The level of intent may be determined from various information about the user 112 (within the perception zone 114) that may be determined based on the other radar data. The interaction manager 106 may determine the level of intent of the user 112 without verbal, touch, or other input from the user. For example, the interaction manager 106 may determine the user's intent to view, communicate, or otherwise interact with the smartphone 102 by using the other radar data or one or more other subsets of the other radar data to determine the body position or posture of the user 112 relative to the smartphone 102.

[0040] The determination of the body position and posture of the user 112 may include determining one or more of a variety of different non-verbal body language cues, body position, or body postures. The cues, positions, and postures may include the absolute position or distance of the user 112 relative to the smartphone 102, changes in the position or distance of the user 112 relative to the smartphone 102 (e.g., whether the user 112 is moving closer to or away from the smartphone 102), the speed of the user 112 when moving near the smartphone 102 (e.g., whether the user 112 is pausing when near the smartphone 102), whether the user 112 is moving toward or away from the smartphone 102, whether the user 112 is leaning toward, waving, touching, or pointing at the smartphone 102, and the like.

[0041] Consider an example in which the user 112 moves toward the smartphone 102. When the user 112 does not slow down and continues to pass by the smartphone 102, the interaction manager 106 determines that the user 112 does not intend to interact with the smartphone 102. Similarly, when the user 112 slows down or even stops near the smartphone 102 but moves away from the smartphone 102, the interaction manager 106 also determines that the user 112 does not intend to interact with the smartphone 102. In contrast, when the user 112 stops near the smartphone 102 and turns toward (or is reaching for) the smartphone 102, the interaction manager 106 determines that the user 112 intends to interact with the smartphone 102. In this way, using the different body positioning and postures of the user 112, the interaction manager 106 can accurately determine whether the user 112 intends to interact with the smartphone 102.

[0042] When the interaction manager 106 determines that the user 112's level of intent indicates that the user 112 does not intend to interact with the smartphone 102, the interaction manager 106 may cause the radar system 104 to exit the attention mode and enter (or re-enter) the idle mode. When the radar system 104 enters the idle mode, the interaction manager 106 may also cause the smartphone 102 to enter a lower power state (e.g., if the smartphone 102 was in a higher power state when the radar system was in the attention mode). Thus, the interaction manager 106 may use the user 112's level of intent to help manage the power consumption of the radar system 104 and, optionally, the smartphone 102.

[0043] Conversely, when interaction manager 106 determines that the level of intent of user 112 indicates that user 112 intends to interact with smartphone 102, interaction manager 106 may cause radar system 104 to exit attention mode and enter interaction mode. Interaction mode is a radar mode that allows radar system 104 to provide additional information about objects within perception zone 114. For example, while in interaction mode, radar system 104 may provide additional radar data (including one or more additional subsets of radar data as described herein) that may be used to enable radar system 104 to determine 3D gestures made by user 112 and process the 3D gestures in a manner that enables user 112 to interact with smartphone 102 via the 3D gestures. Figures 10 to 16 Additional description and examples of how to use 3D gestures to interact with smartphone 102 are described.

[0044] The interaction mode may also provide additional information to the user 112. Continuing with the above example in which the user 112 moves toward the smart phone 102, assume that the display of the smart phone 102 is dimmed (e.g., because the smart phone 102 is in a lower power state as described above). Additionally, assume that the interaction manager 106 determines, based on the intent level, that the user 112 intends to interact with the smart phone 102 (e.g., because the user 112 is stopped near the smart phone 102). In response, the interaction manager 106 may cause the smart phone 102 to change the state of the display (e.g., increase the display brightness of the current screen or change the screen to display an authentication screen or smart assistant). In this way, the interaction manager 106 may provide the user 112 with timely access to the smart phone 102 while conserving battery power and providing increased privacy.

[0045] In some embodiments, the interactive mode requires no more than about 90 mW of power. In other embodiments, the interactive mode may require a different amount of power, such as about 20 mW or about 55 mW. In addition, when the interaction manager 106 maintains the radar system 104 in the interactive mode, the interaction manager 106 may also maintain the smartphone 102 in an appropriate power mode (e.g., a full power mode, an awake mode or an active mode as described with reference to the attention mode, a sleep mode as described with reference to the idle mode, or another power mode) while the user 112 is interacting with the smartphone 102. In this way, by determining whether the user 112 (or another person) intends to interact with the smartphone 102, the interaction manager can help save battery power by inducing an appropriate power mode for the radar system 104 and, optionally, for the smartphone 102, which is appropriate for the interaction level of the user 112.

[0046] Various techniques may be used to adjust the power consumed by radar system 104 in idle mode, attention mode, and interactive mode. For example, radar system 104 may reduce power consumption by collecting radar data at different duty cycles (e.g., lower frequencies may use less power), shutting down various components when components are inactive, or adjusting power amplification levels. Figure 3 Additional details regarding power management of radar system 104 (and smartphone 102 ) are described.

[0047] The interaction manager 106 may also control or communicate with the non-radar sensors 108. For example, when the radar system 104 is in an idle state, the interaction manager 106 may maintain the non-radar sensors 108 in a non-operating state. The interaction manager 106 may also cause the non-radar sensors 108 to enter an operating state when the presence of the user 112 (or another object) within the perception zone 114 is determined and the radar system 104 enters an attention mode. In another example, the interaction manager 106 may maintain the non-radar sensors 108 in a non-operating state when the radar system 104 is in an idle mode, and cause the non-radar sensors 108 to enter an operating state when the interaction manager 106 causes the radar system 104 to enter an interaction mode. In this way, by managing the use of battery power and being able to observe or record the operation of sensors and components of the user 112, the interaction manager 106 may save battery life, increase privacy and security, and may improve the user's experience with the smartphone 102.

[0048] In some implementations, the smart phone 102 may also include, or be associated with, one or more other modules, interfaces, or systems. Figure 1 As shown in FIG. 1 , smartphone 102 includes 3D gesture module 116 and smartphone power management interface 118. 3D gesture module 116 may store information related to determining 3D gestures based on radar data and information related to actions corresponding to the 3D gestures.

[0049] The 3D gesture may be any of a variety of gestures, including a roll gesture made by moving a hand over the smartphone 102 along a horizontal dimension (e.g., from the left side of the smartphone 102 to the right side of the smartphone 102), a swipe gesture made by rotating the user's arm around the elbow, a push gesture made by moving the user's hand over the smartphone 102 along a vertical dimension (e.g., from the bottom side of the smartphone 102 to the top side of the smartphone 102). Other types of 3D gestures or motions may also be made, such as a reach gesture made by moving the user's hand toward the smartphone 102, a doorknob rotation gesture made by curling the fingers of the user's hand to hold an imaginary doorknob and rotating it in a clockwise or counterclockwise manner to simulate the action of turning an imaginary doorknob, and a spindle twist gesture made by rubbing the thumb and at least one other finger together. Each of these example gesture types may be detected by the radar system 104.

[0050] Based on the radar data, the interaction manager 106 can detect a 3D gesture by the user 112 and determine (e.g., using the 3D gesture module 116) that the gesture corresponds to a specific action by or by the smart phone 102. The specific action can be any of a variety of actions, such as activating one or more sensors (e.g., non-radar sensors 108), interacting with an application (e.g., browsing, selecting, or opening an application), controlling a user interface of a media player or other application, composing and taking pictures, interacting with reminders or notifications, or managing phone calls. In this way, the radar system 104 can provide touchless control of the smart phone 102. Exemplary 3D gestures and corresponding actions are described below with reference to Figures 10 to 16 Give a description.

[0051] As reference Figures 3 to 6 As described, radar system 104 can use radar field 110 to sense and analyze reflections from objects in radar field 110 in a manner that enables high resolution and accuracy for both gesture recognition and body gestures. Thus, 3D gestures can be any of various types of 3D gestures, such as arm gestures, hand or finger gestures, micro gestures, or other types of gestures. In addition, 3D gestures can be predefined, selected from a list, or customized (e.g., a user can interact with interaction manager 106 and radar system 104 to define a unique gesture or combination of gestures, such as corresponding to a specific action).

[0052] The smartphone power management interface 118 provides a communication link between the smartphone 102 and the interaction manager 106. In some embodiments, the smartphone power management interface 118 also acts as a (refer to Figure 3The interaction manager 106 may use the smartphone power management interface 118 to send and receive updates regarding any of the various power-related features of the smartphone 102 (e.g., power state, battery charge, or remaining battery charge). The smartphone power management interface 118 also enables the interaction manager 106 to send and receive control signals to the smartphone 102, which may be used to adjust, control, manage, or modify power-related features (e.g., enter a lower power state, exit a sleep mode, etc.). For example, as described above, when the interaction manager 106 maintains the radar system 104 in an idle mode, an attention mode, or an interactive mode, the interaction manager 106 may also maintain the smartphone 102 in an appropriate power mode (e.g., a sleep mode, a wake-up mode, or other power level mode) associated with the mode of the radar system 104. The interaction manager 106 may use the smartphone power management interface 118 to perform these functions. Additionally, in some embodiments, as described above, the smartphone power management interface 118 may also be used to maintain the non-radar sensors 108 in a non-operating state.

[0053] like Figure 1 , the 3D gesture module 116 and the smartphone power management interface 118 are depicted as part of the interaction manager 106. However, in other embodiments, either or both of the 3D gesture module 116 and the smartphone power management interface 118 may be separate entities, which may be part of the smartphone 102 or separate therefrom.

[0054] In more detail, consider Figure 2 , Figure 2 An example implementation 200 is illustrated in which a smartphone 102 (including a radar system 104 , an interaction manager 106 , and non-radar sensors 108 ) may implement a smartphone-based radar system for determining user intent in a low-power mode. Figure 2 The smartphone 102 of FIG. 1 is illustrated with various example devices, including a smartphone 102-1, a tablet computer 102-2, a laptop computer 102-3, a desktop computer 102-4, a computing watch 102-5, computing glasses 102-6, a gaming system 102-7, a home automation and control system 102-8, and a microwave oven 102-9. The smartphone 102 may also include other devices such as televisions, entertainment systems, audio systems, cars, drones, trackpads, drawing tablets, netbooks, e-readers, home security systems, and other home appliances. Note that the smartphone 102 may be wearable, non-wearable but mobile, or relatively non-mobile (e.g., desktops and appliances).

[0055] It should be noted that exemplary overall lateral dimensions of smartphone 102 may be, for example, approximately eight centimeters by approximately fifteen centimeters. An exemplary footprint of radar system 104 may be further limited, such as approximately four millimeters by six millimeters including the antenna. An exemplary power consumption of radar system 104 may be on the order of a few milliwatts (mW) to several mW (e.g., between approximately 2 mW and 20 mW). Such limited footprint requirements for radar system 104, which are required to accommodate many other desirable features of smartphone 102 (e.g., fingerprint sensor, non-radar sensor 108, etc.) in such a space-constrained package combined with power and processing constraints, may result in tradeoffs in accuracy and effectiveness of radar gesture detection, at least some of which may be overcome based on the teachings herein.

[0056] The smart phone 102 also includes one or more computer processors 202 and one or more computer readable media 204, including memory media and storage media. Applications and / or operating systems (not shown) implemented as computer readable instructions on the computer readable media 204 can be executed by the computer processor 202 to provide some or all of the functionality described herein. The smart phone 102 can also include a network interface 206. The smart phone 102 can use the network interface 206 to transmit data through a wired, wireless or optical network. As an example and not limitation, the network interface 206 can transmit data through a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), a wide area network (WAN), an intranet, the Internet, a peer-to-peer network, a point-to-point network, or a mesh network.

[0057] Various implementations of the radar system 104 may include a system on a chip (SoC), one or more integrated circuits (ICs), a processor having embedded processor instructions or configured to access processor instructions stored in a memory, hardware having embedded firmware, a printed circuit board having various hardware components, or any combination thereof. The radar system 104 operates as a monostatic radar by sending and receiving its own radar signals. In some implementations, the radar system 104 may also cooperate with other radar systems 104 within the external environment to implement a bistatic radar, a multistatic radar, or a network radar. However, the constraints or limitations of the smart phone 102 may affect the design of the radar system 104. For example, the smart phone 102 may have limited power available to operate the radar, limited computing power, size constraints, layout limitations, an external housing that attenuates or distorts the radar signal, and the like. The radar system 104 includes several features that enable advanced radar functionality and high performance despite these constraints, as described below with respect to Figure 3 further described. Note that in Figure 2 , the radar system 104 and the interaction manager 106 are illustrated as part of the smart phone 102. In other embodiments, either or both of the radar system 104 and the interaction manager 106 may be separate from or remote from the smart phone 102.

[0058] These and other capabilities and configurations and Figure 1 The way entities act and interact will be described in more detail below. These entities can be further divided, merged, etc. Figure 1 Environment 100 and Figures 2 to 18 The detailed diagrams illustrate some of the many possible environments and devices in which the described techniques can be employed. Figures 3 to 6 Additional details and features of radar system 104 are described. Figures 3 to 6 , radar system 104 is described in the context of smart phone 102, but as mentioned above, the applicability of the features and advantages of the described systems and techniques is not necessarily limited thereto, and other embodiments involving other types of electronic devices may also be within the scope of the present teachings.

[0059] Figure 3 An example implementation 300 of a radar system 104 is illustrated that can be used to enable a smartphone-based radar system for determining user intent in a lower power mode. In the example 300, the radar system 104 includes at least one of each of the following components: a communication interface 302, an antenna array 304, a transceiver 306, a processor 308, and a system medium 310 (e.g., one or more computer-readable storage media). The processor 308 can be implemented as a digital signal processor, a controller, an application processor, another processor (e.g., the computer processor 202 of the smartphone 102), or some combination thereof. The system medium 310, which can be included in or separate from the computer-readable medium 204 of the smartphone 102, includes one or more of the following modules: a fade mitigator 314, a digital beamformer 316, an angle estimator 318, or a power manager 320. These modules can compensate for or mitigate the effects of integrating the radar system 104 into the smartphone 102, thereby enabling the radar system 104 to recognize small or complex gestures, distinguish between different orientations of the user, continuously monitor the external environment, or achieve a target false alarm rate. These features can be used in Figure 2 Radar system 104 is implemented in a variety of different devices as shown.

[0060] Using the communication interface 302, the radar system 104 can provide radar data to the interaction manager 106. The communication interface 302 can be a wireless interface or a wired interface based on whether the radar system 104 is implemented separately from the smartphone 102 or integrated therein. Depending on the application, the radar data can include raw or minimally processed data, in-phase and quadrature (I / Q) data, range-Doppler data, processed data including target location information (e.g., range, azimuth, elevation), cluttered map data, etc. Typically, the radar data contains information that can be used by the interaction manager 106 to determine the user's intent based on the smartphone in a lower power mode.

[0061] The antenna array 304 includes at least one transmitting antenna unit (not shown) and at least two receiving antenna units (such as Figure 4 ). In some cases, antenna array 304 may include multiple transmit antenna elements to implement a multiple-input multiple-output (MIMO) radar capable of transmitting multiple different waveforms at a time (e.g., a different waveform for each transmit antenna element). The use of multiple waveforms may improve the measurement accuracy of radar system 104. For embodiments including three or more receive elements, the receive antenna elements may be positioned in a one-dimensional shape (e.g., a line) or a two-dimensional shape. A one-dimensional shape enables radar system 104 to measure one angular dimension (e.g., azimuth or elevation), while a two-dimensional shape enables measurement of two angular dimensions (e.g., both azimuth and elevation). Reference Figure 4 An example two-dimensional arrangement of receive antenna elements is further described.

[0062] Figure 4 An example arrangement 400 of receiving antenna elements 402 is shown. For example, if the antenna array 304 includes at least four receiving antenna elements 402, the receiving antenna elements 402 may be arranged in a rectangular arrangement 404-1, such as Figure 4 Alternatively, if the antenna array 304 includes at least three receiving antenna elements 402, a triangular arrangement 404-2 or an L-shaped arrangement 404-3 may be used.

[0063] Due to the size or layout constraints of the smartphone 102, the element spacing between the receiving antenna elements 402 or the number of receiving antenna elements 402 may not be ideal for the angles that the radar system 104 is to monitor. In particular, the element spacing may cause the presence of angular ambiguity, which makes it challenging for conventional radars to estimate the angular position of a target. Therefore, conventional radars may limit the field of view (e.g., the angle to be monitored) to avoid ambiguous areas with angular ambiguity, thereby reducing false detections. For example, a conventional radar may limit the field of view to angles between approximately -45 degrees and 45 degrees to avoid angular ambiguity that occurs using a wavelength of 5 millimeters (mm) and an element spacing of 3.5 mm (e.g., an element spacing of 70% of the wavelength). Therefore, conventional radars may not be able to detect targets beyond the 45-degree field of view limit. In contrast, the radar system 104 includes a digital beamformer 316 and an angle estimator 318 that resolve the angular ambiguity and enable the radar system 104 to monitor angles beyond the 45-degree limit, such as angles between approximately -90 degrees and 90 degrees, or up to approximately -180 degrees and 180 degrees. These angular ranges may be applied in one or more directions (eg, azimuth and / or elevation). Thus, radar system 104 may achieve low false alarm rates for a variety of different antenna array designs, including element spacings less than, greater than, or equal to half the center wavelength of the radar signal.

[0064] Using antenna array 304, radar system 104 can form beams that are steered or unsteered, wide or narrow, or shaped (e.g., as a hemisphere, cube, sector, cone, or cylinder). As an example, one or more transmit antenna elements (not shown) may have an unsteered omnidirectional radiation pattern, or may be capable of producing a wide beam such as wide transmit beam 406. Any of these techniques enables radar system 104 to illuminate a large volume of space. However, in order to obtain target angular accuracy and angular resolution, receive antenna elements 402 and digital beamformer 316 may be used to generate thousands of narrow and steered beams (e.g., 2000 beams, 4000 beams, or 6000 beams), such as narrow receive beam 408. In this way, radar system 104 can effectively monitor the external environment and accurately determine the angle of arrival of reflections within the external environment.

[0065] return Figure 3, transceiver 306 includes circuits and logic for sending and receiving radar signals via antenna array 304. Components of transceiver 306 may include amplifiers, mixers, switches, analog-to-digital converters, filters, etc. to condition radar signals. Transceiver 306 may also include logic to perform in-phase / quadrature (I / Q) operations, such as modulation or demodulation. Transceiver 306 may be configured for continuous wave radar operation or pulse radar operation. Various modulations may be used to generate radar signals, including linear frequency modulation, triangular frequency modulation, stepped frequency modulation, or phase modulation.

[0066] The transceiver 306 may generate a radar signal in a frequency range (e.g., spectrum) such as between 1 gigahertz (GHz) and 400 GHz, between 4 GHz and 100 GHz, or between 57 GHz and 63 GHz. The spectrum may be divided into a plurality of sub-spectra having similar bandwidths or different bandwidths. The bandwidth may be on the order of 500 megahertz (MHz), 1 GHz, 2 GHz, etc. As an example, different frequency sub-spectra may include frequencies between approximately 57 GHz and 59 GHz, between 59 GHz and 61 GHz, or between 61 GHz and 63 GHz. Multiple frequency sub-spectra having the same bandwidth and which may be continuous or non-continuous may also be selected for coherence. Multiple frequency sub-spectra may be transmitted simultaneously or separated in time using a single radar signal or multiple radar signals. Continuous frequency sub-spectra enable the radar signal to have a wider bandwidth, while non-continuous frequency sub-spectra may further emphasize amplitude and phase differences, which enables the angle estimator 318 to resolve angle ambiguity. The attenuation mitigator 314 or the angle estimator 318 may enable the transceiver 306 to utilize one or more frequency sub-spectra to improve the performance of the radar system 104, such as with respect to Figure 5 and Figure 6 Further described.

[0067] The power manager 320 enables the radar system 104 to save power internally or externally within the smartphone 102. In some embodiments, the power manager 320 communicates with the interaction manager 106 or the smartphone power management interface 118 to save power within one or both of the radar system 104 or the smartphone 102. For example, internally, the power manager 320 can cause the radar system 104 to collect data using a predetermined power mode or a specific duty cycle. In such a case, the power manager 320 dynamically switches between different power modes so that response delays and power consumption are managed together based on the activity within the environment. In general, the power manager 320 determines when and how to save power, and incrementally adjusts power consumption to enable the radar system 104 to operate within the power constraints of the smartphone 102. In some cases, the power manager 320 can monitor the amount of remaining available power and adjust the operation of the radar system 104 accordingly. For example, if the amount of remaining power is low, the power manager 320 can continue to operate in a lower power mode instead of switching to a higher power mode.

[0068] For example, the lower power mode may use a lower duty cycle on the order of a few hertz (e.g., about 1 Hz or less than 5 Hz), which reduces power consumption to a few milliwatts (mW) (e.g., between about 2 mW and 8 mW). On the other hand, the higher power mode may use a higher duty cycle on the order of tens of hertz (Hz) (e.g., about 20 Hz or greater than 10 Hz), which causes the radar system 104 to consume power on the order of several milliwatts (e.g., between about 6 mW and 20 mW). Although the lower power mode may be used to monitor the external environment or detect an approaching user, if the radar system 104 determines that the user is beginning to perform a gesture, the power manager 320 may switch to a higher power mode. Different triggers may cause the power manager 320 to switch between different power modes. Exemplary triggers include motion or lack of motion, the appearance or disappearance of a user, the user moving into or out of a specified area (e.g., an area defined by distance, azimuth, or elevation), a change in the speed of motion associated with the user, or a change in the reflected signal strength (e.g., due to a change in radar cross section). Typically, triggers indicating a low likelihood of user interaction with smartphone 102 or a preference to collect data using longer response delays may result in activation of a lower power mode to conserve power.

[0069] The power manager 320 may also save power by shutting down one or more components within the transceiver 306 (e.g., a voltage controlled oscillator, a multiplexer, an analog-to-digital converter, a phase-locked loop, or a crystal oscillator) during periods of inactivity. These periods of inactivity may occur if the radar system 104 is not actively transmitting or receiving radar signals, which may be on the order of microseconds (µs), milliseconds (ms), or seconds (s). In addition, the power manager 320 may modify the transmission power of the radar signal by adjusting the amount of amplification provided by the signal amplifier. In addition, the power manager 320 may control the use of different hardware components within the radar system 104 to save power. For example, if the processor 308 includes a lower power processor and a higher power processor (e.g., a processor with different amounts of memory and computing power), the power manager 320 may switch between using the lower power processor for low-level analysis (e.g., implementing an idle mode, detecting motion, determining a user's location, or monitoring the environment) and using the higher power processor for situations where the interaction manager 106 requests high fidelity or accurate radar data (e.g., for implementing an attention mode or an interaction mode, gesture recognition, or user orientation).

[0070] In addition to the internal power saving techniques described above, the power manager 320 can also save power by activating or deactivating other external components or sensors within the smart phone 102. These external components may include speakers, camera sensors, global positioning systems, wireless communication transceivers, displays, gyroscopes, or accelerometers. Because the radar system 104 can use a small amount of power to monitor the environment, the power manager 320 can appropriately turn these external components on or off based on where the user is or what the user is doing. In this way, the smart phone 102 can seamlessly respond to the user and save power without using an automatic shutoff timer or the user physically touching or verbally controlling the smart phone 102. The power management techniques described can therefore be used to provide various embodiments of the idle mode, attention mode, and interactive mode as described herein.

[0071] Figure 5Additional details of an example implementation 500 of a radar system 104 within a smartphone 102 are illustrated. In example 500, the antenna array 304 is located below an external housing (such as a glass cover or housing) of the smartphone 102. Depending on its material properties, the external housing can act as an attenuator 502 that attenuates or distorts radar signals sent and received by the radar system 104. The attenuator 502 can include different types of glass or plastic, some of which can be found within the display screen, external housing, or other components of the smartphone 102 and have a dielectric constant (e.g., relative permittivity) between approximately four and ten. Thus, the attenuator 502 is opaque or translucent to the radar signal 506 and can cause a portion of the transmitted or received radar signal 506 to be reflected (as shown by the reflected portion 504). For conventional radars, the attenuator 502 can reduce the effective range that can be monitored, prevent detection of small targets, or reduce overall accuracy.

[0072] Assuming that the transmit power of the radar system 104 is limited and redesigning the external housing is undesirable, one or more attenuation-related characteristics of the radar signal 506 (e.g., the frequency sub-spectrum 508 or the steering angle 510) or the attenuation-related characteristics of the attenuator 502 (e.g., the distance 512 between the attenuator 502 and the radar system 104 or the thickness 514 of the attenuator 502) are adjusted to mitigate the effects of the attenuator 502. Some of these characteristics may be set during manufacturing or adjusted by the attenuation mitigator 314 during operation of the radar system 104. For example, the attenuation mitigator 314 may cause the transceiver 306 to transmit the radar signal 506 using the selected frequency sub-spectrum 508 or steering angle 510, cause the platform to move the radar system 104 closer to or further away from the attenuator 502 to change the distance 512, or prompt the user to apply another attenuator to increase the thickness 514 of the attenuator 502.

[0073] Appropriate adjustments may be made by the attenuation mitigator 314 based on predetermined characteristics of the attenuator 502 (e.g., characteristics stored in the computer-readable medium 204 of the smart phone 102 or within the system medium 310) or by processing the return of the radar signal 506 to measure one or more characteristics of the attenuator 502. Even if certain attenuation-related characteristics are fixed or constrained, the attenuation mitigator 314 may still take these constraints into account to balance each parameter and achieve the target radar performance. As a result, the attenuation mitigator 314 enables the radar system 104 to achieve enhanced accuracy and a greater effective range to detect and track users located on opposite sides of the attenuator 502. These techniques provide an alternative to increasing the transmit power (which increases the transmit power of the radar system 104) or changing the material properties of the attenuator 502 (which may be difficult and expensive once the device is in production).

[0074] Figure 6An example scheme 600 implemented by radar system 104 is illustrated. Portions of scheme 600 may be performed by processor 308, computer processor 202, or other hardware circuits. Scheme 600 may be customized to support different types of electronic devices and radar-based applications (e.g., interaction manager 106), and enables radar system 104 to achieve target angle accuracy despite design limitations.

[0075] The transceiver 306 generates raw data 602 based on the individual responses of the receive antenna units 402 to the received radar signals. The received radar signals may be associated with one or more frequency sub-spectra 604 selected by the angle estimator 318 to facilitate angular ambiguity resolution. For example, the frequency sub-spectra 604 may be selected to reduce the number of side lobes or reduce the amplitude of the side lobes (e.g., reduce the amplitude by 0.5 dB, 1 dB, or more). The number of frequency sub-spectra may be determined based on the target angle accuracy or the computational limitations of the radar system 104.

[0076] The raw data 602 includes digital information (e.g., in-phase and quadrature data) for a time period, different wave numbers, and multiple channels associated with the receiving antenna unit 402, respectively. A fast Fourier transform (FFT) 606 is performed on the raw data 602 to generate pre-processed data 608. The pre-processed data 608 includes digital information across time periods for different distances (e.g., a range grid) and for multiple channels. A Doppler filtering process 610 is performed on the pre-processed data 608 to generate range-Doppler data 612. The Doppler filtering process 610 may include another FFT that generates amplitude and phase information for multiple range grids, multiple Doppler frequencies, and for multiple channels. The digital beamformer 316 generates beamforming data 614 based on the range-Doppler data 612. The beamforming data 614 includes digital information for a set of azimuth and / or elevation angles, which represents the field of view of different steering angles or beams formed by the digital beamformer 316. Although not depicted, the digital beamformer 316 may alternatively generate beamformed data 614 based on the preprocessed data 608, and the Doppler filtering process 610 may generate range-Doppler data 612 based on the beamformed data 614. To reduce the amount of computation, the digital beamformer 316 may process a portion of the range-Doppler data 612 or the preprocessed data 608 based on a range, time, or Doppler frequency interval of interest.

[0077] The digital beamformer 316 may be implemented using a single-look beamformer 616, a multi-look interferometer 618, or a multi-look beamformer 620. Typically, the single-look beamformer 616 may be used for deterministic objects (e.g., point source targets with a single phase center). For non-deterministic targets (e.g., targets with multiple phase centers), the multi-look interferometer 618 or multi-look beamformer 620 is used to improve accuracy relative to the single-look beamformer 616. A person is an example of a non-deterministic target and has multiple phase centers 622 that may vary based on different line-of-sight angles (as shown in 624-1 and 624-2). The variation in constructive or destructive interference generated by the multiple phase centers 622 may pose a challenge to conventional radars in accurately determining angular position. However, the multi-look interferometer 618 or multi-look beamformer 620 performs coherent averaging to increase the accuracy of the beamformed data 614. The multi-look interferometer 618 coherently averages the two channels to generate phase information that can be used to accurately determine angle information. On the other hand, the multi-look beamformer 620 can coherently average two or more channels using a linear or nonlinear beamformer such as Fourier, Capon, multiple signal classification (MUSIC), or minimum variance distortionless response (MVDR). The increased accuracy provided via the multi-look beamformer 620 or multi-look interferometer 618 enables the radar system 104 to recognize small gestures or distinguish between multiple parts of a user.

[0078] Angle estimator 318 analyzes beamforming data 614 to estimate one or more angular positions. Angle estimator 318 may utilize signal processing techniques, pattern matching techniques, or machine learning. Angle estimator 318 also resolves angle ambiguities that may be caused by the design of radar system 104 or the field of view monitored by radar system 104. Example angle ambiguities are shown in amplitude graph 626 (e.g., amplitude response).

[0079] The magnitude graph 626 depicts the magnitude differences that may occur for different angular positions of the target and for different steering angles 510. A first magnitude response 628-1 (shown with a solid line) is shown for a target at a first angular position 630-1. Similarly, a second magnitude response 628-2 (shown with a dashed line) is shown for a target at a second angular position 630-2. In this example, differences are considered at angles between -180 degrees and 180 degrees.

[0080] As shown in the amplitude graph 626, there is an ambiguous region for two angular positions 630-1 and 630-2. The first amplitude response 628-1 has the highest peak at the first angular position 630-1 and a smaller peak at the second angular position 630-2. Although the highest peak corresponds to the actual position of the target, the smaller peak causes the first angular position 630-1 to be ambiguous because it is within a certain threshold and the conventional radar may not be able to confidently determine whether the target is at the first angular position 630-1 or the second angular position 630-2. In contrast, the second amplitude response 628-2 has a smaller peak at the second angular position 630-2 and a larger peak at the first angular position 630-1. In this case, the smaller peak corresponds to the position of the target.

[0081] While conventional radars may be limited to using the highest peak amplitude to determine angular position, angle estimator 318 instead analyzes subtle differences in the shapes of amplitude responses 628-1 and 628-2. Characteristics of the shapes may include, for example, roll-off, peak or null width, angular position of peaks or nulls, height or depth of peaks and nulls, sidelobe shapes, symmetry in amplitude response 628-1 or 628-2, or lack of symmetry within amplitude response 628-1 or 628-2. Similar shape characteristics may be analyzed in the phase response, which may provide additional information to resolve angle ambiguity. Thus, angle estimator 318 maps a unique angle signature or pattern to an angular position.

[0082] The angle estimator 318 may include an algorithm or toolkit that may be selected based on the type of smartphone 102 (e.g., computing power or power constraints) or the target angle resolution of the interaction manager 106. In some embodiments, the angle estimator 318 may include a neural network 632, a convolutional neural network (CNN) 634, or a long short-term memory (LSTM) network 636. The neural network 632 may have various depths or numbers of hidden layers (e.g., three hidden layers, five hidden layers, or ten hidden layers), and may also include different numbers of connections (e.g., the neural network 632 may include a fully connected neural network or a partially connected neural network). In some cases, the CNN 634 may be used to increase the computational speed of the angle estimator 318. The LSTM network 636 may be used to enable the angle estimator 318 to track a target. Using machine learning techniques, the angle estimator 318 employs a nonlinear function to analyze the shape of the magnitude response 628-1 or 628-2 and generates angle probability data 638, which indicates the likelihood that the user or a portion of the user is within an angular bin. The angle estimator 318 may provide angle probability data 638 for a few angle grids (such as two angle grids) to provide the probability that the target is to the left or right of the smartphone 102, or provide angle probability data 638 for thousands of angle grids (eg, to provide angle probability data 638 for continuous angle measurements).

[0083] Based on the angle probability data 638, the tracker module 640 generates angular position data 642, which identifies the angular position of the target. The tracker module 640 can determine the angular position of the target based on the angle grid with the highest probability in the angle probability data 638 or based on prediction information (e.g., previously measured angular position information). The tracker module 640 can also track one or more moving targets so that the radar system 104 can distinguish or identify the targets with confidence. Other data can also be used to determine the angular position, including range, Doppler, velocity, or acceleration. In some cases, the tracker module 640 may include an alpha-beta tracker, a Kalman filter, a multiple hypothesis tracker (MHT), and the like.

[0084] Quantizer module 644 takes angular position data 642 and quantizes the data to produce quantized angular position data 646. Quantization may be performed based on the target angular resolution used for interaction manager 106. In some cases, fewer quantization levels may be used so that quantized angular position data 646 indicates whether the target is to the right or left of smart phone 102, or identifies that the target is located within a 90 degree quadrant. For some radar-based applications, such as user proximity detection, this may be sufficient. In other cases, a large number of quantization levels may be used so that quantized angular position data 646 indicates the angular position of the target within an accuracy of a fraction of a degree, one degree, five degrees, etc. This resolution may be used for higher resolution radar-based applications, such as gesture recognition, or in the embodiments of the attention mode or interaction mode described herein. In some embodiments, digital beamformer 316, angle estimator 318, tracker module 640, and quantizer module 644 are implemented together in a single machine learning module.

[0085] These and other capabilities and configurations and Figures 1 to 6 The manner in which the entities act and interact is set forth below. The described entities may be further divided, combined, used with other sensors or components, etc. In this manner, different embodiments of smartphones 102 having different configurations of radar system 104 and non-radar sensors may be used to implement a smartphone-based radar system for determining user intent in a lower power mode. Figure 1 An example operating environment 100 and Figures 2 to 6 The detailed illustrations illustrate only some of the many possible environments and devices in which the described techniques can be employed. Example Method

[0086] Figure 7-11 Depicted are example methods 700 and 1000 for implementing a smartphone-based radar system for determining user intent in a lower power mode. Methods 700 and 1000 can be performed using an electronic device that uses a radar system to provide a radar field. The radar field is used to determine the presence of an object in the radar field. The radar field can also be used to determine the object's intent level, and the intent level can be used to determine whether the object intends to interact with the electronic device. Based on the determination of the object's intent, the electronic device can enter and exit different modes of functionality and power usage.

[0087] The method 700 is shown as a set of boxes that specify the operations performed, but is not necessarily limited to the order or combination shown for performing the operations by the corresponding boxes. In addition, any one or more of these operations may be repeated, combined, reorganized, or linked to provide a variety of additional and / or alternative methods. In the various parts of the following discussion, reference may be made to Figure 1 The example operating environment 100 of Figure 2-6 The entities or processes detailed in the description are for example reference only. These techniques are not limited to being performed by one entity or multiple entities operating on one device.

[0088] At 702, a radar field is provided. This radar field can be provided by any of a variety of electronic devices (e.g., the smartphone 102 described above) that include or are associated with a radar system (e.g., the radar system 104) and an interaction manager (e.g., the interaction manager 106, which may also include any one or both of the 3D gesture module 116 and the smartphone power management interface 118). Additionally, the radar field may be any of a variety of types of radar fields, such as the radar field 110 described above.

[0089] At 704, reflections from objects in the radar field are sensed by the radar system. The object may be any of a variety of objects, such as wood, plastic, metal, fabric, or organic material (e.g., a person, such as the above-mentioned objects, or a body part of a person). For clarity, while describing method 700, the objects are referred to as "one or more users."

[0090] At 706, reflections from objects in the radar field are analyzed. The analysis may be performed by any of a variety of entities (eg, radar system 104, interaction manager 106, or another entity) and may include various operations or determinations, such as reference Figure 3-6 Describes the action or determination.

[0091] At 708, based on the analysis of the reflections, radar data (eg, reference Figure 1-6 The radar data described herein may be provided by any of a variety of entities, such as radar system 104, interaction manager 106, or another entity. In some embodiments, the radar system may provide the radar data and communicate the radar data to other entities (e.g., any of the radar-based applications, interaction managers, or modules described herein). A description of method 700 is provided in Figure 8 Continue in Figure 7 The letter "A" after the box 708 of Figure 8 Corresponding to the letter “A” before box 710 of .

[0092] At 710, the radar system is maintained in an idle mode (eg, by the interaction manager). Figure 1As noted, the idle mode may be a lower power radar mode that requires no more than 30 milliwatts (mW) of power. In some implementations, the idle mode may require a different amount of power, such as about 2 mW or about 8 mW. Additionally, the interaction manager may also maintain the electronic device in a lower power state (e.g., a sleep mode or other power saving mode) when the radar system is maintained in the idle mode.

[0093] At 712, based on the subset of radar data, the presence of a user within a perception zone of the electronic device is determined. A perception zone is a zone around the radar system where the presence of a user can be accurately determined by the user without verbal, touch, or other input (e.g., perception zone 114). Figure 1 As noted, the perception zone may extend any of a variety of distances from the radar system.The presence of a user within the perception zone may be determined by any of a variety of entities, such as an interaction manager.

[0094] At 714, in response to determining that the user is within the perception zone, the radar system can enter an attention mode. The radar system can be placed into an attention mode by the interaction manager or by another entity. The attention mode is a radar mode that allows the radar system to provide additional information about objects (such as users) within the perception zone. Figure 1 As noted, the attention mode may require no more than about 60 mW of power. In other implementations, the attention mode may require a different amount of power, such as between about 8 mW and about 55 mW or between about 2 mW and about 20 mW. Additionally, when the radar system is maintained in the attention mode, the electronic device may be maintained in a lower power state that may be used with the idle mode, or the electronic device may exit the lower power state and enter another power state (e.g., awake mode, active mode, etc.).

[0095] At 716, in response to the electronic device entering the attention mode, and based on another subset of the radar data, a level of intent of a user within the perception zone is determined. The level of intent can be determined based on various information about the user within the perception zone that can be determined based on the other subset of the radar data. For example, while in the attention mode, the radar system can provide other radar data (including one or more other subsets of radar data) that can be used as reference Figure 1 Determine any one or both of the user's body position or posture relative to the electronic device as described (including changes in the user's body position or posture). A description of method 700 is provided in Fig. 9 Continue in Figure 7 The letter "B" after the box 716 of FIG. Fig. 9 The letter "B" preceding box 718 corresponds to the letter "B".

[0096] At 718, based on the level of intent, it is determined whether the user intends to interact with the electronic device. Figure 1 As described, the interaction manager (or another module or entity) is able to use the level of intent (including one or more of various non-verbal body language cues, body positioning, or body posture) to determine whether the user intends to communicate or interact with the electronic device or whether the user does not intend to communicate or interact with the electronic device.

[0097] At 720, in response to determining that the level of intent indicates that the user does not intend to interact with the electronic device, the radar system can exit the attention mode and enter an idle mode. The mode of the radar system can be changed by the interaction manager or by another entity. Figure 1 As noted, when the radar system enters (or re-enters) the idle mode, the electronic device may also enter a lower power state. Note that the attention mode described in blocks 714-720 is optional. In some implementations, as described below, when (e.g., at block 712) the presence of a user within the perception zone of the electronic device is determined, the radar system may exit the idle mode and enter the interactive mode without entering the attention mode.

[0098] At 722, in response to determining that the level of intent indicates that the user intends to interact with the electronic device, the radar system can exit the attention mode and enter the interaction mode. Figure 1 As noted, the interactive mode may require no more than about 90 mW of power. In other implementations, the interactive mode may require different amounts of power, such as about 55 mW or about 20 mW. Figure 1 As described, when the radar system is maintained in an interactive mode (e.g., while a user is interacting with the electronic device), the electronic device can exit a lower power state and be maintained in another appropriate power mode. The radar system can be entered into the interactive mode by an interaction manager or by another entity.

[0099] The interactive mode is a radar mode that allows the radar system to provide additional information about the user (or other objects) in the perception zone. For example, while in the interactive mode, the radar system can provide additional radar data (including one or more additional subsets of radar data as described herein) that can be used to enable the radar system to determine 3D gestures made by the user and process the 3D gestures in a manner that enables the user to interact with smartphone 102 via the 3D gestures. For example, as described in reference Figure 3-6 As described, a radar system is able to use the radar field to sense and analyze reflections from objects in the radar field in a manner that enables high resolution and accuracy for 3D gesture recognition and for determining body positioning and posture.

[0100] The method 1000 is shown as a set of boxes that specify the operations performed, but is not necessarily limited to the order or combination shown for performing the operations by the corresponding boxes. In addition, any one or more of these operations may be repeated, combined, reorganized, or linked to provide a variety of additional and / or alternative methods. In the various parts of the following discussion, reference may be made to Figure 1 The example operating environment 100 of Figure 2-9 The entities or processes detailed in the description are for example reference only. These techniques are not limited to being performed by one entity or multiple entities operating on one device.

[0101] At 1002, a radar field is provided. This radar field can be provided by any of a variety of electronic devices (eg, electronic devices similar to those described with reference to method 700 at block 702).

[0102] At 1004, reflections from objects in the radar field are sensed by the radar system (eg, in a manner similar to that described with reference to method 700 at block 704). For clarity, the objects are referred to as "one or more users" while describing method 1000.

[0103] At 1006, reflections from objects in the radar field are analyzed. The analysis may be performed by any of a variety of entities (eg, as described with reference to method 700 at block 706).

[0104] At 1008, radar data is provided based on the analysis of the reflections (eg, as described with reference to method 700 at block 708). A description of method 1000 is provided at Fig.11 Continue in Fig.10 The letter "A" after the box 1008 of Fig.11 Corresponding to the letter “A” before box 1010.

[0105] At 1010, the radar system is maintained in a lower power mode (e.g., by the interaction manager). The lower power mode may be an idle mode or an attention mode as described above or another lower power mode. In some implementations, the lower power mode requires no more than about 30 mW of power (e.g., when the lower power mode is an idle mode). In other implementations, the lower power mode requires no more than about 60 mW of power (e.g., when the lower power mode is an attention mode). Additionally, the interaction manager may also maintain the electronic device in a lower power state (e.g., a sleep mode or other power saving mode) while the radar system is maintained in the lower power mode.

[0106] At 1012 , based on the subset of radar data, the presence of a user within a perception zone of the electronic device (eg, the perception zone described with reference to method 700 at block 712 ) is determined.

[0107] Optionally, at 1014, in response to determining that the user is within the perception zone, the radar system can enter an attention mode. The radar system can be entered into the attention mode by the interaction manager or by another entity. The attention mode can be a radar mode that allows the radar system to provide additional information about objects (such as users) near the electronic device (e.g., within the perception zone). In some cases, the attention mode may require no more than about 60 mW of power. In other implementations, the attention mode may require a different amount of power, such as between about 8 mW and about 55 mW or between about 2 mW and about 20 mW. In addition, when the radar system is maintained in the attention mode, the electronic device can be maintained in a lower power state that can be used with the idle mode, or the electronic device can exit the lower power state and enter another power state (e.g., wake-up mode, active mode, etc.). In addition, in some implementations, the intent level of the user within the perception zone is determined, and based on the intent level, it is determined whether the user intends to interact with the electronic device (e.g., as described with reference to method 700 at blocks 716 and 718). In response to determining that the level of intent indicates that the user does not intend to interact with the electronic device, the radar system can exit the attention mode and enter an idle mode (eg, as described with reference to method 700 at block 720 ).

[0108] At 1016, in response to determining the presence of a user within the perception zone, the radar system may exit the idle mode and enter the interactive mode. Figure 1 As noted, the interactive mode may require no more than about 90 mW of power. In other implementations, the interactive mode may require different amounts of power, such as about 55 mW or about 20 mW. Figure 1 As described, when the radar system is maintained in an interactive mode (e.g., while a user is interacting with the electronic device), the electronic device can exit a lower power state and be maintained in another appropriate power mode. The radar system can be entered into the interactive mode by an interaction manager or by another entity.

[0109] Consider for example Figures 12 to 18 , Figures 12 to 18 An example implementation of an electronic device capable of implementing additional details of method 700 is illustrated. Fig.12An example implementation 1200 of an electronic device (in this case, a smartphone 102, which is in the pocket of a user 1202 who is listening to music using an application on the smartphone 102) is depicted. The smartphone 102 includes a radar system 104 that is providing a radar field 110. In the example implementation 1200, it is assumed that the interaction manager has determined (e.g., by determining that the user 1202 is leaning toward the smartphone 102) that the user 1202 intends to interact with the smartphone 102. In response, the interaction manager is maintaining the radar system in an interactive mode so that the user 1202 can interact with the smartphone 102. For example, Fig.12 , user 1202 is interacting with smartphone 102 by making a sweep gesture indicated by arrow 1024. The sweep gesture can cause the music application on smartphone 102 to skip to the next track.

[0110] Other 3D gestures can be used to control other features or applications. For example, a set of 3D gestures can control the playback experience of multimedia applications (e.g., music, video, presentations, or collaborative applications and games) from a distance. Other 3D gestures can include air taps for play / pause, swipes up or down for adjusting volume, pinch gestures for copying and pasting text, or rotation gestures for selecting new content. The same types of 3D gestures can be used in a "casting" context. For example, if a user is projecting content from a smartphone 102 (or another device) to another device (e.g., a large-screen TV), the user can control the media experience on the other device using 3D gestures via the smartphone 102 in interactive mode.

[0111] Similarly, Fig.13 An example implementation 1300 of an electronic device (in this case, a smartphone 102) is depicted, which is located on a countertop near a user 1302. The smartphone 102 includes a radar system 104 that is providing a radar field 110. In the example implementation 1300, assume that the user 1302 is mixing wet ingredients in a bowl according to a recipe displayed on the smartphone 102 and that the user's hands are covered with the wet ingredients. Further assume that the interaction manager has determined that the user 1302 intends to interact with the smartphone 102 (e.g., by determining that the user 1302 is leaning toward the smartphone 102 and reaching out). In response, the interaction manager is maintaining the radar system in an interactive mode so that the user 1302 can interact with the smartphone 102. For example, as Fig.13 , user 1302 is interacting with smart phone 102 by making a reach gesture indicated by arrow 1304. The reach gesture can cause smart phone 102 to scroll or advance the displayed recipe so that user 1302 can read the next steps.

[0112] Fig.14An example implementation 1400 of an electronic device, in this case a smartphone 102, is depicted, which is being held up by a user 1402, who is positioning the smartphone 102 to take a picture (e.g., a "selfie") with another user 1404. The smartphone 102 includes a radar system 104 that is providing a radar field 110. In the example implementation 1000, it is assumed that the interaction manager has determined (e.g., by determining that the user 1404 is reaching toward the smartphone 102) that the user 1404 intends to interact with the smartphone 102. In response, the interaction manager is maintaining the radar system in an interactive mode, enabling the user 1404 to interact with the smartphone 102. For example, as Fig.14 , user 1404 is interacting with smartphone 102 by making a sweeping gesture indicated by arrow 1406. The sweeping gesture can cause smartphone 102 to apply filters, adjust zoom level, change flash settings, etc. Other 3D gestures such as tapping, twisting, or rotating can control other camera functions, allowing users 1402 and 1404 to comfortably take pictures without overstretching or posing in an unnatural way to reach the controls.

[0113] Fig.15 An example implementation 1500 of an electronic device (in this case, a smartphone 102) is depicted, which is being held by a user 1502 who is engaged in a task that requires the use of multiple applications on the smartphone 102. The smartphone 102 includes a radar system 104 that is providing a radar field 110. In the example implementation 1500, it is assumed that the interaction manager has determined that the user 1502 intends to interact with the smartphone 102 (e.g., by determining that the user 1502 leans toward the smartphone 102 and picks up the smartphone 102). In response, the interaction manager is maintaining the radar system in an interactive mode so that the user 1502 can interact with the smartphone 102. For example, as Fig.15 , user 1502 is interacting with smartphone 102 by making a sweep gesture indicated by arrow 1504. The sweep gesture can cause smartphone 102 to switch between different applications (eg, forward and backward chronologically or in a predefined order).

[0114] Fig.16An example implementation 1600 of an electronic device (in this case, a smartphone 102, which is located on a nightstand near a user 1602) is depicted. The smartphone 102 includes a radar system 104 that is providing a radar field 110. In the example implementation 1600, it is assumed that the interaction manager has determined that the user 1602 intends to interact with the smartphone 102 (e.g., by determining that the user 1602 is reaching toward the smartphone 102). In response, the interaction manager is maintaining the radar system in an interactive mode so that the user 1602 can interact with the smartphone 102. For example, in Fig.16 , further assume that a morning alarm has awakened user 1602 who is interacting with smartphone 102 by making a swipe gesture indicated by arrow 1604. The swipe gesture can cause smartphone 102 to turn off or delay (e.g., snooze) the alarm so that user 1602 can prepare to get up without having to look for touch input on the alarm interface.

[0115] Fig.17 An example implementation 1700 of an electronic device (in this case, a smartphone 102) is depicted, which is located on a work surface near a user 1702, who is working on a task that does not require the use of the smartphone 102. The smartphone 102 includes a radar system 104 that is providing a radar field 110. In the example implementation 1700, it is assumed that the interaction manager has determined that the user 1702 intends to interact with the smartphone 102 (e.g., by determining that the user 1702 is reaching toward the smartphone 102, even if the user 1702 is not actually turning toward the smartphone 102). In response, the interaction manager is maintaining the radar system in an interactive mode so that the user 1702 can interact with the smartphone 102. For example, in Fig.17 , it is further assumed that the smart phone 102 is ringing (receiving a phone call), but the user 1702 does not want to be interrupted. The user 1702 interacts with the smart phone 102 by making a sweep gesture indicated by arrow 1704. The sweep gesture can cause the smart phone 102 to silence the ringer or send the phone call to voicemail, allowing the user 1702 to complete the current task without having to answer the phone or find a touch input.

[0116] Fig.18An example implementation 1800 of an electronic device (in this case, a smartphone 102) is depicted, which is located on a countertop near a user 1802, who is dining with another user. The smartphone 102 includes a radar system 104 that is providing a radar field 110. In the example implementation 1800, it is assumed that the interaction manager has determined that the user 1802 intends to interact with the smartphone 102 (e.g., by determining that the user 1802 is reaching toward the smartphone 102, even though the user 1802 is not actually turning toward the smartphone 102). In response, the interaction manager is maintaining the radar system in an interaction mode, enabling the user 1802 to interact with the smartphone 102. For example, in Fig.18 , it is further assumed that the smart phone 102 is displaying notifications, but the user 1802 does not want to be interrupted. The user 1802 interacts with the smart phone 102 by making a sweep gesture indicated by arrow 1806. The sweep gesture can cause the smart phone 102 to hide and postpone the notifications, allowing the user 1802 to enjoy the meal without having to resolve the notifications or look for touch inputs.

[0117] like Figure 12-18 As shown in the examples illustrated in , the techniques of method 700 and / or 1000 can enable a user to interact with a smartphone 102 without removing the smartphone 102 from a pocket or other container or having to touch the smartphone 102. These techniques thus provide several advantages. For example, the interaction space of the smartphone 102 (and other electronic devices) is extended beyond the screen, which keeps the electronic device useful and accessible when touching the screen or other input methods would be inconvenient, unsafe, or uncomfortable. In addition, these techniques allow users to quickly access and interact with their electronic devices (e.g., silence an alarm or alert) without being distracted, allowing the user to take advantage of updates and information available via the electronic device, but try not to be distracted.

[0118] It should be noted that these techniques for determining user intent in a lower power mode based on a smartphone radar system may be more secure than other techniques. Not only are 3D gestures (especially user-defined gestures, micro-gestures, and gestures based on posture or orientation) generally inaccessible to unauthorized persons (e.g., unlike passwords), but also because the user's radar image includes the user's face, the user cannot be visually identified like a photo or video. Even so, in addition to the above description, the user may be provided with a control that allows the user to make a choice about whether and when any of the systems, programs, managers, modules, or features described in this document can enable the collection of user information (e.g., information about the user's social network, social actions or activities, occupation, user's preferences, or the user's current location) and whether and when the user is sent content or communication from the server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of the user may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (such as to a city, zip code / postal code, or state level) when positioning information is obtained so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how that information is used, and what information is provided to or about the user. Example computing system

[0119] Fig.19 Illustrate various components of an example computing system 1900, which may be implemented as described with reference to the previous Figures 1 to 18 Any type of client, server, and / or electronic device described may be used to implement a smartphone-based radar system for determining user intent in a lower power mode.

[0120] The computing system 1900 includes a communication device 1902 that enables wired and / or wireless communication of device data 1904 (e.g., radar data, 3D gesture data, authentication data, reference data, received data, data being received, data scheduled for broadcast, and data packets of data). The device data 1904 or other device content may include configuration settings for the device, media content stored on the device, and / or information associated with a user of the device (e.g., the identity of a person within a radar field or customized gesture data). The media content stored on the computing system 1900 may include any type of radar, biometric, audio, video, and / or image data. The computing system 1900 includes one or more data inputs 1906, via which any type of data, media content, and / or input may be received, such as human speech, interaction with a radar field, touch input, user-selectable input (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source. The data input 1906 may include, for example, the interaction manager 106 , the 3D gesture module 116 , or the smartphone power management interface 118 .

[0121] The computing system 1900 also includes a communication interface 1908, which can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and any other type of communication interface. The communication interface 1908 provides a connection and / or communication link between the computing system 1900 and a communication network through which other electronic, computing, and communication devices communicate data with the computing system 1900.

[0122] The computing system 1900 includes one or more processors 1910 (e.g., any of a microprocessor, controller, or other controller) that can process various computer-executable instructions to control the operation of the computing system 1900 and enable a smartphone-based radar system for determining user intent in a lower power mode or a technology in which a smartphone-based radar system for determining user intent in a lower power mode can be implemented. Alternatively or additionally, the computing system 1900 can be implemented using any one or combination of hardware, firmware, or fixed logic circuits implemented in conjunction with processing and control circuits, which are generally identified at 1912. Although not shown, the computing system 1900 may include a system bus or data transmission system that connects various components within the device. The system bus may include any one or combination of different bus structures (such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of a variety of bus architectures).

[0123] The computing system 1900 also includes computer-readable media 1914, such as one or more memory devices that enable persistent and / or non-transitory data storage (i.e., as opposed to mere signal transmission), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and disk storage devices. The disk storage device may be implemented as any type of magnetic or optical storage device (such as a hard drive), a recordable and / or rewritable compact disk (CD), any type of digital versatile disk (DVD), etc. The computing system 1900 may also include a mass storage medium device (storage medium) 1916.

[0124] The computer-readable medium 1914 provides a data storage mechanism to store device data 1904 as well as various device applications 1918 and any other type of information and / or data related to the operational aspects of the computing system 1900. For example, an operating system 1920 may be maintained as a computer application with the computer-readable medium 1914 and executed on the processor 1910. The device applications 1918 may include a device manager (e.g., any form of control application), software applications, signal processing and control modules, native code for a specific device, abstract modules, gesture recognition modules, and other modules. The device applications 1918 may also include system components, engines, or managers to implement a smartphone-based radar system for determining user intent in a lower power mode, such as the radar system 104, the interaction manager 106, the 3D gesture module 116, or the smartphone power management interface 118. The computing system 1900 may also include or have access to one or more machine learning systems.

[0125] Several examples are described in the following paragraphs.

[0126] Example 1: A smart phone, comprising: a radar system, the radar system being implemented at least in part in hardware and configured to: provide a radar field; sense reflections from an object in the radar field; analyze the reflections from the object in the radar field; and provide radar data based on the analysis of the reflections; one or more computer processors; and one or more computer-readable media having instructions stored thereon, the instructions implementing a persistent radar-based interaction manager in response to being executed by the one or more computer processors, the persistent radar-based interaction manager being configured to: maintain the radar system in an idle mode, the idle mode requiring no more than approximately 30 milliwatts of power; determine, based on a first subset of the radar data, the object in the sensing of the smart phone; in response to determining the presence of the object within the perception zone, causing the radar system to enter an attention mode, the attention mode requiring no more than approximately 60 milliwatts of power; in response to entering the attention mode, and based on a second subset of the radar data, determining an intent level of the object; based on the intent level, determining whether the object intends to interact with the smart phone; in response to determining that the intent level indicates that the object does not intend to interact with the smart phone, causing the radar system to exit the attention mode and enter the idle mode; and in response to determining that the intent level indicates that the user intends to interact with the smart phone, causing the radar system to exit the attention mode and enter an interaction mode, the interaction mode requiring no more than approximately 90 milliwatts of power.

[0127] Example 2: The smartphone of Example 1, wherein the persistent radar-based interaction manager is further configured to maintain the smartphone in a power saving mode in response to maintaining the radar system in the idle mode.

[0128] Example 3: The smartphone of Example 2, wherein the persistent radar-based interaction manager is further configured to cause the smartphone to exit the power saving mode in response to causing the radar system to enter the interaction mode.

[0129] Example 4: The smartphone of Example 2, wherein the persistent radar-based interaction manager is further configured to maintain the smartphone in the power saving mode in response to causing the radar system to enter the attention mode.

[0130] Example 5: The smartphone of Example 2, wherein the persistent radar-based interaction manager is further configured to cause the smartphone to exit the power saving mode in response to causing the radar system to enter the attention mode.

[0131] Example 6: The smartphone of any of the preceding examples, wherein the idle mode requires no more than about eight milliwatts of power.

[0132] Example 7: The smartphone of any of the preceding examples, wherein the attention mode requires between about eight milliwatts of power and about 55 milliwatts of power.

[0133] Example 8: The smartphone of Example 1, wherein the interactive mode requires no more than about 55 milliwatts of power.

[0134] Example 9: The smartphone of any of the preceding examples, wherein the radar system further comprises a digital beamformer and an angle estimator, and wherein the radar system is configured to monitor angles in the field of view between approximately -90 degrees and approximately 90 degrees.

[0135] Example 10: A smart phone according to any of the preceding examples, wherein: the idle mode enables the radar system to provide the radar field to scan the environment outside the smart phone and determine the presence of the object; and the interactive mode enables the radar system to determine a 3D gesture made by the object in the radar field and process the 3D gesture, thereby effectively enabling the object to interact with the smart phone via the 3D gesture.

[0136] Example 11: A smart phone according to any of the preceding examples, wherein the persistent radar-based interaction manager is further configured to: determine the presence of the user without requiring the user to provide verbal or touch input; and determine the user's level of intent without requiring the user to provide verbal or touch input.

[0137] Example 12: The smartphone of Example 1, wherein the persistent radar-based interaction manager is further configured to cause a change in a state of a display of the smartphone based on the determined level of intent of the user.

[0138] Example 13: A method implemented in an electronic device including a radar system, the method comprising: providing a radar field by the radar system; sensing reflections from objects in the radar field by the radar system; analyzing the reflections from the objects in the radar field; providing radar data based on the analysis of the reflections; maintaining the radar system in an idle mode, the idle mode requiring no more than approximately 30 milliwatts of power; determining the presence of the object within a perception zone of the electronic device based on a first subset of the radar data; in response to determining the presence of the object within the perception zone, causing the radar system to enter an attention mode, the attention mode requiring no more than approximately 60 milliwatts of power; in response to entering the attention mode and based on a second subset of the radar data, determining an intent level of the object; based on the intent level, determining whether the object intends to interact with the electronic device; in response to determining that the intent level indicates that the object does not intend to interact with the electronic device, causing the radar system to exit the attention mode and enter the idle mode; and in response to determining that the intent level indicates that the user intends to interact with the electronic device, causing the radar system to exit the attention mode and enter an interaction mode, the interaction mode requiring no more than approximately 90 milliwatts of power.

[0139] Example 14: The method according to Example 13 further includes: in response to maintaining the radar system in the idle mode, maintaining the electronic device in the power saving mode; and in response to the radar system entering the interactive mode, causing the electronic device to exit the power saving mode.

[0140] Example 15: The method of Example 14 further comprises, in response to the radar system entering the attention mode, maintaining the electronic device in the power saving mode.

[0141] Example 16: The method of Example 14 further comprises, in response to the radar system entering the attention mode, causing the electronic device to exit the power saving mode.

[0142] Example 17: The method of any of Examples 13-16, wherein the idle mode requires no more than approximately eight milliwatts of power.

[0143] Example 18: The method of any of Examples 13-17, wherein the attention mode requires between about eight milliwatts of power and about 55 milliwatts of power.

[0144] Example 19: The method of any of Examples 13-18, wherein the interactive mode requires no more than about 55 milliwatts of power.

[0145] Example 20: A smart phone, comprising: a radar system, the radar system being implemented at least in part in hardware and configured to: provide a radar field; sense reflections from objects in the radar field; analyze the reflections from the objects in the radar field; and provide radar data based on the analysis of the reflections; one or more computer processors; and one or more computer-readable media having instructions stored thereon, the instructions, in response to being executed by the one or more computer processors, implementing a persistent radar-based interaction manager, the persistent radar-based interaction manager being configured to: maintain the radar system in a lower power mode, the lower power mode requiring no more than approximately 30 milliwatts of power; determine, based on a first subset of the radar data, the presence of the object within a perception zone of the electronic device; and in response to determining the presence of the object within the perception zone, cause the radar system to exit the lower power mode and enter an interactive mode, the interactive mode requiring no more than approximately 90 milliwatts of power.

[0146] Example 21: A method implemented in an electronic device including a radar system, the method comprising: providing a radar field by the radar system; sensing reflections from objects in the radar field by the radar system; analyzing the reflections from the objects in the radar field; providing radar data based on the analysis of the reflections; maintaining the radar system in a lower power mode, the lower power mode requiring no more than approximately 30 milliwatts of power; determining the presence of the object within a perception zone of the electronic device based on a first subset of the radar data; and in response to determining the presence of the object within the perception zone, causing the radar system to exit the lower power mode and enter an interactive mode, the interactive mode requiring no more than approximately 90 milliwatts of power.

[0147] Example 22: The method of Example 21, wherein the lower power mode is an idle mode or an attention mode. in conclusion

[0148] Although embodiments of techniques and apparatus for enabling a smartphone-based radar system for determining user intent in a lower power mode have been described in language specific to features and / or methods, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example embodiments of implementing a smartphone-based radar system for determining user intent in a lower power mode.

Claims

1. An electronic device, comprising: A radar system, the radar system being implemented at least partially in hardware, the radar system being configured to: Provide radar fields; sensing reflections from objects in the radar field; providing an analysis of said reflection from said object; and providing radar data based on said analyzing of said reflections; as well as One or more computer-readable media having instructions stored thereon, the instructions in response to being executed by one or more computer processors, implementing a persistent radar-based interaction manager, the persistent radar-based interaction manager being configured to: Maintaining the radar system in an idle mode, wherein: requiring power not exceeding a first threshold; and including operating the radar system with a first set of radar characteristics, the first set of radar characteristics comprising at least one of a first maximum transmit power available to the radar system, a first set of sensors available to the radar system, a first set of one or more processors available to the radar system, or a first duty cycle of the radar system; determining whether the presence of the object is within a sensing area of ​​the electronic device based on the radar data; as well as In response to determining that the object is within the perception zone, causing the radar system to exit the idle mode and enter an interactive mode, wherein: A power not exceeding a second threshold is required, the power of the second threshold being greater than the power of the first threshold; as well as The radar system is included in operating with a second set of radar characteristics, the second set of radar characteristics comprising at least one of a second maximum transmit power available to the radar system, a second set of sensors available to the radar system, a second set of one or more processors available to the radar system, or a second duty cycle of the radar system.

2. The electronic device according to claim 1, wherein: The persistent radar-based interaction manager is further configured to cause a display of the electronic device to display at least one of an authentication screen, an intelligent assistant, or a notification in response to causing the radar system to enter the interaction mode.

3. The electronic device according to claim 1, wherein: The interaction mode enables the radar system to determine 3D gestures made by the object in the radar field and process the 3D gestures, which effectively enable the object to interact with the electronic device without speech or touch input.

4. The electronic device according to claim 3, wherein: The electronic device further comprises: a non-radar sensor configured to determine a context of the electronic device, wherein the persistent radar-based interaction manager is further configured to: Based on the context of the electronic device, interpreting the 3D gesture as a specific action; and The specific action is caused to be performed on the electronic device.

5. The electronic device according to claim 4, wherein: The object includes a user of the electronic device; and The context of the electronic device includes at least one of: the electronic device is in a user's clothing pocket, the electronic device is placed on a flat surface, the electronic device is held in the user's hand, the electronic device is in an environment with a low amount of ambient light, a multimedia application on the electronic device is playing media, a notification is available on the electronic device, an application with scrollable content is displayed on the electronic device, or an image capture application is opened on the electronic device.

6. The electronic device according to claim 4, wherein: The electronic device includes a smart phone; The subject comprises a user of the smart phone; The context of the electronic device includes the smartphone being held away from the user and facing the user, and an image capture application being open on the smartphone; as well as The 3D gestures include at least one of: a horizontal swipe gesture for applying a filter, adjusting the zoom level, or changing the flash setting; an air tap for capturing an image or video; a vertical swipe gesture for adjusting a setting of the image capture application; or a rotation gesture for adjusting another setting of the image capture application.

7. The electronic device according to claim 4, wherein: The electronic device includes a smart phone; The subject comprises a user of the smart phone; The context of the electronic device includes the smartphone on a flat surface and an application with content displayed on a display of the smartphone; as well as The 3D gesture includes at least one of the following: a vertical sweep gesture for scrolling the content on the display; Or a pinch gesture made by the user's fingers to zoom in or out the content of the display.

8. The electronic device according to claim 4, wherein: The electronic device comprises a smart phone; The subject comprises a user of the smart phone; The context of the electronic device includes the smartphone being on a flat surface and an alarm on the smartphone being activated; and The 3D gesture includes at least one of: a sweep gesture for closing or delaying the alarm; an air tap for delaying or closing the alarm; or a rotation gesture for resetting the alarm.

9. The electronic device according to claim 4, wherein: The electronic device comprises a smart phone; The subject comprises a user of the smart phone; The context of the electronic device includes the smartphone being on a flat surface and the smartphone displaying a notification on a display; as well as The 3D gesture includes at least one of: a sweep gesture for hiding or postponing the notification; an air tap for causing the smart phone to display additional information related to the notification; Or a rotation gesture to dismiss the notification.

10. The electronic device according to claim 1, wherein: The electronic device includes at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a computing watch, computing glasses, a gaming system, a home automation and control system, a television, an entertainment system, an audio system, a car, a drone, a trackpad, a drawing tablet, a netbook, an e-reader, a home security system, or a home appliance.

11. A method implemented in an electronic device including a radar system, the method comprising: providing a radar field by said radar system; sensing, by the radar system, reflections from objects in the radar field; analyzing said reflections from said object; providing radar data based on analysis of the reflections; Maintaining the radar system in an idle mode, wherein: requiring a power not exceeding a first threshold; as well as including operating the radar system with a first set of radar characteristics, the first set of radar characteristics comprising at least one of a first maximum transmit power available to the radar system, a first set of sensors available to the radar system, a first set of one or more processors available to the radar system, or a first duty cycle of the radar system; determining whether the presence of the object is within a sensing area of ​​the electronic device based on the radar data; as well as In response to determining that the object is within the perception zone, causing the radar system to exit the idle mode and enter an interactive mode, wherein: A power not exceeding a second threshold is required, the power of the second threshold being greater than the power of the first threshold; as well as The radar system is included in operating with a second set of radar characteristics, the second set of radar characteristics comprising at least one of a second maximum transmit power available to the radar system, a second set of sensors available to the radar system, a second set of one or more processors available to the radar system, or a second duty cycle of the radar system.

12. The method according to claim 11, further comprising: In response to causing the radar system to enter the interactive mode, causing a display of the electronic device to display at least one of an authentication screen, a smart assistant, or a notification.

13. The method according to claim 11, wherein: The interaction mode enables the radar system to determine 3D gestures made by the object in the radar field and process the 3D gestures, which effectively enable the object to interact with the electronic device without speech or touch input.

14. The method according to claim 13, further comprising: determining a context of the electronic device by a non-radar sensor; Based on the context of the electronic device, interpreting the 3D gesture as a specific action; as well as The specific action is caused to be performed on the electronic device.

15. The method according to claim 14, wherein: The object includes a user of the electronic device; and The context of the electronic device includes at least one of: the electronic device is in a user's clothing pocket, the electronic device is placed on a flat surface, the electronic device is held in the user's hand, the electronic device is in an environment with a low amount of ambient light, a multimedia application on the electronic device is playing media, a notification is available on the electronic device, an application with scrollable content is displayed on the electronic device, or an image capture application is opened on the electronic device.

16. The method of claim 14, wherein: The electronic device includes a smart phone; The subject comprises a user of the smart phone; The context of the electronic device includes the smartphone being held away from the user and facing the user, and an image capture application being open on the smartphone; as well as The 3D gestures include at least one of: a horizontal swipe gesture for applying a filter, adjusting the zoom level, or changing the flash setting; an air tap for capturing an image or video; a vertical swipe gesture for adjusting a setting of the image capture application; or a rotation gesture for adjusting another setting of the image capture application.

17. The method of claim 14, wherein: The electronic device comprises a smart phone; The subject comprises a user of the smart phone; The context of the electronic device includes the smartphone on a flat surface and an application with content displayed on a display of the smartphone; as well as The 3D gesture includes at least one of the following: a vertical sweep gesture for scrolling the content on the display; Or a pinch gesture made by the user's fingers to zoom in or out the content of the display.

18. The method of claim 14, wherein: The electronic device comprises a smart phone; The subject comprises a user of the smart phone; The context of the electronic device includes the smartphone being on a flat surface and an alarm on the smartphone being activated; and The 3D gesture includes at least one of: a sweep gesture for closing or delaying the alarm; an air tap for delaying or closing the alarm; or a rotation gesture for resetting the alarm.

19. The method of claim 14, wherein: The electronic device comprises a smart phone; The subject comprises a user of the smart phone; The context of the electronic device includes the smartphone being on a flat surface and the smartphone displaying a notification on a display; as well as The 3D gesture includes at least one of: a sweep gesture for hiding or postponing the notification; an air tap for causing the smart phone to display additional information related to the notification; Or a rotation gesture to dismiss the notification.

20. The method according to claim 11, wherein: The electronic device includes at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a computing watch, computing glasses, a gaming system, a home automation and control system, a television, an entertainment system, an audio system, a car, a drone, a trackpad, a drawing tablet, a netbook, an e-reader, a home security system, or a home appliance.

Images