MACRO-MICRO GESTURE DETECTION

Abstract

Macro-micro gesture detection is provided. The execution of a macro segment of a macro-micro gesture is detected by a mobile device using a vehicle's Phone-as-a-Key (PaaK) sensors, where the macro-micro gesture requests a function call from the vehicle. Using one or more sensors of an inertial measurement unit (IMU) on the mobile device, the execution of a micro segment of the macro-micro gesture is detected. Arbitration is performed to validate that the vehicle function is to be executed with respect to both the micro and macro segments. The vehicle function is then called in response to a successful validation.

Description

AREA OF TECHNOLOGY

[0001] Aspects of the disclosure relate to macro-micro gesture detection using Phone-as-a-Key distance measurement (PaaK distance measurement) in combination with sensors of an inertial measurement unit (IMU) (IMU sensors) of a mobile device. GENERAL STATE OF THE ART

[0002] Some vehicles can be unlocked or started with a digital key. A digital key, sometimes implemented with a smartphone (e.g., PaaK), relies on communication between a mobile device, such as a smartphone, and the vehicle. When an application is activated on the mobile device and the device is held in a specific location relative to the vehicle, such as next to the door handle, the vehicle unlocks the doors. A digital key can communicate with the vehicle using Bluetooth®, near-field communication (NFC), and / or ultra-wideband (UWB).

[0003] An IMU is an electronic device that measures and reports the specific force of a body, the angular velocity and sometimes the orientation of the body using a combination of accelerometers, gyroscopes and sometimes magnetometers. SUMMARY

[0004] In one or more illustrative examples, a macro-micro gesture detection method includes: detecting, using phone-as-a-key (PaaK) sensors of a vehicle, the execution of a macro-section of a macro-micro gesture by a mobile device, wherein the macro-micro gesture requests a call to a function of the vehicle; detecting, using one or more sensors of an inertial measurement unit (IMU) of the mobile device, the execution of a micro-section of the macro-micro gesture; arbitrating to validate that the function of the vehicle is to be executed with respect to the execution of both the micro-section and the macro-section; and calling the function of the vehicle in response to the successful validation.

[0005] In one or more illustrative examples, a macro-micro gesture detection system includes a vehicle transceiver; a plurality of UWB anchors; and a controller communicating with the transceiver and the plurality of UWB anchors. The controller is configured to detect, using the vehicle's UWB anchors, the execution of a macro-section of a macro-micro gesture by a mobile device, where the macro-micro gesture requests a function call from the vehicle; to detect a micro-section of the macro-micro gesture using one or more sensors of an inertial measurement unit (IMU) on the mobile device; to be arbitrated by the vehicle to validate that the vehicle function is to be executed with respect to both the micro-section and the macro-section; and to call the vehicle function in response to the successful validation.

[0006] In one or more illustrative examples, a non-transient, computer-readable medium comprises instructions which, when executed by a vehicle controller communicating with a transceiver and a plurality of UWB anchors, cause the controller to perform operations, including: detecting, using the vehicle's UWB anchors; executing a macro-section of a macro-micro gesture by a mobile device, wherein the macro-micro gesture requests a call to a function of the vehicle; detecting a micro-section of the macro-micro gesture using one or more sensors of an inertial measurement unit (IMU) of the mobile device; and arbitrating, by the vehicle, to validate that the vehicle's function is to be executed with respect to the execution of both the micro-section and the macro-section.and calling up the vehicle's function in response to the successful validation. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates an exemplary system, including a vehicle, that implements macro-micro gesture detection; Fig. Figure 2 illustrates an example of the mobile device beginning to approach the vehicle; Fig. Figure 3A illustrates an example of the vehicle's PaaK system, which triangulates the mobile device; Fig. Figure 3B illustrates an alternative example of the mobile device that triangulates its own location using a feed of UWB data from the vehicle; Fig. Figure 4A illustrates an example of the vehicle's PaaK system sending a macro-detection message to the mobile device; Fig. Figure 4B illustrates an example of the mobile device that detects the macro gesture using UWB data feeds; Fig. 5A illustrates an example of a micro-gesture performed using the mobile device; Fig. Figure 6A illustrates an example of the mobile device that sends a micro-transmission detection message to the vehicle; Fig. Figure 6B illustrates an example of the vehicle detecting the micro-gesture using IMU data; Fig. Figure 7 illustrates an example of the vehicle performing the function based on the execution of the combined macro-micro gesture; Fig. Figure 8 illustrates an example of a mobile device leaving the vicinity of the vehicle; Fig. Figure 9 illustrates an exemplary process for performing macro-micro gesture detection using the vehicle in combination with the mobile device; and Fig. Figure 10 illustrates an exemplary computing device for use in implementing macro-micro gesture detection. DETAILED DESCRIPTION

[0007] Depending on the requirements, detailed embodiments of the present invention are disclosed here; however, it is understood that the disclosed embodiments are merely exemplary of the invention, which can be implemented in various and alternative forms. The figures are not necessarily to scale; some features may be greatly enlarged or reduced to show details of specific components. Therefore, specific structural and functional details disclosed in this document are not to be interpreted as limiting, but merely as a representative basis to teach those skilled in the art the diverse applications of the present invention.

[0008] Accessing a vehicle's trunk, especially after a major shopping trip, can be difficult. This can be particularly true for a vehicle not equipped with hands-free operation. However, hands-free technology can present a trade-off between convenience and frustration if the technology doesn't activate as expected or does so unintentionally. For example, when returning home, it might be desirable for the trunk to close automatically when access is no longer needed, but not too soon, in case there are still more items to unload.

[0009] There are other scenarios where triggering vehicle functionality can be helpful. One way to achieve this is through interaction with a phone app. However, this approach typically requires the use of both hands, which is not ideal in many situations. Another solution is to use voice or camera activation. However, voice activation requires additional hardware in the vehicle or the use of the phone's microphone, which can be problematic since the user may not always be in a location that allows for voice recognition. Camera detection can also be difficult to implement due to its higher performance requirements. Capacitive and ultrasonic systems can be used, but such systems can be ineffective at distinguishing between intended and unintended activations.Radar can also be used, but this requires increased costs and performance.

[0010] UWB is a radio access technology (RAT) that has become widely used as part of a digital keying system, such as PaaK, either alone or in combination with Bluetooth Low Energy (BLE), to precisely locate the user device and reduce the risk of playback attacks. An improved approach can analyze a user's gesture by leveraging PaaK features to precisely locate an authorized smartphone by triangulating the travel time of multiple receivers. This solution offers the advantages of detecting a user gesture without requiring any additional hardware. It can also guarantee robust detection, as motion detection is deterministic within the limits of UWB triangulation accuracy.

[0011] Nevertheless, the quality of gesture detection is limited by the UWB sampling rate. A gesture is typically completed in less than a second. Some hand movements are even shorter. At a sampling rate of around 100 ms, this means that potentially only 5 to 10 samples are available for a decision. Additionally, such an approach tracks the phone's position. Since UWB resolution is a few centimeters, small gestures may not be detectable. This requires the user to perform a sweeping swipe, creating a trajectory that is distinctly different from someone typically walking near the vehicle (such as a swipe of the arm while holding the phone above the head and downwards). Such a requirement can be difficult or undesirable.

[0012] By leveraging the combination of the vehicle's PaaK UWB location services and the smartphone's internal sensors and actuators, macro-micro gesture detection can be performed. This approach can recognize a gesture as a combination of a macro movement, detected by the PaaK features, and a micro movement, detected by the IMU features of the smartphone or other mobile user device. This allows for the creation of a vocabulary of multiple gestures. Each gesture can be tailored to the specific task and context. Each gesture can also be predefined or user-created.

[0013] The macro component can be captured using the mobile device's UWB location. For example, the vehicle's PaaK location service can detect an approaching trajectory as the user moves closer to the vehicle (e.g., towards the tailgate). In another example, the vehicle can send UWB location data to the mobile device, and the mobile device can detect the approaching trajectory. In either case, this can be considered the macro part of the macro-micro gesture.

[0014] The microcomponent can be detected using the mobile device's IMU sensors. For example, a request can be sent to the mobile device at a predefined distance to begin locally detecting gestures using its IMU. The IMU can detect a very wide range of microgestures, such as swiping movements, tracked using a combination of accelerometer and gyroscope signals; tilting / rotating movements using only the gyroscope; or strong vibrations / impulses using only the accelerometer.In another example, the mobile device can provide the vehicle with a feed of data from one or more sensors of an inertial measurement unit (IMU) of the mobile device (and / or a wearable device communicating with the mobile device), and the execution of the micro-segment of the macro-micro gesture by the vehicle can be detected. This can be considered the micro-segment of the macro-micro gesture.

[0015] Arbitration can be performed to ensure that both the macro and micro sections of the macro-micro gesture are executed. This arbitration can be carried out by the mobile device and / or the vehicle. If the timing of the macro and micro sections is too far apart, or if the action cannot be performed for other reasons (e.g., if the door is blocked), feedback can be provided to the user indicating that the action cannot be performed. Further aspects of the disclosure are discussed in detail in this document.

[0016] Fig. Figure 1 illustrates an exemplary system 100, including a vehicle 102 that implements macro-micro gesture detection. As shown, the vehicle 102 includes a variety of UWB anchors 104, a transceiver 106, and a controller 108. The system 100 can be used to track the position of mobile devices 110. The mobile devices 110 can also include IMUs 112, which can be used to enable the vehicle 102 to understand aspects of the user's behavior.

[0017] Especially with regard to Fig. 1. Vehicle 102 can be any passenger or commercial vehicle, such as a car, truck, SUV, crossover, van, minivan, taxi, bus, etc. Vehicle 102 can include various types of automobiles, soft-roaders (crossover utility vehicles - CUVs), SUVs, trucks, motorhomes, motorcycles, boats, aircraft, or other mobile machinery for transporting people or goods. Such Vehicle 102 can be driven by people or be autonomous. In many cases, Vehicle 102 can be powered by a gasoline, diesel, or hydrogen engine. Alternatively, Vehicle 102 can be a battery electric vehicle powered by one or more electric motors.Alternatively, vehicle 102 could be a hybrid electric vehicle that is powered by both an internal combustion engine and one or more electric motors, such as a series hybrid electric vehicle, a parallel hybrid electric vehicle, or a parallel / series hybrid electric vehicle.

[0018] The UWB Anchor 104 communicates wirelessly with the Mobile Device 110 using radio waves. The UWB Anchor 104 uses an ultra-wideband signal, i.e., a low-energy signal propagated over a large portion of the radio spectrum. The Federal Communications Commission and the International Telecommunications Union Radiocommunication Sector define ultra-wideband as an antenna transmission where the bandwidth of the emitted signal exceeds 500 MHz or 20% of the arithmetic center frequency. The UWB Anchor 104 can use any suitable modulation technique, such as orthogonal frequency-division multiplexing (OFDM), phase-shift keying (PSK), pulse-position modulation (PPM), etc.

[0019] To enable robust user localization, the vehicle 102 can be equipped with UWB responders strategically positioned within the vehicle's interior and body structure to provide UWB network coverage of the environment in and around the vehicle 102, i.e., wherever the user's mobile device 110 may be located. Depending on the physical design and shape of the vehicle 102, some of the UWB anchors 104 may be located within the vehicle's body walls (e.g., four, each positioned near or at each corner of the front and rear bumpers), on the center console (e.g., between the driver and passenger seats), and within the roof (e.g., near the front center, near the rear center).

[0020] As in the example from Fig. Figure 1 shows seven UWB anchors 104. These include a first UWB anchor 104a, a second UWB anchor 104b, a third UWB anchor 104c, a fourth UWB anchor 104d, a fifth UWB anchor 104e, a sixth UWB anchor 104f, and a seventh UWB anchor 104g. The UWB anchors 104 are spaced apart, e.g., distributed over the vehicle 102, to increase the ability to distinguish a location when used for trilateration. For example, four of the UWB anchors 104 can be located at respective corners of the vehicle 102 to maximize the horizontal distribution of the UWB anchors 104, and the remaining three UWB anchors 104 can be located within a footprint of the vehicle 102 at different heights than the corner-mounted UWB anchors 104 to provide a vertical distribution. To perform trilateration, calculating the intersection of three or more circles or spheres can provide the location of the detected device.

[0021] The transceiver 106 can be configured to transmit signals wirelessly using a communication protocol other than that used by the UWB anchor 104, such as cellular, Bluetooth®, BLE, WiFi, Institute of Electrical and Electronics Engineers (IEEE) Standard 802.11a / b / g / p, Cellular-V2X (CV2X), Dedicated Short-Range Communications (DSRC), etc. The transceiver 106 is designed to communicate using a protocol also used by the mobile device 110. In particular, the transceiver 106 can use BLE. The transceiver 106 can be a device or include a separate transmitter and receiver.

[0022] The controller 108 can be a microprocessor-based computing device, e.g., a generic computing device that includes a processor and memory, an electronic controller or the like, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a combination of the above, etc. Typically, a hardware description language, such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) hardware description language), is used in electronic design automation to describe digital and mixed-signal systems, such as FPGAs and ASICs.For example, an ASIC is manufactured based on VHDL programming provided prior to manufacturing, whereas logical components within an FPGA may be configured based on VHDL programming stored, for example, in memory electrically connected to the FPGA circuit. The Controller 108 can thus include a processor, memory, and so on. The memory of the Controller 108 can include media for storing instructions that can be executed by the processor, as well as for electronically storing data and / or databases, and / or the Controller 108 can include structures such as those mentioned above, through which programming is provided. The Controller 108 can consist of several interconnected computers.

[0023] The controller 108 can transmit and receive data via a communication network, such as a Controller Area Network (CAN) bus, Ethernet, WiFi, a Local Interconnect Network (LIN), an On-Board Diagnostics (OBD-II) port, and / or any other wired or wireless communication network. The controller 108 can communicate with the UWB anchors 104, a transceiver 106, and other components via the communication network. The controller 108 can locate the position of the mobile device 110 using trilateration based on the distance information collected between each of the UWB anchors 104 and the mobile device 110.

[0024] The UWB anchor 104 and the transceiver 106 can communicate with at least one mobile device 110. The mobile devices 110 can include portable computing devices, such as smart key fobs; mobile phones, e.g., smartphones; wearable devices, e.g., smartwatches, headsets, etc.; tablets; smart tools, etc. The mobile devices 110 are computing devices that include their respective processors and memory. The mobile devices 110 can belong to and be carried by individuals who may be operators and / or owners of the vehicle 102.

[0025] To perform trilateration, a calculation of the intersection of three or more circles or spheres can be carried out. The UWB anchors 104 can be configured to transmit and receive signals (within signal power thresholds) over UWB channel frequencies (e.g., UWB channel 9 (7.737–8.236 GHz) to channel 5 (6.240–6.739 GHz) or other possible channels adopted by the UWB standard). Under ideal radio frequency (RF) conditions, e.g., when the mobile device 110 is within line of sight (LOS), three UWB anchors 104 may be sufficient to locate the mobile device 110, i.e., the initiator, and thereby enable trilateration-based localization of the user by measuring the distance between the responder and the initiator.However, due to the possibility of less favorable RF conditions, data from more than three UWB anchors 104 can be used by the controller 108 to ensure adequate wireless UWB coverage to locate the mobile device 110.

[0026] In addition to its use in PaaK, the controller 108 can also use the UWB anchors 104 to perform various presence detection. These presence detection features can include intrusion detection or the detection of children's presence. This can be achieved by measuring characteristics of a wireless channel between the inner transmit and receive UWB anchors 104.

[0027] For example, a channel impulse response (CIR) between the UWB anchors 104 can be used to characterize the wireless environment of the vehicle 102. The CIR can describe how a wireless channel responds to an impulse signal, which is a very short and high-energy signal. The CIR detects the amplitude, phase, and delay of the multipath components transmitted by a transmitter and received by a receiver after being reflected, refracted, or scattered within the environment. By observing the multipath components of the CIR caused by scattering at target objects, the movement of people inside the vehicle 102 can be detected.

[0028] The IMU 112 of the mobile device 110 can include various devices configured to detect the movements of the user of the mobile device 110. These IMUs 112 can include one or more accelerometers, gyroscopes, and / or magnetometers. The IMU 112 can be used for gesture detection, for example, by tracking the movements of the user of the mobile device 110.

[0029] A gesture to be recognized can be detected using a combination of a macro movement detected by the PaaK features (e.g. UWB anchor 104 and transceiver 106) and a micro movement detected by the features of the IMU 112 of the mobile device 110.

[0030] The macro component of the macro-micro gesture can be detected using the PaaK location of the mobile device 110. For example, the PaaK location services of the vehicle 102 can be used to detect an approaching movement path as the user moves closer to the vehicle 102, for instance, towards the tailgate. Conversely, the PaaK location services of the vehicle 102 can be used to detect a retreating movement path as the user moves away from the vehicle 102. These determinations can be made by the controller 108 using data from the UWB anchors 104 and / or by the mobile device 110 receiving a feed of location data from the UWB anchors 104.

[0031] The micro-component of the macro-micro gesture can be detected using the features of the IMU 112 of the mobile device 110. In one example, the vehicle 102 can, in response to the detection of the macro gesture, send a request to the mobile device 110 to begin local detection of the micro gesture using its IMU 112. In another example, the mobile device 110 can begin local detection of the micro gesture (e.g., in response to a detection of the proximity of the vehicle 102 via a detection of messages from the UWB anchors 104 and / or the transceiver 106 of the vehicle 102), regardless of whether the vehicle 102 detects the macro gesture. In yet another example, the mobile device 110 can send a feed of data from the IMU 112 to the vehicle 102, and the vehicle 102 can then determine whether the micro gesture is performed.The data from the one or more IMUs 112 can be used to detect a spectrum of micro-gestures, such as swiping movements tracked using a combination of accelerometer and gyroscope signals, tilting / rotating movements using only the gyroscope, or strong vibrations / impulses using only the accelerometer.

[0032] If both the macro and micro gestures of the macro-micro gesture are detected (and the macro-micro gesture is assigned to a function of vehicle 102), the corresponding function of vehicle 102 can be activated. Vehicle 102 and / or mobile device 110 can perform arbitration based on the occurrence of the macro and micro gestures. For example, vehicle 102 and / or mobile device 110 can confirm that both the macro and micro gestures are performed within a predefined time period. In some examples, the sequence may be required, so that the macro gesture must be performed first, followed by the micro gesture. In other examples, the sequence may be reversed, so that the micro gesture must be performed first, followed by the macro gesture.In other examples, the macro gesture and the micro gesture can be performed in any order and / or with at least partial temporal overlap within the predefined time period. Vehicle 102 can call the function if the timing is correct and if the function is capable of being activated. Vehicle 102 can also provide feedback to the user if the action cannot be performed (e.g., if an obstacle is detected in the path of a door whose opening is being requested).

[0033] Fig. Figure 2 illustrates an example 200 of the mobile device 110, which begins to approach the vehicle 102. For example, a user might carry the mobile device 110 in a pouch, purse, bag, or hand while approaching the vehicle 102. The transceiver 106 can be used to track the mobile device 110 when the user is still relatively far from the vehicle 102 (e.g., more than twenty meters in one example). This user distance monitoring can be performed based on a Bluetooth received signal strength indicator (RSSI). In response to the user's device being detected by the transceiver 106, the controller 108 can activate the UWB anchors 104 (e.g., as shown, UWB anchors 104a, 104d, 104g).As shown, the mobile device 110 is currently outside the range of the UWB anchors 104 of the PaaK system of vehicle 102. Vehicle 102 can begin tracking the user's position using UWB anchors 104, since vehicle 102 may know that the mobile device 110 is nearby and a distance measurement would be useful. Alternatively, vehicle 102 can initiate a feed of UWB data to the mobile device 110, and the mobile device 110 can then begin tracking the user's position using UWB anchors 104.

[0034] Vehicle 102 and / or mobile device 110 can be configured to use the PaaK system to detect the execution of a macro gesture. The macro gesture can comprise a sequence of mobile device 110 locations over time, which can be compared and matched against a predefined sequence of mobile device 110 locations over time. If the sequences match, the macro gesture can be considered executed. The macro gesture can include various gestures and / or movements detectable by Vehicle 102's PaaK system, such as kicking, punching, moving back and forth, walking forward and backward, jumping, etc.

[0035] Fig. Figure 3A illustrates an example 300A of the PaaK system of vehicle 102, which triangulates the mobile device 110. As noted above, the user carrying the mobile device 110 may have approached and stopped the vehicle 102. At this point, the controller 108 can localize the position of the mobile device 110 using trilateration based on the distance information collected between each of the UWB anchors 104 and the mobile device 110. For example, vehicle 102 may use three or four of the UWB anchors 104 that are closest to the mobile device 110 to perform the localization.

[0036] Fig. Figure 3B illustrates an alternative example 300B of the mobile device 110, which triangulates its own location using a feed 302 of UWB data from the vehicle 102. As noted above, the user carrying the mobile device 110 may have approached and stopped the vehicle 102. At this point, the controller 108 can initiate the transmission of raw data from the vehicle 102 to the mobile device 110, enabling the mobile device 110 to localize its position using trilateration. This feed 302 can, in one example, be sent via the transceiver 106. Alternatively, the mobile device 110 can analyze data directly from the UWB anchors 104 based on its communication with them. In these variations, the mobile device 110 can locate its own position based on the distance information collected between each of the UWB anchors 104 and the mobile device 110.

[0037] The macro gesture can be used to determine that the vehicle 102 and / or the mobile device 110 has stopped moving and is now near a feature of the vehicle 102. The threshold for determining the end of movement can be within a predefined range to allow for some movement, since the stationary portion of the macro-micro gesture's end of movement is only one component of the overall gesture and would not trigger the function itself. In the illustrated example, the function to be activated is a tailgate, but it could be any other function of the vehicle 102, such as a door, a rear door, a light, a speaker, a winch, a window, etc., that can be activated via gesture input.

[0038] Fig. Figure 4A illustrates an example 400A of the PaaK system of vehicle 102, which sends a macro gesture detection message 402 to the mobile device 110. Sending the macro gesture detection message 402 can be performed in response to the detection of the macro gesture by vehicle 102. Essentially, instead of activating the feature, vehicle 102 can send the macro gesture detection message 402 to the mobile device 110 in response to determining the occurrence of the macro gesture. In one example, the macro gesture detection message 402 can be sent from the transceiver 106 of vehicle 102 to the mobile device 110, for example, via Bluetooth or Bluetooth Low Energy (BLE). The macro-gesture detection message 402 can be received by the mobile device 110 to cause the mobile device 110 to begin detecting the micro section of the macro-micro gesture.

[0039] Fig. Figure 4B illustrates an example 400B of the mobile device 110, which detects the macro gesture using UWB data feeds. In such a case, the macro gesture detection message 402 is not sent from the vehicle 102 to the mobile device 110. Instead, the mobile device 110 determines locally that the macro gesture is performed.

[0040] Fig. Figure 5 illustrates an example 500 of a micro-gesture performed using the mobile device 110. This micro-gesture detection can be performed by the user for the mobile device 110 and can be detectable by one or more IMUs 112 of the mobile device 110. The micro-gesture can comprise a sequence of vibrations or other movements of the mobile device 110 over time, which can be compared and matched with a predefined sequence of vibrations or other movements of the mobile device 110 over time. If the sequences match, the micro-gesture can be considered to have been performed.

[0041] It should be noted that the IMUs 112 can be used to detect a very wide range of micro-gestures. These can include, as some non-limiting examples, swiping movements (tracked, for example, using a combination of accelerometer and gyroscope IMUs 112), tilting / rotating movements (tracked, for example, using only the gyroscope IMU 112), and / or strong jolts / impulses (tracked, for example, using only the accelerometer IMU 112). Other examples of micro-gestures could be the user gently tapping the floor with their foot (e.g., to generate vibration impulses), tapping the mobile device 110 in the user's pocket with the side of their hand, etc. Additionally or alternatively, the mobile device 110 can trigger a tactile notification, for example, to...to notify the user that they can begin performing the micro-gesture to activate the function of the vehicle 102. Additionally or alternatively, the mobile device 110 can utilize IMUs 112 contained in other externally connected devices, such as the user's fitness smartbands.

[0042] Fig. Figure 6A illustrates an example 600A of the mobile device 110, which sends a micro-gesture detection message 602 to the vehicle 102. In such an example, the mobile device 102 compares the data from the IMU 112 with the micro-gesture. In response to the mobile device 110 detecting a valid micro-gesture associated with the function of the vehicle 102, the micro-gesture detection message 602 can be sent to the vehicle 102 to inform the vehicle 102 of the detection of the micro-gesture by the mobile device 110.

[0043] Fig. Figure 6B illustrates an example 600B of vehicle 102, which detects the micro-gesture. In this variation, the mobile device 110 sends a feed of IMU data 604 to vehicle 102, and the macro-gesture is detected by vehicle 102, which processes the feed of IMU data 604. In such a case, a micro-gesture detection message 602 cannot be sent from mobile device 110 to vehicle 102.

[0044] Regardless of the approach, this valid micro-gesture, in combination with the valid macro-gesture, can request that a corresponding function of vehicle 102 be activated.

[0045] Fig. Figure 7 illustrates an example 700 of the vehicle 102, which performs the function based on the execution of the combined macro-micro gesture. In response to receiving the micro-gesture detection message 602, the vehicle 102 and / or the mobile device 110 can perform arbitration to determine whether to invoke the feature or to provide feedback to the user that the function cannot be invoked.

[0046] In one example, arbitration might involve the vehicle 102 and / or the mobile device 110 confirming that the macro-gesture and the micro-gesture were performed within a maximum permissible timeframe. For example, if the micro-gesture is performed too long after the macro-gesture, it might not be considered that the micro-gesture completes the macro-micro-gesture. In another example, arbitration might involve confirming that the requested function is capable of being performed. For example, the sensors of the vehicle 102 could be used to confirm that the requested opening of the rear door (as in Fig. (as shown in Figure 7) is not blocked by a wall, car, person, or other obstacle that could prevent the function from being performed. In response to successful arbitration, vehicle 102 can perform the feature (e.g., by vehicle 102 in response to performing the arbitration, in response to mobile device 102 requesting the feature based on mobile device 110 performing the arbitration, etc.). In response to unsuccessful arbitration, system 100 can provide feedback to the user that the function cannot be called. As shown, the macro-micro gesture is acknowledged, and the tailgate is opened.

[0047] Fig. Figure 8 illustrates an example 800 of a mobile device 110 leaving the vicinity of the vehicle 102. In this example, the PaaK system of the vehicle 102 (e.g., the UWB anchors 104 and / or the transceiver 106) can be used by the mobile device 110 and / or the vehicle 102 to detect that the mobile device 110 has changed from a stationary or stationary position near the tailgate to moving away from the vehicle 102. This macro gesture is again insufficient to decide to automatically close the tailgate, as the user may need to return to unload more cargo. Since a complete macro-micro gesture was not performed, the tailgate is not, in effect, automatically closed.

[0048] Instead, as noted in this document, a second macro-gesture detection can be performed using the IMUs 112 (and / or third-party wearable accessory sensors, such as smartbands, smartwatches, etc.) to detect micro-gestures.

[0049] In response to the detection of a micro-gesture, such as a double tap with the foot or side of the hand with the phone in your pocket, or a quick wrist turn to trigger detection on a smartwatch, the arbitration is performed again and the function can be activated. For example, if the vehicle receives the micro-gesture to close the tailgate, it can ensure that the tailgate closing process is not obstructed and then activate the tailgate closing function if necessary.

[0050] Fig. Figure 9 illustrates an exemplary process 900 for performing macro-micro gesture detection using the vehicle 102 in combination with the mobile device 110. Each macro-micro gesture can be tailored to a specific function. For example, the function to be activated can be any function of the vehicle 102, such as a door, a tailgate, a tailgate, a light, a speaker, a winch, a window, etc., which can be activated via a gesture input. Each macro-micro gesture can also be predefined or created by the user.

[0051] In process 902, a macro section of a macro-micro gesture is detected. In one example, the macro section is detected using information from UWB anchors 104 of vehicle 102 to triangulate a location of the mobile device 110 over time and compare this location with a predefined macro gesture. The macro gesture can include a sequence of locations of the mobile device 110 over time, which can be compared and matched with a predefined sequence of locations of the mobile device 110 over time. In some cases, the detection can be performed on the vehicle 102 (as in Fig. 3A), while in other cases detection can be carried out by the mobile device 110 (as in Fig. (3B shown). Nevertheless, if the sequences match, the macro gesture can be considered performed. The macro gesture can include various gestures and / or movements detectable by the PaaK system of vehicle 102, such as kicking, punching, moving back and forth, walking forwards and backwards, jumping, etc.

[0052] In process 904, a micro-segment of a macro-micro-gesture is detected. For example, micro-gesture detection can be performed by the user for the mobile device 110 and can be detected by one or more IMUs 112 of the mobile device 110. The micro-gesture can comprise a sequence of vibrations or other movements of the mobile device 110 over time, which can be compared and matched with a predefined sequence of vibrations or other movements of the mobile device 110 over time. If the sequences match, the micro-gesture can be considered to have been performed. In some cases, detection can be performed on the mobile device 110 (as in Fig. 6A), while in other cases detection can be carried out by vehicle 102 (as in Fig. 6B shown).

[0053] In process 906, the macro-micro gesture is arbitrated to validate that the function of vehicle 102 can be performed. This is done with regard to the performance of both the micro and macro sections. In one example, the arbitration might involve vehicle 102 confirming that the macro and micro gestures were performed within a maximum permissible time frame. In another example, the arbitration might involve vehicle 102 confirming that the requested function is capable of being performed, for example, that there are no obstacles preventing the function from being carried out. The arbitration can be performed by the mobile device 110 and / or by vehicle 102.

[0054] In process 908, it is determined whether the macro-micro gesture has been validated. If the gesture has been validated, the control system proceeds to process 910. Otherwise, the control system proceeds to process 912.

[0055] In operation 910, vehicle 102 performs its function. In one example, vehicle 102, which arbitrated the macro-micro gesture, calls the function. In another example, mobile device 110, which arbitrated the macro-micro gesture, sends a message to vehicle 102 to call the function. After operation 910, process 900 ends.

[0056] In operation 912, a notification is provided by mobile device 110. In one example, the notification message can inform the user of mobile device 110 that the validation was unsuccessful and the function will not be called. In one example, vehicle 102, which arbitrated the macro-micro gesture, sends the notification to mobile device 110 for display. In another example, mobile device 110, which arbitrated the macro-micro gesture, generates the notification for display. After operation 912, process 900 ends.

[0057] Various combinations of the aforementioned techniques can be used to perform process 900. For example, the controller 108, which communicates with the transceiver 106 and the plurality of UWB anchors 104, can be configured to detect, via the UWB anchors 104 of the vehicle 102, the execution of a macro-section of a macro-micro gesture by a mobile device 110, where the macro-micro gesture requests a call to a function of the vehicle 102. The transceiver 106 can send a macro-gesture detection message 402 to the mobile device 110, requesting that the mobile device 110 monitor for the execution of a micro-section of the macro-micro gesture, which is to be detected by the mobile device 110.In response to the mobile device 110 detecting the micro-section of the macro-micro gesture via data from one or more IMUs 112, the controller 108 can receive a micro-gesture detection message 602 from the mobile device 110 to the transceiver 106, indicating the execution of the micro-section of the macro-micro gesture. The controller 108 can arbitrate for the vehicle 102 to validate that the vehicle 102's function is to be performed with respect to the execution of both the micro-section and the macro-section. The controller 108 can then call the vehicle 102's function in response to successful validation.

[0058] In another example, the mobile device 110 can be configured to receive a feed 302 of UWB data from the vehicle 102 via the transceiver 106 and to detect the macro gesture based on the feed 302. The mobile device 110 can further detect the micro section of the macro-micro gestures via data from one or more IMUs 112. The mobile device 110 can perform the arbitration and can send a message to the controller 108 to invoke the function of the vehicle 102 in response to successful validation. It should be noted that these are only two variations and various other approaches consistent with the disclosure can be used.

[0059] Fig. Figure 10 illustrates an exemplary computing device 1002 for implementing the improved tracking approach to selectively shut down various UWB anchors 104 in order to conserve energy from the vehicle 102. With reference to Fig. 8 and with reference to the Fig.References 1-9, the vehicle 102, the UWB anchor 104, the transceiver 106, the controller 108, and the mobile device 110 can be examples of such computing devices 1002. Computing devices 1002 generally contain computer-executable instructions, wherein the instructions can be executed by one or more computing devices 1002. Computer-executable instructions can be compiled or interpreted by computer programs created using a variety of programming languages ​​and / or technologies, including, but not limited to, Java™, C, C++, C#, Visual Basic, JavaScript, Python, Perl, etc., either individually or in combination. In general, a processor (e.g., a microprocessor) receives instructions, e.g., from memory, a computer-readable medium, etc., and executes these instructions, thereby performing one or more processes that include one or more of the processes described herein.Such instructions and other data can be stored and transmitted using a variety of computer-readable media.

[0060] As shown, the computing device 1002 can include a processor 1004, which is operatively connected to a memory 1006, a network device 1008, an output device 1010, and an input device 1012. It should be noted that this is merely an example and computing devices 1002 can be used with more, fewer, or different components.

[0061] The 1004 processor can include one or more integrated circuits that implement the functionality of a central processing unit (CPU) and / or graphics processing unit (GPU). In some examples, the 1004 processors are a system-on-a-chip (SoC) that integrates the functionality of the CPU and GPU. The SoC can optionally include other components, such as the 1006 memory and the 1008 network device, in a single integrated device. In other examples, the CPU and GPU are interconnected via a peripheral connector, such as Peripheral Component Interconnect (PCI) Express or another suitable peripheral data connection.In one example, the CPU is a commercially available central processing device that executes a set of instructions, such as one from the x86, ARM, Power, or Microprocessor-without-Interlocked-Pipeline-Stages (MIPS) instruction set family.

[0062] Regardless of the specifics, the processor 1004 executes stored program instructions during operation, which are retrieved from the memory device 1006. The stored program instructions include software that controls the operation of the processors 1004 to perform the operations described in this document. The memory device 1006 may include both non-volatile and volatile memory devices. The non-volatile memory includes solid-state memory, such as emergency NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the system is powered off or its power supply is interrupted. The volatile memory includes static and dynamic random-access memory (RAM) that stores program instructions and data during the operation of the system 1006.

[0063] The GPU can include hardware and software for displaying at least two-dimensional (2D) and optionally three-dimensional (3D) graphics on the output device 1010. The output device 1010 can include a graphical or visual display device, such as an electronic display screen, a projector, a printer, or any other suitable device that reproduces a graphical display. As another example, the output device 1010 can include an audio device, such as a loudspeaker or headphones. As yet another example, the output device 1010 can include a tactile device, such as a mechanically liftable device, which in one example can be configured to display Braille or other physical output that can be touched to provide information to a user.

[0064] An input device 1012 can include any of the various devices that enable the computing device 1002 to receive control inputs from users. Examples of suitable input devices 1012 that receive inputs via a human interface can include keyboards, mice, trackballs, touchscreens, microphones, graphics tablets, and the like.

[0065] The network devices 1008 can each include any of the various devices that enable the described components to send and / or receive data from external devices over networks. Examples of suitable network devices 1008 include an Ethernet interface, a Wi-Fi transceiver, a cellular transceiver, a Bluetooth or BLE transceiver, or any other network adapter or peripheral connection device that receives data from another computer or external storage device, which can be useful for efficiently receiving large datasets.

[0066] With regard to the processes, systems, procedures, heuristics, etc., described in this document, it is understood that although the steps of such processes, etc., have been described as occurring according to a specific, ordered sequence, such processes could be implemented in practice by performing the described steps in a sequence that differs from the sequence described herein. Furthermore, it is understood that certain steps could be performed simultaneously, other steps added, or certain steps described herein omitted. In other words, the descriptions of processes herein serve the purpose of illustrating certain embodiments and should in no way be interpreted as limiting the patent claims.

[0067] Accordingly, it is understood that the foregoing description is intended to be illustrative and not limiting. Many other embodiments and applications beyond the examples provided will become apparent from reading the preceding description. The scope should not be determined by reference to the foregoing description, but instead by reference to the attached claims, together with the full scope of equivalents to which these claims entitle. It is anticipated and intended that there will be future developments in the prior art discussed in this document and that the disclosed systems and methods will be incorporated into such future embodiments. Overall, it is understood that the application may be modified and varied.

[0068] All terms used in the claims shall be assigned their most comprehensive and comprehensible constructions and their general meanings as they would be known to persons skilled in the art in the art of the techniques described herein, unless expressly stated otherwise. In particular, the use of singular articles such as "a", "an", "the", "a", etc., shall be understood as referring to one or more of the elements indicated, unless a claim expressly limits this to the contrary.

[0069] The summary of disclosure is provided to enable the reader to quickly grasp the nature of the technical disclosure. It is submitted on the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, it is evident from the preceding detailed description that, for the purpose of simplifying the disclosure, various features in different embodiments have been grouped together. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly mentioned in each claim. Rather, as reflected in the following claims, the subject matter of the invention consists of fewer than all the features of any single disclosed embodiment.The following patent claims are hereby included in the detailed description, each patent claim being a separately claimed subject matter.

[0070] While exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms of the disclosure. Rather, the terms used in the description are descriptive rather than limiting, and it is understood that various modifications can be made without departing from the spirit and scope of the disclosure. Furthermore, the features of different implementing embodiments can be combined to form further embodiments of the disclosure.

[0071] According to the present invention, a method for macro-micro gesture detection comprises the following: detecting, using phone-as-a-key sensors (PaaK sensors) of a vehicle, the execution of a macro-section of a macro-micro gesture by a mobile device, wherein the macro-micro gesture requests a call to a function of the vehicle; detecting, using one or more sensors of an inertial measurement unit (IMU) sensors of the mobile device, the execution of a micro-section of the macro-micro gesture; arbitrating to validate that the function of the vehicle is to be executed with respect to the execution of both the micro-section and the macro-section; and calling the function of the vehicle in response to the successful validation.

[0072] In one aspect of the invention, the method includes the following: receiving, from the mobile device to the vehicle, a feed of data from the one or more IMU sensors of the mobile device; and detecting the execution of the micro-section of the macro-micro gesture by the vehicle.

[0073] In one aspect of the invention, the method includes the following: detecting the execution of the micro-section of the macro-micro gesture by the mobile device; and in response to the mobile device detecting the micro-section of the macro-micro gesture, sending, from the mobile device to the vehicle, a micro-gesture detection message.

[0074] In one aspect of the invention, the method includes the following: sending data from the vehicle's PaaK sensors to the mobile device; and detecting the execution of the macro section of the macro-micro gesture by the mobile device.

[0075] In one aspect of the invention, the method includes the following: detecting the execution of the macro section of the macro-micro gesture by the vehicle; and in response to the vehicle detecting the macro section of the macro-micro gesture, sending, by the vehicle to the mobile device, a macro-gesture transmission detection message.

[0076] In one aspect of the invention, arbitration involves confirming that both the micro-section and the macro-section occur within a predefined period.

[0077] In one aspect of the invention, the detection of the execution of the macro section and the detection of the execution of the macro section are performed simultaneously or overlapping.

[0078] In one aspect of the invention, arbitration involves sending a notification message to the mobile device to inform it that the validation was unsuccessful and the function will not be called, in response to the macro section not occurring within a predefined time period.

[0079] In one aspect of the invention, arbitration involves sending a notification message to the mobile device to inform it that the function will not be called, in response to vehicle sensors indicating that the function cannot be performed.

[0080] In one aspect of the invention, the microsection is detected at least partially using one or more IMU sensors of wearables in wireless communication with the mobile device.

[0081] In one aspect of the invention, the method involves sending, from the vehicle to the mobile device, a definition of the micro section of the macro-micro gesture adapted for the mobile device.

[0082] In one aspect of the invention, the method involves maintaining, by means of the mobile device, a definition of the micro-section of the macro-micro gesture to be detected.

[0083] In one aspect of the invention, the method includes the following: detecting, by the vehicle's PaaK sensors, the execution of the macro-section of the macro-micro gesture by the mobile device, wherein the macro-micro gesture requests the invocation of the vehicle's function; sending, from a vehicle transceiver to the mobile device, a macro-gesture detection message requesting that the mobile device monitor the execution of the micro-section of the macro-micro gesture to be detected by the mobile device; receiving, from the mobile device to the transceiver, in response to the mobile device detecting the micro-section of the macro-micro gesture, a micro-gesture detection message indicating the execution of the micro-section of the macro-micro gesture; and arbitrating, by the vehicle, to validate,that the function of the vehicle is to be carried out with regard to the execution of both the micro-section and the macro-section, and calling the function of the vehicle in response to the successful validation.

[0084] According to the present invention, a macro-micro gesture detection system is provided, comprising: a vehicle transceiver; a plurality of ultra-wideband (UWB) anchors;and a controller in communication with the transceiver and the multitude of UWB anchors, configured to: detect, using the UWB anchors, the execution of a macro-section of a macro-micro gesture by a mobile device, wherein the macro-micro gesture requests a call to a function of the vehicle; detect a micro-section of the macro-micro gesture using one or more sensors of an inertial measurement unit (IMU) of the mobile device; arbitrate, by the vehicle, to validate that the function of the vehicle is to be executed with respect to the execution of both the micro-section and the macro-section; and call the function of the vehicle in response to the validation being successful.

[0085] According to one embodiment, the controller is further configured to perform one or more of the following: detecting the execution of the micro-section of the macro-micro gesture by the vehicle based on a feed of data from the one or more IMU sensors received by the mobile device; and / or detecting the execution of the micro-section of the macro-micro gesture by receiving, from the mobile device, a micro-gesture detection message indicating the execution of the micro-section of the macro-micro gesture as detected by the mobile device.

[0086] According to one embodiment, the controller is further configured to do one or more of the following: send data from the vehicle's UWB anchors to the mobile device to cause the mobile device to detect the execution of the macro section of the macro-micro gesture; and / or detect the execution of the macro section of the macro-micro gesture by the controller, which analyzes the data from the UWB anchors.

[0087] According to one embodiment, one or more of the following: the arbitration includes confirming that both the micro-section and the macro-section occur within a predefined time period; and / or detecting the execution of the macro-section and detecting the execution of the macro-section are performed simultaneously or overlapping.

[0088] According to one embodiment, the arbitration includes one or more of the following: sending a notification message to be displayed by the mobile device to inform that the validation was unsuccessful and the function will not be called in response to the macro section not occurring within a predefined time period; and / or generating and displaying the notification message by the mobile device to inform that the validation was unsuccessful and the function will not be called.

[0089] According to one embodiment, the controller is further configured to send from the vehicle to the mobile device a definition of the micro section of the macro-micro gesture adapted for the mobile device.

[0090] According to the present invention, a non-transient, computer-readable medium is provided, comprising instructions which, when executed by a vehicle controller in communication with a transceiver and a plurality of UWB anchors, cause the controller to perform operations, including: detecting, using the vehicle's UWB anchors, the execution of a macro-section of a macro-micro gesture by a mobile device, wherein the macro-micro gesture requests a call to a function of the vehicle; detecting a micro-section of the macro-micro gesture using one or more sensors of an inertial measurement unit (IMU) of the mobile device; arbitrating, by the vehicle, to validate that the vehicle's function is to be executed with respect to the execution of both the micro-section and the macro-section;and calling up the vehicle's function in response to the successful validation.

Claims

[1] Methods for macro-micro gesture detection, comprising: Detecting, using a vehicle's Phone-as-a-Key (PaaK) sensors, the execution of a macro section of a macro-micro gesture by a mobile device, wherein the macro-micro gesture requests a call to a function of the vehicle; Detecting, using one or more sensors of an inertial measurement unit (IMU) of the mobile device, a performance of a micro-section of the macro-micro gesture; Arbitrate to validate that the vehicle's function can be performed with regard to the execution of both the micro-section and the macro-section; and Calling up the vehicle's function in response to a successful validation. [2] Method for macro-micro gesture detection according to claim 1, further comprising: Receiving, from the mobile device to the vehicle, a feed of data from the one or more IMU sensors of the mobile device; and Detecting the execution of the micro-section of the macro-micro gesture by the vehicle. [3] Method for macro-micro gesture detection according to claim 1, further comprising: Detecting the execution of the micro-segment of the macro-micro gesture by the mobile device; and In response to the mobile device detecting the micro section of the macro-micro gesture, the mobile device sends a micro-gesture detection message to the vehicle. [4] Method for macro-micro gesture detection according to claim 1, further comprising: Sending data from the vehicle's PaaK sensors to the mobile device; and Detecting the execution of the macro section of the macro-micro gesture by the mobile device. [5] Method for macro-micro gesture detection according to claim 1, further comprising: Detecting the execution of the macro-section of the macro-micro gesture by the vehicle; and In response to the vehicle detecting the macro section of the macro-micro gesture, the vehicle sends a macro-gesture detection message to the mobile device. [6] Method for macro-micro gesture detection according to claim 1, wherein the arbitration includes confirming that both the micro section and the macro section occur within a predefined period of time. [7] Method for macro-micro gesture detection according to claim 1, wherein the detection of the execution of the macro section and the detection of the execution of the macro section are carried out simultaneously or overlapping. [8] Method for macro-micro gesture detection according to claim 1, wherein the arbitration includes sending a notification message to the mobile device to inform that the validation was unsuccessful and the function will not be called, in response to the macro section not occurring within a predefined period of time. [9] Method for macro-micro gesture detection according to claim 1, wherein the arbitration includes sending a notification message to the mobile device to inform that the function is not being called, in response to vehicle sensors indicating that the function cannot be performed. [10] Method for macro-micro gesture detection according to claim 1, wherein the micro section is detected at least partially using one or more IMU sensors of wearables in wireless communication with the mobile device. [11] Method for macro-micro gesture detection according to claim 1, further comprising sending, from the vehicle to the mobile device, a definition of the micro section of the macro-micro gesture adapted for the mobile device. [12] Method for macro-micro gesture detection according to claim 1, further comprising maintaining, by the mobile device, a definition of the micro section of the macro-micro gesture to be detected. [13] Method for macro-micro gesture detection according to claim 1, further comprising: Detecting, through the vehicle's PaaK sensors, the execution of the macro section of the macro-micro gesture by the mobile device, wherein the macro-micro gesture requests the invocation of the vehicle's function, Sending, from a transceiver of the vehicle to the mobile device, a macro-gesture detection message requesting that the mobile device monitor the execution of the micro-section of the macro-micro-gesture, which is to be detected by the mobile device, Received from the mobile device to the transceiver in response to the mobile device detecting the micro-section of the macro-micro gesture, a micro-gesture transmission detection message indicating the execution of the micro-section of the macro-micro gesture, Arbitrate, by the vehicle, to validate that the vehicle's function is to be carried out with regard to the execution of both the micro-section and the macro-section, and Calling up the vehicle's function in response to a successful validation. [14] Macro-micro gesture detection system, comprising: a vehicle transceiver; a large number of ultra-wideband (UWB) anchors; and a controller in communication with the transceiver and the plurality of UWB anchors, configured to perform the method according to one of claims 1-13. [15] Non-transient computer-readable medium comprising instructions which, when executed by a vehicle control system in communication with a transceiver and a plurality of UWB anchors, cause the control system to carry out the method according to any one of claims 1-13.

Images