Smart device based radar system for vehicle operation

Abstract

Techniques and apparatuses are described that implement a smart device-based radar system (104) that is capable of determining characteristics of objects outside a vehicle (408), occupants within a vehicle (506, 508), and objects proximate to an outdoor vehicle (608, 610). In particular, the system enables a smart device (102) to perform a number of vehicle operations, such as collision avoidance, occupant detection, and parking assistance, in vehicle (100-1, 100-2) and outdoor vehicle (100-3) environments without integrated radar technology. By using a smart device (102) to perform such actions, existing vehicles and outdoor vehicles that do not have integrated radar functionality can be able to take advantage of radar-based vehicle operations.

Description

Background Art

[0001] Implementing radar technology in vehicle environments has enabled many forms of enhanced driver experience and increased vehicle safety. While some vehicle manufacturers are building radar-based systems into their vehicles, these systems are generally limited to newer or more expensive models, which limits access for many vehicle buyers. Furthermore, while some vehicles may be retrofittable with radar sensors, processors, and displays, the integration of such equipment is typically expensive, complex, and time-consuming. Therefore, there is a need to easily enable a greater percentage of drivers to benefit from radar technology in vehicle environments where the technology is not present or lacks functionality. Summary of the Invention

[0002] Techniques and apparatus are described for implementing a smart device-based radar system for vehicle operations. Specifically, the system enables smart devices to perform radar-enabled vehicle operations such as collision avoidance, occupant detection, and parking assistance. By using smart devices to perform such actions, existing vehicle environments (e.g., cars, trucks, motorcycles, boats, airplanes, bicycles, or outdoor vehicles) can benefit from radar-based vehicle operations, even if those existing vehicles do not have an integrated radar-based system.

[0003] Aspects described below include an apparatus for radar-based vehicle operations, the apparatus comprising a processor and a computer-readable storage medium containing instructions that cause the apparatus to transmit a radar transmit signal from an interior of a vehicle via a radar system of the device, receive a radar receive signal comprising a reflection of the radar transmit signal from an object external to the vehicle, and initiate an operation based on characteristics of the object determined from the radar receive signal.

[0004] Aspects described below also include an apparatus for radar-based vehicle operations, the apparatus comprising a processor and a computer-readable storage medium containing instructions that cause the apparatus to transmit a radar transmit signal via a radar system of the apparatus through a display screen of the apparatus, receive a radar receive signal comprising a reflection of the radar transmit signal from an occupant of the vehicle, and initiate an operation based on a determined activity of the occupant determined from the radar receive signal.

[0005] Aspects described below also include an apparatus for radar-based vehicle operations, the apparatus comprising a processor and a computer-readable storage medium containing instructions that cause the apparatus to transmit a radar transmit signal when the device is mounted on an outdoor vehicle, receive a radar receive signal comprising reflections of the radar transmit signal from an object in the vicinity of the outdoor vehicle, and initiate an operation based on characteristics of the object determined from the radar receive signal.

[0006] Aspects described below also include a method for performing radar-based vehicle operations, wherein an apparatus transmits a radar transmit signal from an interior of a vehicle via a radar system within the apparatus, receives a radar receive signal comprising a reflection of the radar transmit signal from an object external to the vehicle, and initiates an operation based on characteristics of the object determined from the radar receive signal.

[0007] Aspects described below also include a method of performing radar-based vehicle operations, wherein a device transmits a radar transmit signal via a radar system of the device through a display screen of the device, receives a radar receive signal including reflections of the radar transmit signal from an occupant of the vehicle, and initiates an operation based on a determined activity of the occupant determined from the radar receive signal.

[0008] Aspects described below also include a method of performing radar-based vehicle operations, wherein a device transmits a radar transmit signal while the device is mounted on an outdoor vehicle, receives a radar receive signal comprising reflections of the radar transmit signal from an object in the vicinity of the outdoor vehicle, and initiates an operation based on characteristics of the object determined from the radar receive signal.

[0009] Aspects described below include a computer-readable storage medium containing instructions that cause a processor to cause transmission of a radar transmit signal from an interior of a vehicle via a radar system, receive, via the radar system, a radar receive signal comprising a reflection of the radar transmit signal from an object exterior to the vehicle, and initiate an operation based on characteristics of the object determined from the radar receive signal.

[0010] Aspects described below include a computer-readable storage medium containing instructions that cause a processor to cause transmission of a radar transmit signal via a radar system of the device through a display screen of the device, receive a radar receive signal comprising a reflection of the radar transmit signal from an occupant of a vehicle, and initiate an operation based on a determined activity of the occupant determined from the radar receive signal.

[0011] Aspects described below also include a computer-readable storage medium containing instructions that cause a processor to cause a radar transmit signal to be transmitted while the apparatus is being mounted on an outdoor vehicle, receive a radar receive signal comprising a reflection of the radar transmit signal from an object in the vicinity of the outdoor vehicle, and initiate an operation based on characteristics of the object determined from the radar receive signal.

[0012] The aspects described below also include means for transmitting a radar transmit signal from the interior of a vehicle, means for receiving a radar receive signal comprising reflections of the radar transmit signal from an object external to the vehicle, and means for initiating an action based on characteristics of the object determined from the radar receive signal.

[0013] The aspects described below also include means for transmitting a radar transmit signal through a display screen of a device, means for receiving a radar receive signal comprising the radar transmit signal reflected from an occupant of a vehicle, and means for initiating an action based on a determined activity of the occupant determined from the radar receive signal.

[0014] The aspects described below also include means for transmitting a radar transmit signal when the device is mounted on an outdoor vehicle, means for receiving a radar receive signal comprising reflections of the radar transmit signal from an object near the outdoor vehicle, and means for initiating an operation based on characteristics of the object determined from the radar receive signal.

[0015] Further to the above and below descriptions, a user may be provided with controls that allow the user to choose whether and when the systems, programs, or features described herein may be able to collect user information (e.g., information about the user's social network, social actions, social activities, occupation, the user's preferences, the user's current location, speed, and heading, information about objects in or around the vehicle, the type or model of the vehicle, the number of passengers, the type of passengers, or health information about the passengers) and whether to send content or communications from a server to the user. In addition, before storing or using specific data, the specific data may be processed in one or more ways so that personally identifiable information can be removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location, speed, or heading from which location information (such as a city, zip code, or state and county level) can be generalized so that the user's specific location cannot be determined. Thus, the user can control what information about the user is collected, how the information is used, and what information is provided to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Devices and techniques for implementing a smart device-based radar system capable of vehicle operation are described with reference to the following drawings. Throughout the drawings, like reference numerals are used to indicate like features and components:

[0017] Figure 1 An example environment in which a smart device-based radar system capable of vehicle operation can be implemented is illustrated.

[0018] Figure 2 An example implementation of a radar system as part of a smart device is illustrated.

[0019] Figure 3 Example operations of a radar system for transmitting through a display screen of a smart device and / or through a windshield of a vehicle are illustrated.

[0020] Figure 4 An example implementation of a smart device-based radar system for vehicle operation corresponding to objects external to the vehicle is illustrated.

[0021] Figure 5 An example implementation of a smart device-based radar system for vehicle operation corresponding to occupants within the vehicle is illustrated.

[0022] Figure 6 An example implementation of a smart device-based radar system for vehicular operations corresponding to objects in the vicinity of the vehicle outdoors is illustrated.

[0023] Figure 7 An example method for operating a smart device-based radar system capable of detecting information about objects through a vehicle's windshield is illustrated.

[0024] Figure 8 An example method for operating a smart device-based radar system capable of detecting information about objects external to a vehicle is illustrated.

[0025] Figure 9 An example method for operating a smart device-based radar system capable of detecting information about occupants within a vehicle is illustrated.

[0026] Figure 10 An example method for operating a smart device-based radar system capable of detecting information about objects in the vicinity of an outdoor vehicle is illustrated. DETAILED DESCRIPTION

[0027] Overview

[0028] Radar technology enables many vehicle / driver assistance operations in certain types of vehicles, such as parking assistance, collision avoidance, driver monitoring, autonomous driving, and route guidance. To perform some of these operations, conventional radar systems are typically customized for the vehicle, including radar components that are fixed relative to the vehicle (e.g., a radar sensor mounted in the front bumper). Generally, these conventional systems cannot be implemented in a different environment (e.g., another vehicle) and do not work effectively when the conventional system is moving relative to the vehicle.

[0029] Implementing radar-enhanced vehicle operations using a smart device-based radar system presents numerous challenges, most of which are not encountered in traditional radar implementations. For example, the radar system may need to generate a radar transmit signal that can penetrate a windshield. Furthermore, the system may need to ignore portions of the vehicle within the radar's transmission field that are not necessary for a particular operation (e.g., reflections off the windshield or interior trim). In some cases, the smart device may need to determine the system's position relative to the vehicle so that meaningful data can be generated.

[0030] Thus, the apparatus and techniques described herein provide smart device-based radar systems and methods capable of detecting features of objects external to a vehicle, occupants within a vehicle, and objects near an outdoor vehicle (e.g., a bicycle or motorcycle). These detected features enable radar-enhanced vehicle operations via a radar-enabled smart device.

[0031] Sample Environment

[0032] Figure 1 is an illustration of an example environment in which a smart device-based radar system capable of detecting characteristics of objects external to a vehicle, occupants within a vehicle, and objects outdoors near the vehicle's side may be embodied. In the depicted environments 100-1, 100-2, and 100-3, a smart device 102 includes a radar system 104 capable of characterizing objects external to a vehicle (100-1), occupants within a vehicle (100-2), and objects outdoors near the vehicle's side (100-3). Although Figure 1 The smart device 102 is shown as a smart phone, but the smart device 102 can be implemented as any suitable computing or electronic device, such as Figure 2 described in further detail.

[0033] In environments 100-1 through 100-3, various characteristics of objects / occupants within the transmission field of radar system 104 are detected by radar system 104. For example, environment 100-1 illustrates the detection of characteristics of objects outside a vehicle, such as another vehicle approaching. Environment 100-2 illustrates the detection of characteristics of occupants within a vehicle, such as the driver's activity or attention level. Environment 100-3 illustrates the detection of characteristics of objects near the vehicle outdoors, such as an approaching tree or vehicle.

[0034] Some embodiments of the radar system 104 are particularly advantageous when applied in the context of a smart device 102, for which a convergence of issues exists. This can include the need to limit the spacing and placement of the radar system 104, as well as low power. Exemplary overall lateral dimensions of the smart device 102 can be, for example, approximately eight centimeters by approximately fifteen centimeters. An exemplary coverage area of ​​the radar system 104 can be even more limited, such as approximately four millimeters by six millimeters including the antenna. An exemplary power consumption of the radar system 104 can be on the order of a few milliwatts to tens of milliwatts (e.g., between approximately two milliwatts and twenty milliwatts). This limited coverage area and power consumption requirement for the radar system 104 enables the smart device 102 to include other desired features (e.g., camera sensors, fingerprint sensors, displays, etc.) in a space-constrained package. In traditional radar implementations in a vehicular environment, these features are generally of little concern. Figure 2 The smart device 102 and the radar system 104 are further described.

[0035] Figure 2 Radar system 104 is shown as part of smart device 102. Smart device 102 can be any suitable computing or electronic device, such as a tablet, laptop, smartphone, smartwatch, or wearable device. Smart device 102 can be wearable or non-wearable but mobile. Radar system 104 can be used as a standalone radar system (e.g., a peripheral device) or embedded in smart device 102.

[0036] The smart device 102 includes at least one computer processor 202 and at least one non-transitory computer-readable medium 204. The computer-readable medium 204 includes memory media and storage media. An operating system (not shown) and at least one application embodied as computer-readable instructions on the computer-readable medium 204 can be executed by the computer processor 202 to provide some of the functionality described herein. For example, the computer-readable medium 204 includes at least one radar-based application 206 that uses radar data generated by the radar system 104 to perform the functionality described herein, such as characterizing objects outside the vehicle, occupants of the vehicle, or objects near the vehicle outdoors.

[0037] The smart device 102 also includes a network interface 208 for transmitting data via a wired, wireless, or optical network. For example, the network interface 208 transmits data via a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), a wired local area network (WAN), an intranet, the Internet, a peer-to-peer network, a point-to-point network, a mesh network, etc. The smart device 102 may also include a display, a speaker, or a tactile feedback device (not shown).

[0038] Radar system 104 includes communication interface 210 to transmit radar data to smart device 102, but communication interface 210 may not be required if radar system 104 is integrated within smart device 102. Typically, the radar data provided by communication interface 210 is in a format usable by radar-based application 206.

[0039] The radar system 104 also includes at least one antenna 212 and at least one transceiver 214 to transmit and receive radar signals. The antenna 212 can be circularly polarized, horizontally polarized, or vertically polarized. In some cases, the radar system 104 includes multiple antennas 212 implemented as antenna elements of an antenna array. The antenna array can include at least one transmitting antenna element and at least two receiving antenna elements. In some cases, the antenna array includes multiple transmitting antenna elements to implement a multiple-input multiple-output (MIMO) radar that can transmit multiple different waveforms (e.g., different waveforms per transmitting antenna element) at a given time. For embodiments including three or more receiving antenna elements, the receiving antenna elements can be positioned in a one-dimensional shape (e.g., a line) or a two-dimensional shape (e.g., a triangle, rectangle, or L-shape). The one-dimensional shape enables the radar system 104 to measure one angular dimension (e.g., azimuth or elevation), while the two-dimensional shape enables measurement of two angular dimensions (e.g., azimuth and elevation).

[0040] By using antenna arrays, the radar system 104 is able to form a beam that is steered or non-steered, wide or narrow, or shaped (e.g., a hemisphere, cube, sector, cone, or cylinder). By utilizing multiple arrays (e.g., one facing forward and one facing backward), a complete sphere can be generated, enabling the radar system 104 to detect objects in all directions relative to the smart device 102. The transmitting antenna elements can have a non-steered omnidirectional radiation pattern or can be capable of generating a wide steerable beam. Either of these technologies enables the radar system 104 to illuminate a large amount of space with radar signals. To achieve target angular accuracy and angular resolution, the receiving antenna elements can be used to generate thousands of narrow steered beams (e.g., 2000 beams, 4000 beams, or 6000 beams) using digital beamforming. In this way, the radar system 104 can efficiently monitor the external environment and characterize objects.

[0041] The transceiver 214 includes circuitry and logic for transmitting and receiving radar signals via the antenna 212. Components of the transceiver 214 can include amplifiers, mixers, switches, analog-to-digital converters, filters, and the like for conditioning the radar signal. The transceiver 214 also includes logic for performing in-phase / quadrature (I / Q) operations, such as modulation or demodulation. Various modulations can be used to generate radar signals, including linear frequency modulation, triangular frequency modulation, stepped frequency modulation, or phase modulation. The transceiver 214 can be configured to support continuous wave or pulsed radar operation.

[0042] The spectrum (e.g., the range of frequencies) that transceiver 214 can use to generate radar signals can cover frequencies between 1 and 400 gigahertz (GHz), between 1 and 24 GHz, between 2 and 6 GHz, between 4 and 100 GHz, or between 57 and 63 GHz. In some cases, the spectrum can be divided into multiple sub-spectra with similar or different bandwidths. Example bandwidths can be on the order of approximately 500 megahertz (MHz), 1 GHz, 2 GHz, etc. The different frequency sub-spectra can include, for example, frequencies between approximately 57 and 59 GHz, between 59 and 61 GHz, or between 61 and 63 GHz. Although the example frequency sub-spectra described above are continuous, other frequency sub-spectra may not be continuous. To achieve coherence, transceiver 214 can use multiple frequency sub-spectra (continuous or discontinuous) with the same bandwidth to generate multiple radar signals, which are transmitted simultaneously or separated in time. In some cases, multiple continuous frequency sub-spectra can be used to transmit a single radar signal, thereby enabling the radar signal to have a wide bandwidth.

[0043] The radar system 104 may also include its own dedicated system processor (not shown) and / or its own dedicated system media (also not shown). For example, the dedicated system processor may be separate from the transceiver 214 or may be implemented as a digital signal processor or a low-power processor within the transceiver 214. The dedicated system processor may execute computer-readable instructions stored within the dedicated system media or computer-readable media 204. Example digital operations performed by the dedicated system processor may include fast Fourier transforms (FFTs), filtering, modulation or demodulation, digital signal generation, digital beamforming, etc. Figure 3 Radar system 104 is further described.

[0044] Figure 3The diagram illustrates additional details of an example implementation 300 of a radar system 104 within a smart device 102. In this example, antenna 212 is positioned beneath the housing of electronic device 102, such as a glass cover and / or an external casing. Furthermore, as discussed below, radar signals transmitted by antenna 212 may also need to penetrate the windshield or display screen of smart device 102. Depending on their material properties, these components may act as attenuators, attenuating or distorting radar signals transmitted and received by radar system 104. Attenuator 302 represents any material or combination of materials (e.g., a housing, a casing, a display screen, and / or a windshield) that transmissions from antenna 212 may need to penetrate. Attenuator 302 may include various materials having corresponding dielectric constants (e.g., relative permittivity) between approximately 4 and 10. Thus, attenuator 302 is opaque or translucent to radar signal 306 and may cause a portion of transmitted or received radar signal 306 to be reflected (as shown by reflected portion 304). For conventional radars, attenuator 302 may reduce the effective range that can be monitored, prevent small targets from being detected, or reduce overall accuracy.

[0045] Assuming that the transmit power of radar system 104 is limited and redesigning attenuator 302 is undesirable or impossible, one or more attenuation-related characteristics of radar signal 306 (e.g., frequency sub-spectrum 308 or steering angle 310) can be adjusted to mitigate the effects of attenuator 302. Additionally, some attenuation-related features of portions of attenuator 302 (e.g., distance 312 between components of attenuator 302 and radar system 104 or thickness 314 of a portion of attenuator 302) can be adjusted to mitigate the effects of attenuator 302. Some of these features can be set during manufacturing, while some may need to be adjusted by an attenuation mitigator (not shown) during operation of radar system 104.

[0046] For example, the attenuation mitigator can cause the transceiver 214 to transmit the radar signal 306 using the selected frequency sub-spectrum 308 or steering angle 310, cause the platform to move the radar system 104 closer to or further away from a portion of the attenuator 302 (e.g., to effect a change in distance 312), or prompt the user to change an aspect of the attenuator 302 (e.g., to effect a change in thickness 314). Appropriate adjustments can be made by the attenuation mitigator based on predetermined characteristics of the attenuator 302 (e.g., characteristics stored in the computer-readable medium 204 or dedicated system memory) or by processing the return of the radar signal 306 to measure one or more characteristics of the attenuator 302. In the case of a vehicle environment, each windshield may have different attenuation characteristics. Furthermore, different locations of the device 102 relative to the same windshield may result in different attenuation characteristics. Even if some attenuation-related characteristics are fixed or constrained, the attenuation mitigator can account for these constraints to balance each parameter and achieve the desired radar performance. Thus, the attenuation mitigator enables the radar system 104 to achieve enhanced accuracy and a greater effective range for detecting and tracking objects located on the opposite side of any object (e.g., a housing, display, and / or windshield) acting as the attenuator 302. These techniques provide an alternative to increasing transmit power (which may not be possible) and, therefore, increasing the power consumption of the radar system 104. These techniques also prevent the need to change the material properties of the attenuator 302, which in the case of smart device materials can be difficult and expensive once the device is in production, or in the case of a windshield, which can be nearly impossible (windshield properties must comply with various certifications and regulations).

[0047] External object detection

[0048] Figure 4 An example implementation of a smart device 102 for radar-based vehicle operations corresponding to objects external to the vehicle is illustrated.

[0049] During operation, smart device 102 generates and provides radar transmit signal 402. Radar transmit signal 402 is, for example, a continuous wave frequency modulated signal generated by radar system 104. Radar transmit signal 402 strikes internal objects, such as windshield trim 404 and windshield 406, and external objects, such as vehicle 408. Consequently, a radar receive signal may be received that includes radar reflections 410, 412, and 414, which are caused by reflections of radar transmit signal 402 from windshield trim 404, windshield 406, and vehicle 408, respectively. Each of radar reflections 410, 412, and 414 includes at least a portion of radar transmit signal 402. An internal reflection refers to any radar signal that is reflected from a portion of a vehicle (e.g., radar reflections 410 and 412). Although radar reflections 410 and 412 have been reflected from portions of the interior of the vehicle, internal reflections may also come from portions of the exterior of the vehicle, such as side mirrors, hood emblems, hoods, or fenders. External reflections refer to any radar signal that is reflected from an object that is not part of the vehicle (e.g., radar reflection 414).

[0050] although Figure 4 A single external object (e.g., vehicle 408) is shown, but the system can distinguish and track multiple external objects so that operations such as collision avoidance and parking assistance can be performed based on the determined characteristics of one or more of the external objects. For example, in a parking assistance operation, the range of the vehicle in front and the sides of the vehicle can be tracked to help avoid collisions. In addition, although the smart device 102 is shown as being mounted in a "portrait" orientation and offset to the left on the dashboard, the following techniques are applicable to any position and orientation, as long as the object to be sensed (e.g., vehicle 408) reflects the radar transmit signal 402 back to the smart device 102 in a given position.

[0051] To determine the characteristics of external objects, the position of smart device 102 (relative to the vehicle's position and orientation) is calculated. Smart device 102 can analyze the received radar signal to distinguish internal reflections (e.g., radar reflections 410 and 412) from external reflections (e.g., radar reflection 414). For example, if the vehicle is moving, the internal reflections have a very small relative velocity compared to the external reflections. Smart device 102 can then analyze the internal reflections to determine a spatial representation of the vehicle's interior relative to smart device 102, such as a view of the vehicle's interior from the perspective of smart device 102. Based on the known characteristics of the vehicle and the spatial representation of the vehicle's interior relative to the device, the position of smart device 102 relative to the vehicle device can be determined. This determination can be made by comparing the spatial representation with the known characteristics, calculating known landmarks within the vehicle, other distances, etc.

[0052] The known characteristics of the vehicle may include a vehicle map, including interior and exterior maps, enabling determination of the relative location of the vehicle's exterior range based on the known interior location. In one example, a user performs a vehicle setup process to map a specific vehicle. In another example, a vehicle spatial map is retrieved from a vehicle spatial mapping library. Once the location of the smart device 102 relative to the vehicle is known, the vehicle's range and direction relative to the vehicle can be determined by the smart device 102.

[0053] Other sensors, such as accelerometers, gyroscopes, imagers, and proximity sensors, may be utilized alone or in combination with internal reflections to determine the position of the smart device 102 relative to the vehicle.

[0054] To maximize the detectability of external objects, internal reflections, such as those from windshield 404, can also be used for attenuation mitigation. For example, radar reflections 412 can be used to adjust future outgoing radar transmit signals, as described above with respect to attenuation mitigators. In this way, as the position of the smart device changes relative to the windshield, the radar transmit signal can be adjusted to compensate for this change. Furthermore, smart device 102 can utilize the position and / or information from radar reflections 412 to beamform radar transmit signal 402 in a meaningful direction, such as forward from windshield 404.

[0055] The smart device can periodically determine its position relative to the vehicle. This periodic determination helps the smart device update attenuation mitigation and / or update its position for external object detection (discussed below). In this way, the device can move within the vehicle without having a substantial negative impact on the performance of the system.

[0056] Certain characteristics of external objects can be determined without first determining the location of the smart device 102. For example, characteristics such as size, relative speed, and distance from the smart device 102 to the external object can be determined without knowing the location of the smart device 102. However, as discussed above, to determine the direction and distance of the external object relative to the vehicle, the smart device 102 utilizes the known location of the smart device 102 and known aspects of the vehicle.

[0057] Other characteristics of external objects can be determined based on the determined position of smart device 102 relative to the vehicle, and therefore the location of the vehicle's range relative to smart device 102. External radar reflections (e.g., radar reflection 414) can be analyzed to determine the location, distance, size, identity, and speed of external objects relative to the vehicle. To do this, certain characteristics of the object, such as distance and direction, can be compared with the distance and direction of the vehicle's range (determined above) to determine the distance and direction relative to the vehicle. For example, the operator of a vehicle often wishes to know from which direction an object is approaching in order to make the information useful. For example, when parking, it is useful to know that an object near the right front corner of the vehicle is "approaching" so that steering inputs can be corrected. Since the vehicle map contains information about the vehicle's external range (e.g., bumper location, mirror location, exterior outline / footprint), and the smart device has determined the range location and the location of the external object, the object's heading and distance can be determined.

[0058] The smart device 102 can then utilize the determined characteristics of the external objects (with or without the known location of the smart device 102) to perform a function. For example, the function may involve displaying the identification of one or more of the external objects on the device's display screen, along with the object's distance, direction, and / or relative speed. This information may be displayed relative to the vehicle's "surround view" or "driver's view," or some other representation relative to the vehicle. In some embodiments, a warning or notification may be displayed based on the determined characteristics of the external objects. For example, if the smart device 102 determines that a collision is imminent, a notification may be presented to alert the driver (visually, audibly, tactilely, etc.). The smart device 102 may also utilize onboard sensors as part of its operation. For example, a GPS system may be able to determine the vehicle's speed and adjust thresholds for collision avoidance maneuvers based on the vehicle's determined speed. The smart device 102 may also be able to communicate with the vehicle to determine speed, direction, steering input, braking input, throttle input, yaw, pitch, roll, or any other data available at the vehicle.

[0059] For parking assist maneuvers, the location of external objects relative to the vehicle can be displayed. For example, the distance to the car in front of the vehicle can be displayed when parking. Similarly, the distance to the side of the vehicle relative to the corners of the vehicle (e.g., the corners of the front bumper) can be displayed when pulling into or backing into perpendicular.

[0060] The system also enables any number of other radar-based operations based on external objects that are not specifically listed here and would be recognized by one of ordinary skill in the art. As long as an object is able to reflect a radar transmission back to the smart device, information about that object can be ascertained and operations performed relative to it.

[0061] Occupant detection

[0062] Figure 5 An example embodiment of a smart device 102 for radar-based vehicle operations corresponding to occupants in a vehicle is illustrated. Figure 4 The described technique utilizes the following techniques.

[0063] During operation, smart device 102 generates and provides radar transmit signal 502. Radar transmit signal 502 is, for example, a continuous wave frequency modulated signal generated by radar system 104. Radar transmit signal 502 strikes interior objects, such as driver's seat 504, and occupants, such as driver 506 and passenger 508. Consequently, a radar receive signal may be received that includes radar reflections 510, 512, and 514, which are caused by radar transmit signal 502 reflecting off driver's seat 504, driver 506, and passenger 508, respectively. Each of radar reflections 510, 512, and 514 includes at least a portion of radar transmit signal 502. An internal reflection refers to any radar reflection (e.g., radar reflection 510) from a portion of the vehicle. While radar reflection 510 is reflected from a portion of the vehicle's interior, internal reflections may also originate from portions of the vehicle's exterior, such as side mirrors, hood emblems, hoods, or fenders. Occupant reflections refer to any radar reflections from an occupant within the vehicle (eg, radar reflections 512 and 514 ).

[0064] although Figure 5 Two occupants are shown (driver 506 and passenger 508), but the smart device 102 can distinguish and determine the characteristics of many passengers. Figure 4 Many positions or orientations of the smart device 102 may be utilized as long as the radar transmit signal 502 can be reflected from an occupant (eg, driver 506 , directly or indirectly from the smart device 102 ) and received by the smart device 102 in a given location.

[0065] Certain characteristics of the occupant can be determined without first determining the location of the smart device 102. For example, in some embodiments, detecting that the occupant is within a certain distance of the steering wheel can enable the smart device 102 to determine that the occupant is the driver 506. Similarly, movement near the side of the car seat would indicate that the associated occupant is a child in the back seat. In addition, assumptions can be made about the occupant, such as when the screen of the smart device 102 is generally pointed toward the rear of the vehicle, radar reflections from the right side will be from the driver 506 (at least in left-hand driver vehicles).

[0066] To determine certain characteristics of the occupant, the position (position and orientation relative to the vehicle) of smart device 102 can be calculated. Smart device 102 analyzes the radar receive signal to distinguish internal reflections (e.g., radar reflection 510) from occupant reflections (e.g., radar reflections 512 and 514). For example, objects corresponding to internal reflections will typically not move relative to smart device 102, whereas occupants typically do move. Smart device 102 can then analyze the internal reflections to determine a spatial representation of the vehicle's interior relative to smart device 102. Based on the known characteristics of the vehicle and the spatial representation of the vehicle's interior relative to the device, the apparatus can determine the position of smart device 102 relative to the vehicle.

[0067] The known characteristics of the vehicle can include a vehicle map, including an interior map, such that the relative locations of corresponding seats in the vehicle can be known based on the known interior locations. In one example, a user performs a vehicle setup process to map the interior of a particular vehicle. In another example, a vehicle space map is retrieved from a library of vehicle space maps. In these and other ways, the position of the smart device 102 relative to the vehicle is determined, which enables seat locations, and therefore occupant differentiation, to be performed by the smart device 102.

[0068] Other sensors, such as accelerometers, gyroscopes, imagers, and proximity sensors, can be used alone or in combination with internal reflections to determine the position of the smart device 102 relative to the vehicle. The smart device 102 can periodically determine its position relative to the vehicle in order to update its position for occupant detection (discussed below). In this way, the smart device 102 can be moved around the interior of the vehicle without negatively impacting the performance of the system.

[0069] Certain data about the occupants can be determined in response to determining the position of smart device 102 relative to the vehicle. Occupant reflections (e.g., radar reflections 506 and 508) can be analyzed to determine characteristics about the respective occupants. For example, the system can monitor the attention level of driver 506 but ignore the attention level of passenger 508.

[0070] The smart device 102 can then utilize the determined characteristics about the occupant to perform functions. For example, if the smart device 102 determines that the driver 506 is falling asleep or has not been paying attention for a period of time, an alert (visual, auditory, tactile, etc.) can be generated. As part of the operation, as discussed above, the smart device 102 can also use onboard sensors or interface with sensors of the vehicle. For example, a GPS system can be able to determine the speed of the vehicle and adjust the threshold of the attention-based operation based on the determined speed of the vehicle.

[0071] In another example, the smart device 102 may be able to determine that a child is present in the back seat of a vehicle. As discussed above, the smart device 102 is able to determine the presence of an occupant and the occupant's location relative to the vehicle. By further determining that the occupant is a child in a car seat (based on proximity to the car seat, size, movement, heartbeat, characteristic reflections from tissue, etc.), if the driver leaves the vehicle (by also tracking the driver or if the driver removes the smart device 102 from the vehicle), the smart device 102 can alert the driver 506 to the presence of a child in the car seat.

[0072] In another example, the smart device 102 can monitor the vital signs of one or more of the occupants, e.g., the driver 506 , the passenger 508 , or a child in a car seat, and generate an alert and / or request emergency services if the smart device 102 determines that service is needed.

[0073] Any number of other radar-based operations based on occupants can be performed using the system discussed above. As long as an occupant is able to reflect a radar transmission (directly or indirectly) back to a smart device, information about that occupant can be ascertained and operated upon.

[0074] Outdoor transportation operation

[0075] Figure 6 An example embodiment of a smart device 102 for radar-based vehicle operation corresponding to an object near an outdoor vehicle, e.g., a vehicle whose range is unknown or of little concern to the vehicle operation, is illustrated. While a typical bicycle is shown, the following techniques are applicable to many vehicles, such as recumbent bicycles, quad bikes, ATVs, motorcycles with or without automotive windshields, scooters, mopeds, rickshaws, golf carts, handcarts, electric bicycles, and tricycles.

[0076] During operation, smart device 102 generates and provides radar transmit signal 602. Although radar transmit signal 602 is shown as transmitting both forward and rearward, e.g., as in the spherical beam pattern discussed above, smart device 102 may transmit only in the forward or rearward direction, or a portion thereof. As an example, radar transmit signal 602 is a continuous wave frequency modulated signal generated by radar system 104. Radar transmit signal 602 strikes outdoor vehicle objects, e.g., occupant 604 and outdoor vehicle 606 (shown as a bicycle); as well as external objects, e.g., external vehicle 608 and tree 610. Consequently, a radar receive signal may be received that includes radar reflections 612, 614, 616, and 618, which are caused by reflections of radar transmit signal 602 from occupant 604, outdoor vehicle 606, external vehicle 608, and tree 610, respectively. Each of radar reflections 612, 614, 616, and 618 includes at least a portion of radar transmit signal 602. Outdoor vehicle reflections refer to radar reflections (e.g., radar reflections 612 and 614) from portions of outdoor vehicle 606 or occupants 604 of outdoor vehicle 606. External reflections refer to radar reflections (e.g., radar reflections 616 and 618) from objects that are not occupants 604 or part of outdoor vehicle 606.

[0077] To determine the characteristics of external objects, the orientation of the smart device 102 relative to the outdoor vehicle or the ground plane is calculated. The smart device 102 can analyze the radar received signal to determine the reflection from the ground. The direction perpendicular to the ground can be determined, which can calculate the orientation of the smart device 102. Other sensors such as accelerometers, gyroscopes, imagers, and proximity sensors can be used alone or in combination with reflections to determine the orientation of the smart device 102. The smart device can periodically update its orientation for external object detection (discussed below). This can enable the smart device 102 to move around the outdoor vehicle 606 without negatively impacting performance.

[0078] Certain characteristics of external objects can be determined without determining the orientation of the smart device 102. For example, aspects of an external object, such as size, relative speed, and distance to the smart device 102, can be determined without knowing the orientation of the smart device 102; however, the direction of the external object relative to the outdoor vehicle is based on the known orientation of the smart device 102, as will be discussed below.

[0079] However, certain other characteristics about the external object can be determined based on the determined orientation of the smart device 102. External radar reflections (e.g., radar reflections 616 and 618) can be analyzed using the known orientation to determine the location, direction, distance, size, identity, and speed of the external object relative to the outdoor vehicle 606. In some embodiments, reflections from the occupant 604 and the outdoor vehicle 606 (e.g., radar reflections 612 and 614) can be ignored for purposes of external objects.

[0080] Smart device 102 can then utilize the determined information about the external objects (with or without regard to the determined orientation of smart device 102) to perform a function. For example, the function may involve displaying the identification of one or more of the external objects on the display screen of smart device 102, along with the object's distance, direction, and / or relative speed. This information may be displayed relative to a "surround view" or "driver's view" of the outdoor vehicle, or relative to some other representation of the outdoor vehicle. While there are potential radar "blind spots" caused by occupants 604 or portions of outdoor vehicle 606, the system can still track cars approaching from behind when mounted on the handlebars, for example, because reflections from vehicles are very strong and are generally not completely obscured by occupants 604 or outdoor vehicle 606. In some embodiments, a warning or notification may be displayed based on the determined characteristics of the external objects. For example, if smart device 102 determines that a collision with, for example, external vehicle 608 is imminent, a notification may be presented to alert occupants 604. Smart device 102 may utilize onboard sensors as part of its operation. For example, the GPS system may be able to determine the speed of the outdoor vehicle and adjust the threshold for the collision avoidance maneuver based on the determined speed of the outdoor vehicle 606 .

[0081] Other radar-based operations for outdoor vehicle operations can be performed using the system discussed above. As long as an object is able to reflect a radar transmission signal back to the smart device, information about the object can be ascertained and operations can be performed relative to it.

[0082] Example Method

[0083] In the following discussion, please refer to Figure 1 Environments 100-1 to 100-3 and Figures 4 to 6 The entities detailed in , references thereto are for example only. These techniques are not limited to being performed by one entity or multiple entities running on one device. Figures 4 to 6The operations (or actions) set shown are performed, but are not necessarily limited to the order or combination of the operations shown in this article. Further, any one of one or more of the operations can be repeated, combined, reorganized or linked to provide a wide range of additional and / or alternative methods.

[0084] Figure 7 An example method 700 is depicted for operating a smart device-based radar system capable of characterizing objects through a vehicle's windshield.

[0085] At 702, the smart device transmits a radar transmission signal from the interior of the vehicle through the windshield of the vehicle. Figure 4 As shown in FIG, smart device 102 can be mounted somewhere on the dashboard or pillar of a vehicle so that radar transmit signal 402 can reach an external object (e.g., car 408). The smart device can adjust the radar transmit signal to effectively penetrate the windshield. For example, a first radar transmit signal can be transmitted to determine attenuation caused by the windshield, and the radar transmit signal can be adjusted based on received reflections of the first radar transmit signal.

[0086] At 704 , the smart device receives an external radar reflection signal that passes through the windshield, the external radar reflection signal being caused by the radar transmission signal reflecting off an object external to the vehicle. For example, the external radar reflection signal may include a reflection from another vehicle (e.g., reflection 414 ).

[0087] At 706, the external radar reflection signal is analyzed to detect characteristics of the object. The characteristics may include the object's identity, the object's orientation relative to the smart device, the distance between the object and the smart device, or the relative speed between the object and the smart device. In some embodiments, the smart device filters out or otherwise compensates for attenuation caused by the windshield to determine the characteristics.

[0088] At 708, an action is initiated based on the detected characteristic of the object. The action may involve displaying the characteristic on a display screen of the smart device, generating and presenting a notification or alert if the characteristic meets or exceeds certain thresholds, displaying the characteristic along with one or more representations of a vehicle, and the like.

[0089] Figure 8 An example method 800 is depicted for operating a smart device-based radar system capable of characterizing objects external to a vehicle.

[0090] At 802, the smart device transmits one or more radar transmission signals from the interior of the vehicle. Figure 4As shown in FIG, the smart device 102 may be mounted somewhere on the dashboard or pillar of a vehicle so that the radar transmit signal 402 can reach an external object (eg, a car 408).

[0091] At 804, the smart device receives an internal radar reflection signal caused by one of the radar transmission signals reflecting off a portion of the vehicle. For example, the internal radar reflection may be caused by a reflection from a windshield of the vehicle (e.g., reflection 412). In some embodiments, the internal radar reflection signal may originate from a portion of the exterior of the vehicle, such as a side mirror, a hood emblem, a hood, or a fender.

[0092] At 806, the smart device determines the location of the smart device relative to the vehicle. This location determination can be based on the first reflected portion of the radar receive signal, alone or in combination with other sensors of the smart device (e.g., an accelerometer, an inclinometer, or a gyroscope). This determination can also utilize a predefined vehicle mapping selected from a library or created by the user. Furthermore, the location of the smart device can be determined based on another radar receive signal (received before, in parallel with, or after the radar receive signal).

[0093] At 808, the smart device receives an external radar reflection signal caused by one of the radar transmit signals reflecting off an object external to the vehicle. The external radar reflection signal can be a reflection of the same radar transmit signal that caused the internal radar reflection signal or a reflection of a different radar transmit signal (e.g., transmitted before, after, or based on a different modulation scheme). For example, the external radar reflection signal can be reflected from another vehicle (e.g., reflection 414) and received through windshield 406.

[0094] At 810 , the external radar reflection signal is analyzed to detect characteristics of the object. The characteristics may include an identity of the object, a direction of the object relative to the smart device, a distance between the object and the smart device, or a relative speed between the smart device and the object.

[0095] At 812, enhanced characteristics of the object are determined based on the determined characteristics along with the determined location of the smart device. The enhanced characteristics of the object may include a distance and / or a heading from the range of the vehicle to the object. For example, the smart device may determine that the object is 10 meters away, but because the smart device may be 2 meters away from the range of the vehicle (e.g., the front bumper), the object may be only 8 meters away from the impact vehicle.

[0096] At 814, an action is initiated based on the enhanced feature of the object. The action may involve displaying the enhanced feature on a display screen of the smart device, generating and presenting a notification or alert if the enhanced feature meets or exceeds certain thresholds, displaying the enhanced feature along with one or more representations of a vehicle, and the like.

[0097] Figure 9 An example method 900 is depicted for operating a smart device-based radar system capable of in-vehicle occupant detection and tracking.

[0098] At 902, the smart device transmits one or more radar transmission signals from the interior of the vehicle. Figure 5 As shown in FIG, the smart device 102 may be mounted somewhere on the dashboard or pillar of the vehicle so that the radar transmission signal 502 can reach the occupants (eg, the driver 506 and the passenger 508).

[0099] At 904 , the smart device receives an occupant radar reflection signal caused by one of the radar transmit signals reflecting off an occupant of the vehicle. For example, the occupant radar reflection signal may be reflected off the driver of the vehicle (eg, radar reflection 512 ).

[0100] At 906, the occupant radar reflection signal is analyzed to detect characteristics of the occupant. For example, the characteristics may be gaze direction, facial expression, determination of open or closed eyes, or simply the presence of the occupant.

[0101] At 908, an activity of the occupant is determined based on the detected characteristics. For example, the activity determined may be that the occupant is not paying attention to the road or that the occupant has a healthy heartbeat or breathing rate.

[0102] At 910, the smart device receives an interior radar reflection signal caused by one of the radar transmit signals reflecting off an interior surface of the vehicle. The interior radar reflection signal can be a reflection of the same radar transmit signal that caused the occupant radar reflection signal or a reflection of a different radar transmit signal (e.g., transmitted before, after, or based on a different modulation scheme). For example, the interior radar reflection signal can include a reflection from a seat (e.g., radar reflection 510) or a reflection from the steering wheel. In some embodiments, the interior radar reflection signal can originate from a portion of the vehicle's exterior, such as a side mirror, a hood emblem, a hood, or a fender.

[0103] At 912, the smart device determines the location of the occupant relative to the vehicle based on the internal radar reflection signal. For example, the smart device may determine that the occupant is the driver 506, the passenger 508, or a child in a car seat. This determination may be made in conjunction with other sensors of the smart device (e.g., an accelerometer, an inclinometer, or a gyroscope). This determination may utilize a predefined vehicle mapping, selected from a library or user-created, or based on detected objects near the occupant, such as a steering wheel or a car seat.

[0104] At 914, an action is initiated based on the determined activities and the location of the occupant. For example, some of the determined activities may apply only to the driver of the vehicle (looking at the road, paying attention to driving, etc.), while other activities, such as being alive, may apply to any occupant location. The action may involve alerting the occupant (e.g., visually, audibly, via a haptic device, via the vehicle's sound system) or automatically contacting emergency services (e.g., if a crash is detected or if no heartbeat is detected). As another example, an alert may also be generated if the driver of the vehicle exits the vehicle while a child is still in a child seat in the vehicle.

[0105] Figure 10 An example method 1000 is depicted for operating a smart device-based radar system capable of characterizing objects proximate to an outdoor vehicle.

[0106] At 1002, a smart device transmits a radar transmission signal while mounted on an outdoor vehicle. The outdoor vehicle may be any form of vehicle, such as a bicycle, a recumbent bike, a quad bike, an ATV, a motorcycle with or without a windshield, a scooter, a moped, a rickshaw, a golf cart, a handcart, an electric bike, a tricycle, or any other small vehicle where the range of the vehicle may not be important. For example, Figure 6 As shown in FIG, the smart device 102 can be mounted somewhere on the handlebars of the bicycle so that the radar transmission signal 602 can reach external objects (eg, external vehicles 608).

[0107] At 1004 , the smart device receives an external radar reflection signal caused by a radar transmission signal reflecting off an object external to the occupant / outdoor vehicle.

[0108] At 1006 , the external radar reflected signal is analyzed to detect characteristics of the object. The characteristics may include the identity of the object (e.g., 608 , a car, 610 , a tree) or the distance, direction, or speed of the object relative to the smart device.

[0109] The smart device determines the orientation of the smart device relative to the occupant, the outdoor vehicle, or the ground at 1008. This determination can be based on radar reflections (e.g., from the ground, the occupant, or the outdoor vehicle) and / or based on data from other sensors of the smart device (e.g., an accelerometer, an inclinometer, or a gyroscope).

[0110] At 1010, enhanced characteristics of the object are determined based on the characteristics of the object and the determined orientation of the smart device.For example, the smart device may determine that the object is 10 meters away and is generally in the path of an outdoor vehicle.

[0111] At 1012, an action is initiated based on the enhanced feature of the object. The action may involve displaying the enhanced feature on a display screen of the smart device, generating and presenting a notification or warning if the enhanced feature meets or exceeds certain thresholds (e.g., a collision may be imminent), displaying the enhanced feature along with one or more representations of an occupant or outdoor vehicle, and the like.

[0112] in conclusion

[0113] Although techniques for using and devices including a smart device-based radar system in a vehicular environment have been described using language specific to features and / or methods, it will be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations of a smart device-based radar system in a vehicular environment.

Claims

1. A use of a mobile smart device for initiating an operation based on an enhanced characteristic of an object while the mobile smart device is inside a vehicle, the mobile smart device comprising: A radar system, comprising: one or more antennas; at least one transceiver coupled to the antenna and configured to: transmitting radar transmit signals via one or more of the antennas; and receiving radar reflection signals via one or more of the antennas; at least one processor; and At least one computer-readable storage medium including processor-executable instructions that, in response to being executed by the processor, cause the mobile smart device to: transmitting, via the radar system, one or more radar transmission signals while the mobile smart device is within the vehicle; Received via the radar system: an internal radar reflection signal caused by one of the radar transmission signals being reflected from a portion of the vehicle; and an external radar reflection signal that has penetrated a windshield of the vehicle and is caused by one of the radar transmit signals reflecting from the object that is not part of the vehicle, another part of the vehicle, or an occupant within the vehicle; estimating a position of the mobile smart device relative to the vehicle based on the internal radar reflection signal; analyzing the external radar reflection signal to determine characteristics of the object; determining the enhanced characteristic of the object based on the estimated position of the mobile smart device relative to the vehicle and the characteristic of the object; and An action is initiated based on the enhanced characteristic of the object.

2. The use according to claim 1, wherein The at least one computer-readable storage medium further includes processor-executable instructions that, in response to execution by the processor, cause the mobile smart device to periodically determine a position of the mobile smart device relative to the vehicle to update an estimated position, thereby enabling the mobile smart device to be moved around an interior of the vehicle while determining the enhanced characteristic of the object.

3. The use according to claim 1, wherein The internal radar reflection signal and the external radar reflection signal are caused by respective reflections of the same radar transmit signal.

4. The use according to claim 1, wherein: The internal radar reflection signal is caused by the reflection of the first radar transmission signal, The external radar reflection signal is caused by a reflection of the second radar transmission signal, and The second radar transmit signal is based on the internal radar reflection signal.

5. The use according to claim 1, wherein The characteristics of the object include: the identity of the object; the distance to the object; The direction of the object relative to the mobile smart device; or The relative speed between the object and the mobile smart device.

6. The use according to claim 1, wherein The external radar reflection signal differs from the internal radar reflection signal based on: the size, shape, speed or distance of the object; or Attenuation information of the external radar reflection signal passing through the windshield is transmitted.

7. The use according to claim 1, wherein Determining at least one range of the vehicle relative to the mobile smart device based on: an estimated position of the smart device relative to the vehicle; and A predetermined spatial profile of the vehicle.

8. The use according to claim 7, wherein The predetermined spatial profile of the vehicle is one of a plurality of predetermined spatial profiles corresponding to respective vehicle models.

9. The use according to claim 7, wherein The predetermined spatial profile is created by a user via the radar system.

10. The use according to claim 7, wherein The enhanced features of the object include at least one of the following: The direction of the object relative to the vehicle; and The distance between the object and the range of the vehicle.

11. The use according to claim 10, wherein The enhanced features of the object are displayed on a display screen of the mobile smart device relative to a representation of the vehicle.

12. A use of a mobile smart device for initiating an operation based on characteristics of an object when the mobile smart device is mounted on the handlebars of an outdoor vehicle, the mobile smart device comprising: A radar system, comprising: one or more antennas; at least one transceiver coupled to the antenna and configured to: transmitting a radar transmission signal via at least one of the antennas; and receiving a radar reflection signal via at least one of the antennas; at least one processor; and At least one computer-readable storage medium including processor-executable instructions that, in response to being executed by the processor, cause the mobile smart device to: transmitting, via the radar system, one or more radar transmission signals when the mobile smart device is mounted on the handlebar of the outdoor vehicle; Received via the radar system: an external radar reflection signal caused by one of the radar transmission signals reflecting from an object near the outdoor vehicle; and an outdoor vehicle radar reflection signal, the outdoor vehicle radar reflection signal being caused by one of the radar transmission signals reflecting from the outdoor vehicle or an occupant of the outdoor vehicle; distinguishing the external radar reflection signal from the outdoor vehicle radar reflection signal; analyzing the external radar reflection signal to detect at least one characteristic of the object; and An operation is initiated based on the characteristic of the object.

13. The use according to claim 12, wherein The characteristics of the object include: the identity of the object; a direction to the object based on the determined orientation of the mobile smart device; the distance to the object; or The relative speed between the object and the mobile smart device.

14. The use according to claim 13, wherein A notification is generated in response to predicting a collision with the object.

15. The use according to claim 12, wherein The outdoor vehicle radar reflection signal is caused by the radar transmission signal being reflected from the outdoor vehicle and the occupant of the outdoor vehicle.

16. The use according to claim 12, wherein The mobile smart device is configured to track a car approaching the outdoor vehicle from behind based on analyzing the external radar reflection signal.

Images