In-vehicle motion recognition using ultra-wideband radar

Abstract

The invention provides in-vehicle motion recognition with ultra wide band radar. A computer includes a processor and a memory storing instructions executable by the processor to: receive radar data from an ultra wide band radar in a passenger compartment of a vehicle; determining a start time of a time window based on a second derivative of the radar data; identifying motion inside the passenger compartment based on the radar data received during the time window; and actuate a component of the vehicle based on the identification of the motion.

Description

Technical Field

[0001] The present disclosure relates to technology for controlling a vehicle based on in-vehicle motion recognition using ultra-wideband radar. Background Art

[0002] Some radars use ultra-wideband signals, i.e., signals with low energy levels spread over a large area of the radio spectrum. The Federal Communications Commission and the International Telecommunication Union's Radiocommunication Sector define ultra-wideband as antenna transmissions with a transmitted signal bandwidth exceeding the lesser of 500 MHz or 20% of the arithmetic center frequency. Summary of the Invention

[0003] This disclosure describes technology for controlling a vehicle based on identifying motion within the vehicle's passenger compartment (e.g., an occupant's breathing, a person entering or leaving the passenger compartment, etc.). The identification is performed based on radar signals from an ultra-wideband (UWB) radar in the passenger compartment. Using a UWB radar is advantageous because it can be installed on vehicles for other purposes, such as phone-as-a-key (PaaK). The vehicle's computer is programmed to: receive radar data from the UWB radar; determine a start time for a time window based on the second-order derivative of the radar data; identify motion within the passenger compartment based on the radar data received during the time window; and actuate components of the vehicle based on the identification of the motion. Dynamically setting the time window allows the computer to perform identification using radar data from a single time window. In contrast, a fixed window may have too little radar data for identification or may have more radar data than required for identification, thereby increasing latency. If motion is identified, the computer may, for example, output an alert to the occupant in response to the motion satisfying a condition.

[0004] A computer includes a processor and a memory storing instructions executable by the processor to: receive radar data from an ultra-wideband radar in a passenger compartment of a vehicle; determine a start time of a time window based on a second-order derivative of the radar data; identify motion inside the passenger compartment based on the radar data received during the time window; and actuate a component of the vehicle based on the identification of the motion.

[0005] In one example, the second derivative of the radar data may be a second derivative of fast time with respect to slow time.

[0006] In one example, the instructions may further include instructions for refraining from using the radar data received before the start time to identify the motion inside the passenger compartment.

[0007] In one example, the instructions may further include instructions for converting the radar data received during the time window into a frequency domain and identifying the motion in the passenger compartment based on the radar data in the frequency domain. In another example, the instructions may further include instructions for converting the radar data received during the time window into the frequency domain by applying a fast Fourier transform.

[0008] In one example, the time window may have a preset duration.

[0009] In one example, the instructions may further include instructions for applying a filter to the radar data, and the second-order derivative may be a second-order derivative of the radar data after applying the filter. In another example, the filter may be a smoothing filter.

[0010] In yet another example, the filter may be a bandpass filter that isolates the frequencies of human breathing.

[0011] In one example, the time window may be a first time window, the start time may be a first start time, and the instructions may further include instructions for identifying the motion within the passenger cabin based on the radar data received during a second time window, the second time window beginning at the end time of the first time window. In another example, the instructions may further include instructions for refraining from using the radar data received after the current time window to identify the motion within the passenger cabin in response to the second-order derivative being below a threshold amount for at least a threshold duration.

[0012] In one example, the instructions may further include instructions for classifying the motion as a type of living whole motion based on the radar data received during the time window. In another example, the instructions may further include instructions for classifying the motion as a type of living whole motion by executing a neural network classifier using the radar data received during the time window as input.

[0013] In one example, the instructions may further include instructions for identifying a plurality of breathing individuals based on the radar data received during the time window.

[0014] In one example, the instructions may further include instructions for commanding a user interface to output an alert to an occupant of the passenger compartment based on the identification of the motion.

[0015] A method includes receiving radar data from an ultra-wideband radar in a passenger compartment of a vehicle; determining a start time of a time window based on a second derivative of the radar data; identifying motion inside the passenger compartment based on the radar data received during the time window; and actuating a component of the vehicle based on the identification of the motion.

[0016] In one example, the second derivative of the radar data may be a second derivative of fast time with respect to slow time.

[0017] In one example, the method may further include refraining from using the radar data received before the start time to identify the motion within the passenger compartment.

[0018] In one example, the method may further include converting the radar data received during the time window into a frequency domain; and identifying the motion in the passenger compartment based on the radar data in the frequency domain.

[0019] In one example, the method may further include commanding a user interface to output an alert to an occupant of the passenger compartment based on the identification of the motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a diagrammatic top view of an exemplary vehicle with the passenger compartment exposed for illustration purposes.

[0021] Figure 2A Example graphs of signal strength and fast time versus slow time are shown for example radar data before and after applying a first filter.

[0022] Figure 2B Example graphs of signal strength and fast time versus slow time are shown for example radar data before and after applying the second filter.

[0023] Figure 3 An example graph of amplitude versus frequency of example radar data is shown.

[0024] Figure 4 is a flow chart of an example process for identifying motion in a passenger compartment based on radar data. DETAILED DESCRIPTION

[0025] Referring to the drawings, in which like numerals indicate like parts throughout the several views, a computer 105 includes a processor and a memory storing instructions executable by the processor to: receive radar data from an ultra-wideband (UWB) radar 110 in a passenger cabin 115 of the vehicle 100; determine a start time of a time window based on a second derivative of the radar data; identify motion inside the passenger cabin 115 based on the radar data received during the time window; and actuate a component 120 of the vehicle 100 based on the identification of motion.

[0026] refer to Figure 1 The vehicle 100 may be any passenger or commercial vehicle, such as a car, truck, sport utility vehicle, crossover, van, minivan, taxi, bus, etc. The vehicle 100 includes a passenger compartment 115, a computer 105, a communication network 130, a plurality of UWB radars 110, and other components 120, such as a user interface.

[0027] The vehicle 100 includes a passenger compartment 115 for accommodating occupants 140 of the vehicle 100 (if any). The passenger compartment 115 includes one or more seats 125, for example, one or more of the seats 125 in the front row of the passenger compartment 115 and one or more of the seats 125 in the second row behind the front row. The passenger compartment 115 may also include a third row (not shown) of seats 125 at the rear of the passenger compartment 115. The seats 125 are shown as bucket seats in the front row and bench seats in the second row, but other types of seats 125 are possible. The position and orientation of the seats 125 and their components can be adjusted by the occupants 140.

[0028] Computer 105 is a microprocessor-based computing device, such as a general-purpose computing device (including a processor and memory, an electronic controller, etc.), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a combination of the foregoing. Typically, hardware description languages such as VHDL (VHSIC (Very High Speed Integrated Circuit) Hardware Description Language) are used in electronic design automation to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, while the logic components within an FPGA can be configured based on, for example, VHDL programming stored in a memory electrically connected to the FPGA circuitry. Therefore, computer 105 may include a processor, memory, and the like. The memory of computer 105 may include media for storing instructions executable by the processor and for electronically storing data and / or databases, and / or computer 105 may include structures such as the aforementioned structures for providing programming. Computer 105 may be multiple computers coupled together.

[0029] The computer 105 can transmit and receive data via the communication network 130. The communication network 130 can be, for example, a controller area network (CAN) bus, Ethernet, WiFi, a local interconnect network (LIN), an on-board diagnostic connector (OBD-II), and / or any other wired or wireless communication network. The computer 105 can be communicatively coupled to the UWB radar 110 and other components 120 via the communication network 130.

[0030] The UWB radar 110 transmits and receives radio waves. The UWB radar 110 uses ultra-wideband signals, for example, signals with low energy levels spread across a wide range of the radio spectrum. The Federal Communications Commission and the International Telecommunication Union's Radiocommunication Sector define ultra-wideband as antenna transmissions with a transmitted signal bandwidth exceeding the smaller of 500 MHz or 20% of the arithmetic center frequency. The UWB radar 110 can use any suitable modulation method, such as orthogonal frequency division multiplexing (OFDM), phase shift keying (PSK), pulse position modulation (PPM), etc. The UWB radar 110 is configured to transmit UWB waves and receive reflections of those UWB waves to detect physical objects in the environment. The UWB radar 110 can also be configured to use UWB waves to wirelessly communicate with mobile devices, for example, for PaaK functionality. This may be a benefit of using the UWB radar 110 instead of another type of ranging sensor.

[0031] The UWB radar 110 is fixedly mounted to the body 135 of the vehicle 100 in the passenger compartment 115. The UWB radars 110 can be spaced apart from each other, for example, scattered throughout the passenger compartment 115, which can facilitate the ability to distinguish locations when used for trilateration. For example, the UWB radar 110 can include six UWB radars 110, four located at the corners of the passenger compartment 115, and two oriented toward the longitudinal centerline of the passenger compartment 115.

[0032] The vehicle 100 includes other components 120 that can be actuated by the computer 105, such as an ignition, a braking system, a user interface, and the like. The user interface presents information to an occupant 140 (e.g., an operator) of the vehicle 100 and receives information from the occupant. The user interface can be located, for example, on a dashboard in the passenger compartment 115 of the vehicle 100, or anywhere else that the occupant 140 can easily see it. The user interface can include dials, digital readouts, screens, speakers, and the like for providing information to the occupant 140, such as known human-machine interface (HMI) elements. The user interface can also include buttons, knobs, keypads, microphones, and the like for receiving information from the occupant 140.

[0033] refer to Figures 2A to 2B, the computer 105 is programmed to receive radar data from the UWB radar 110, for example, via the communication network 130. The radar data may include the return times of the UWB pulses and the strength of those pulses. For example, the radar data may be represented as a graph 205, 215 of slow time versus fast time and signal strength, i.e., slow time is the independent variable and fast time and signal strength are the dependent variables, such as Figures 2A to 2B As shown in Figure 1 . The slow time dimension is measured in increments of time that the UWB radar 110 transmits a UWB pulse. The fast time dimension measures the return time of the pulse and can be measured in increments at least an order of magnitude more frequent than the slow time dimension. The fast time value indicates the length of time since the most recent pulse was transmitted. The fast time indicates the distance the pulse traveled before reflecting back to the UWB radar 110. The signal strength indicates the strength of the pulse, which can be related to the distance to the object and the object's reflectivity.

[0034] The radar data may include multiple signals 225, 230 that persist over a slow time dimension. Different signals 225, 230 may indicate different objects in the passenger compartment 115, such as seats 125, occupants 140, and the like. The computer 105 may identify a collection of fast time measurements as signals 225, 230 based on the fast times being sufficiently close from one slow time unit to the next, e.g., each pair of consecutive fast time measurements in the collection is within a fast time threshold of each other. The fast time threshold may be selected to indicate that the fast time measurements are likely for the same object.

[0035] The computer 105 may be programmed to select one or a subset of the signals 225 and 230 (referred to as selected signals 225) for processing described below. The computer 105 may select one of the signals 225 and 230 based on the signal 225 and 230 corresponding to an object moving relative to the vehicle 100 (e.g., the occupant 140 rather than the seat 125). For example, the computer 105 may select the selected signal 225 from the signals 225 and 230 based on one or more values characterizing the selected signal 225 (e.g., contrast of fast time, variance of fast time versus slow time, and / or derivative of fast time versus slow time). The contrast of the signals 225 and 230 may be the difference between a maximum value and a minimum value of the fast time. The computer 105 may select the selected signal 225 in response to one or more of the values characterizing the signals 225 and 230 exceeding a threshold value (e.g., peak contrast, variance, contrast normalized by variance, or derivative of fast time versus slow time). Alternatively or additionally, the computer 105 may select the selected signal 225 in response to one or more of the values characterizing the signals 225 , 230 being greater for the selected signal 225 than for either of the unselected signals 230 .

[0036] Computer 105 can be programmed to apply one or more filters to radar data, for example, to selected signal 225. The term "filter" is used herein in its signal processing sense as the process of removing components or features from a signal (here, radar data). Filters can be applied to the fast-time versus slow-time dimension of radar data, i.e., to radar data indicating range. Filters can be selected to isolate data relevant to identifying different types of motion. Multiple filters can be used for different types of motion.

[0037] refer to Figure 2A Computer 105 may apply a bandpass filter to the radar data. The term "bandpass filter" is used herein in its signal processing sense as a filter that passes frequencies within a predefined range and rejects or attenuates frequencies outside the predefined range. The predefined range is stored in the memory of computer 105, for example, as fixed upper and lower bounds. The bandpass filter may isolate the frequency of human breathing. Figure 2A A first graph 205 of radar data before a bandpass filter is applied is shown on the left, and a second graph 210 of radar data after the bandpass filter is applied is shown on the right. For example, the predefined range (e.g., upper and lower bounds) can be selected to cover typical breathing frequencies, e.g., 0.2 Hz to 0.4 Hz. Computer 105 can also apply a gain to the radar data within the predefined range, which can compensate for the small movements involved in breathing.

[0038] refer to Figure 2B , the computer 105 can apply a smoothing filter to the radar data. The smoothing filter can remove noise from the radar data that is not related to frequency. Figure 2B A first graph 215 of radar data before a smoothing filter is applied is shown on the left, and a second graph 220 of radar data after a smoothing filter is applied is shown on the right. Computer 105 may use any suitable algorithm for smoothing the data, such as a moving average, a Savitzky-Golay smoothing filter, local regression, etc.

[0039] Computer 105 may use radar data collected during a data collection period to identify motion within passenger cabin 115, as described below. A data collection period may consist of multiple consecutive time windows, as described below. Computer 105 may avoid using radar data received before the start of a data collection period (i.e., before the start time of the first time window of the data collection period) to identify motion within passenger cabin 115. Computer 105 may also avoid using radar data received after the last time window of the data collection period to identify motion within passenger cabin 115.

[0040] When performing motion recognition, described below, computer 105 uses radar data collected during a time window. Computer 105 may use radar data collected during a single time window at a time, i.e., using radar data collected during one time window alone before continuing to use radar data collected during the next time window. Each time window extends from a start time to an end time. The start time and end time are points in slow time. A time window may have a preset duration, i.e., a preset length of time from the start time to the end time. The preset duration may be stored in computer 105's memory. The preset duration may be selected to provide sufficient data for the motion recognition process described below. Computer 105 may store the radar data collected during each time window in a buffer in computer 105's memory, and computer 105 may clear the buffer at the beginning of the next time window. As described below, the start time is variable and therefore not preset. When determining the start time of a time window, computer 105 may determine the end time of the time window by adding the preset duration to the start time.

[0041] The computer 105 can be programmed to determine the start time of the time window based on the second derivative of the radar data. For example, the computer 105 can determine the start time of the first time window of the data collection period based on the second derivative of the radar data, and thereby determine the start of the data collection period. The second derivative of the radar data can be the second derivative of the fast time with respect to the slow time of the selected signal 225 from the radar data. After applying one of the filters, the computer 105 can perform the second derivative on the radar data (e.g., on the selected signal 225). The computer 105 can determine the start time as the slow time when the second derivative first exceeds a threshold amount (e.g., as for x, in a minimum value) such that the following expression is satisfied:

[0042]

[0043] Where y represents the fast time of the selected signal 225, x represents the slow time, and K represents the threshold amount. The threshold amount K can be selected to indicate the movement of the object. The inventors have determined that the second-order derivative performs well in indicating the state change of the object.

[0044] The computer 105 can be programmed to determine the start time of a time window in the data collection period after the first time window in the data collection period. For example, the start time of a time window can be the end time of the immediately preceding time window. In other words, the time windows in the data collection period can be consecutive, with each time window starting when the previous time window ends.

[0045] Computer 105 can be programmed to determine the final time window for data collection based on the second-order derivative. For example, computer 105 can determine that the current time window is the final time window in response to the second-order derivative being below a threshold amount for at least a threshold duration. The second-order derivative can be the same second-order derivative as described above, that is, the second-order derivative of the fast time of selected signal 225 to the slow time after applying one of the filters. The threshold amount can be the same threshold amount K as the start time for the data collection period. The threshold duration is a length of time and can be selected to be longer than the repetition of the type of motion of interest. Therefore, in response to the second-order derivative being below a threshold amount for at least a threshold duration, computer 105 can determine that the end time of the current time window is the end of the data collection period, and computer 105 can avoid using radar data received after the current time window to identify motion inside passenger compartment 115.

[0046] refer to Figure 3 Computer 105 may be programmed to convert radar data received during the time window into the frequency domain. The conversion to the frequency domain generates a plot of amplitude versus frequency over time. For example, computer 105 may convert radar data received during the time window into the frequency domain by applying a fast Fourier transform. Computer 105 may implement any suitable algorithm for computing a fast Fourier transform, such as Cooley-Tukey, prime factors, Bruun transform, Rader transform, chirp-Z transform, hexagonal transform, and the like.

[0047] Computer 105 is programmed to identify motion within passenger compartment 115 based on radar data received during a time window. Computer 105 may separately analyze radar data filtered with a bandpass filter and radar data filtered with a smoothing filter. The radar data filtered with a bandpass filter and radar data filtered with a smoothing filter can be used to identify different types of motion. For example, radar data filtered with a bandpass filter can be used to identify human breathing, such as multiple individual breaths in passenger compartment 115. As another example, radar data filtered with a smoothing filter can be used to identify animate, global motion, e.g., classifying motion into types of animate, global motion, such as a person sitting on seat 125, a person moving on seat 125, a child moving in a car seat mounted on seat 125, a dog lying on seat 125, etc. The term "animate" indicates that the motion is performed by a living being, and the term "global motion" indicates that the motion is large-scale, rather than a physiological process such as breathing. For one or both of the radar data filtered with the bandpass filter and the radar data filtered with the smoothing filter, computer 105 may identify motion in passenger cabin 115 based on the radar data in the frequency domain.

[0048] For example, computer 105 can be programmed to identify multiple breathing individuals based on radar data received during a time window. Computer 105 can count the number of peaks in the radar data converted to the frequency domain that exceed a threshold amplitude. The threshold amplitude can be selected to be above the typical noise level of radar data in the frequency domain and below the typical amplitude of human breathing. Figure 3 A first graph 305 is shown with zero peaks exceeding the threshold amplitude, indicating that zero individuals were detected breathing; a second graph 310 with one peak exceeding the threshold amplitude, indicating that one individual was detected breathing; and a third graph 315 with three peaks exceeding the threshold amplitude, indicating that three individuals were detected breathing.

[0049] For another example, computer 105 may classify a motion as a type of whole-animal motion based on radar data received during a time window. The motion classification may include the type of creature performing the motion and / or the type of action the motion pertains to. The creature type may include an adult, a child, a dog, a cat, etc. The action type may include sitting down, moving, turning the head, etc. Some actions may be specific to a human type, for example, sitting down may represent a human, while lying down may represent a dog.

[0050] Computer 105 can classify the motion as a type of living, integral motion by executing a neural network classifier using radar data received during a time window as input. The neural network classifier can receive as input radar data received during one of the time windows and filtered using a smoothing filter. The input radar data can be in the time domain or the frequency domain. The neural network classifier can provide as output a classification of the motion as a specific type, for example, from a pre-stored list of types. The neural network classifier can be any suitable type of neural network, such as a convolutional neural network. A convolutional neural network comprises a series of layers, with each layer using the previous layer as input. Each layer contains a plurality of neurons that receive data generated by a subset of neurons in the previous layer as input and generate outputs that are sent to neurons in the next layer. Layer types include: convolutional layers, which compute dot products of weights with a small region of input data; pooling layers, which perform downsampling operations along spatial dimensions; and fully connected layers, which are generated based on the outputs of all neurons in the previous layer. The final layer of the convolutional neural network generates a score for each potential motion type, and the final output is the type with the highest score.

[0051] A neural network classifier can be trained to classify motions in training data. During training, the neural network classifier can output a classification and velocity for each motion. The training data can include radar data for different motions paired with the ground-truth classification and ground-truth velocity of the motion. The ground-truth classification can be derived by performing object recognition on camera data recorded simultaneously with the training radar data. The ground-truth velocity can be the first-order derivative of the fast time with respect to the slow time of a selected signal 225 of the training radar data. The neural network classifier can be trained to minimize a loss function, for example, via backpropagation. The loss function can include terms for a classification loss and a physics loss. The classification loss can penalize output classifications that do not match the ground-truth classification. The physics loss can penalize output velocities based on the magnitude of the difference from the ground-truth velocity. The use of the physics loss can improve the accuracy of the classification output by the neural network classifier compared to training using only the classification loss.

[0052] Computer 105 is programmed to actuate a component 120 of vehicle 100 based on the recognition of motion. For example, computer 105 may instruct a user interface to output an alert to occupants 140 of passenger compartment 115 based on the recognition of motion. Computer 105 may actuate component 120 in response to the recognition of motion satisfying a condition. For example, the condition may be that the number of breathing individuals exceeds a threshold when vehicle 100 is shut down. As another example, the condition may be that the overall motion of an animate object is classified as a person moving from one seat 125 to another.

[0053] Figure 4 is a flow chart illustrating an example process 400 for identifying motion in the passenger compartment 115 based on radar data. The memory of the computer 105 stores executable instructions for performing the steps of process 400 and / or may be programmed in a structure such as that described above. In general terms, the computer 105 receives radar data, selects the selected signal 225 from the radar data, and filters the radar data. If data collection has begun or the criteria for starting data collection are met, the computer 105 stores the radar data in a buffer in the computer 105 memory until the end of the time window. The computer 105 then converts the radar data in the buffer to the frequency domain, clears the buffer, identifies motion, and, if conditions are met, actuates the component 120. If the criteria for doing so are met, the computer 105 stops data collection. Process 400 continues as long as the vehicle 100 remains on.

[0054] Process 400 begins in block 405 , where computer 105 receives radar data from one of UWB radars 110 in passenger compartment 115 of vehicle 100 , as described above.

[0055] Next, in block 410 , the computer 105 selects the selected signal 225 from the radar data and applies a filter to the radar data, as described above.

[0056] Next, at decision block 415, computer 105 determines whether a data collection period is currently in progress, i.e., whether the data collection period has begun, as described above. For example, a flag in a memory of computer 105 may indicate whether a data collection period is in progress or not currently being executed. In response to the data collection period not having begun, process 400 proceeds to decision block 420. In response to the data collection period being in progress, process 400 proceeds to block 425.

[0057] At decision block 420, computer 105 determines whether to begin the first time window of the data collection period based on the second derivative of the radar data, i.e., whether to designate the current time as the start time of the first time window, as described above. Upon designating the start time, computer 105 sets a flag in memory indicating that a data collection period is in progress, and process 400 proceeds to block 425. In response to the current time not being at the start time, process 400 returns to block 405 to continue receiving radar data.

[0058] In block 425 , computer 105 stores the radar data in a buffer in the memory of computer 105 .

[0059] Next, at decision block 430, computer 105 determines whether the current time window has reached its end time, i.e., the end of its preset duration. As discussed above, this end may correspond to a full memory buffer. In response to the time window not being complete, process 400 returns to block 405 to continue filling the buffer with radar data. In response to the current time being at the end time, process 400 proceeds to block 435.

[0060] In block 435 , computer 105 converts the radar data received during the time window to the frequency domain, as described above, and stores the converted radar data elsewhere in memory.

[0061] Next, in block 440 , the computer 105 clears the buffer, ie, deletes the time-domain radar data stored in the buffer, to make room for continued radar data collection.

[0062] Next, in block 445 , computer 105 identifies motion inside passenger compartment 115 based on the radar data received during the time window, as described above.

[0063] Next, in decision block 450, computer 105 determines whether any conditions for actuating component 120 have been met, as described above. In response to one of the conditions being met, process 400 proceeds to block 455. In response to any of the conditions not being met, process 400 proceeds to decision block 460.

[0064] In block 455 , the computer 105 actuates the component 120 of the vehicle 100 based on the identification of the motion, as described above. Following block 455 , the process 400 proceeds to decision block 460 .

[0065] In decision block 460, the computer 105 determines whether the criteria for stopping the data collection period have been met, e.g., the second derivative being below a threshold amount for at least a threshold duration, as described above. In response to the criteria not being met, the process 400 proceeds to block 465. In response to the criteria being met, the process 400 proceeds to decision block 470.

[0066] In block 465, computer 105 sets a flag in memory to indicate that a data collection period is not in progress. Computer 105 thus avoids using radar data received after the current time window to identify motion within passenger compartment 115. After block 465, process 400 proceeds to decision block 470.

[0067] In decision block 470 , computer 105 determines whether vehicle 100 is still on. In response to vehicle 100 being still on, process 400 returns to block 405 to continue collecting radar data. In response to vehicle 100 being off, process 400 ends.

[0068] Generally speaking, the computing systems and / or devices described may utilize any of a variety of computer operating systems, including but not limited to the following versions and / or types: Ford Application; AppLink / Smart Device Link middleware; Microsoft Operating system; Microsoft Operating systems; Unix operating systems (e.g., those distributed by Oracle Corporation of Redwood Shores, California) operating systems); the AIX UNIX operating system distributed by International Business Machines Corporation of Armonk, New York; the Linux operating system; Mac OSX and iOS operating systems distributed by Apple Inc. of Cupertino, California; the BlackBerry operating system distributed by BlackBerry Ltd. of Waterloo, Canada; and the Android operating system developed by Google Inc. and the Open Handset Alliance; or provided by QNX Software Systems CAR Infotainment Platform. Examples of computing devices include, but are not limited to, an in-vehicle computer, a computer workstation, a server, a desktop, notebook, laptop or handheld computer, or some other computing system and / or device.

[0069] Computing devices typically include computer-executable instructions, which can be executed by one or more computing devices such as those listed above. Computer-executable instructions can be compiled or interpreted from computer programs created using a variety of programming languages and / or technologies, including, but not limited to, Java, PHP, and others, either alone or in combination. TM , C, C++, Matlab, Simulink, Stateflow, Visual Basic, Java Script, Python, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines such as the Java virtual machine and the Dalvik virtual machine. Typically, a processor (e.g., a microprocessor) receives instructions from, for example, a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using a variety of computer-readable media. Files in a computing device are typically a collection of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0070] Computer-readable media (also known as processor-readable media) include any non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including but not limited to non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fiber, wires, wireless communications, including internal components that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0071] The databases, data repositories or other data stores described herein may include various mechanisms for storing, accessing / retrieving various data, including hierarchical databases, sets of files in a file system, application databases in specialized formats, relational database management systems (RDBMS), non-relational databases (NoSQL), graph databases (GDB), and the like. Each such data store is typically included in a computing device employing a computer operating system such as one of the above-mentioned and is accessed via a network in any one or more of a variety of ways. The file system can be accessed from the computer operating system and may include files stored in various formats. In addition to languages for creating, storing, editing, and executing stored programs (e.g., the PL / SQL language described above), RDBMS also typically employs structured query language (SQL).

[0072] In some examples, system elements can be implemented as computer-readable instructions (e.g., software) stored on computer-readable media (e.g., disks, memories, etc.) associated with one or more computing devices (e.g., servers, personal computers, etc.). A computer program product may include such instructions stored on a computer-readable medium for performing the functions described herein.

[0073] In the accompanying drawings, like reference numerals indicate like elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes may be practiced by performing the steps in an order different from that described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.

[0074] The present disclosure has been described in an illustrative manner, and it should be understood that the terminology used is intended to be descriptive rather than restrictive. The use of "in response to," "after determining...", and the like indicates a causal relationship, not just a temporal relationship. The adjectives "first" and "second" are used throughout this document as identifiers and are not intended to denote importance, order, or quantity. In light of the above teachings, many modifications and variations of the present disclosure are possible, and the present disclosure may be practiced in other ways than specifically described.

[0075] According to the present invention, a computer is provided, the computer having a processor and a memory, the memory storing instructions executable by the processor to: receive radar data from an ultra-wideband radar in a passenger compartment of a vehicle; determine a start time of a time window based on a second-order derivative of the radar data; identify motion inside the passenger compartment based on the radar data received during the time window; and actuate a component of the vehicle based on the identification of the motion.

[0076] According to one embodiment, the second derivative of the radar data is a second derivative of fast time with respect to slow time.

[0077] According to one embodiment, the instructions further include instructions for refraining from using the radar data received before the start time to identify the motion inside the passenger compartment.

[0078] According to one embodiment, the instructions further include instructions for converting the radar data received during the time window into a frequency domain and identifying the motion in the passenger compartment based on the radar data in the frequency domain.

[0079] According to one embodiment, the instructions further include instructions for converting the radar data received during the time window into the frequency domain by applying a Fast Fourier Transform.

[0080] According to one embodiment, the time window has a preset duration.

[0081] According to one embodiment, the instructions further include instructions for applying a filter to the radar data, and the second order derivative is a second order derivative of the radar data after applying the filter.

[0082] According to one embodiment, the filter is a smoothing filter.

[0083] According to one embodiment, the filter is a bandpass filter that isolates the frequencies of human breathing.

[0084] According to one embodiment, the time window is a first time window, the start time is a first start time, and the instructions also include instructions for performing the following operations: identifying the movement in the passenger compartment based on the radar data received during a second time window, and the second time window starts at the end time of the first time window.

[0085] According to one embodiment, the instructions further include instructions for, in response to the second order derivative being below a threshold amount for at least a threshold duration, refraining from using the radar data received after a current time window to identify the motion inside the passenger cabin.

[0086] According to one embodiment, the instructions further include instructions for classifying the motion as a type of animate, integral motion based on the radar data received during the time window.

[0087] According to one embodiment, the instructions further include instructions for classifying the motion as a type of animate, integral motion by executing a neural network classifier using as input the radar data received during the time window.

[0088] According to one embodiment, the instructions further include instructions for identifying a plurality of breathing individuals based on the radar data received during the time window.

[0089] According to one embodiment, the instructions further include instructions for commanding a user interface to output an alert to an occupant of the passenger compartment based on the recognition of the motion.

[0090] According to the present invention, a method includes: receiving radar data from an ultra-wideband radar in a passenger compartment of a vehicle; determining a start time of a time window based on a second-order derivative of the radar data; identifying motion inside the passenger compartment based on the radar data received during the time window; and actuating a component of the vehicle based on the identification of the motion.

[0091] In one aspect of the invention, the second derivative of the radar data is a second derivative of fast time with respect to slow time.

[0092] In one aspect of the invention, the method includes refraining from using the radar data received before the start time to identify the motion inside the passenger compartment.

[0093] In one aspect of the invention, the method includes converting the radar data received during the time window into a frequency domain; and identifying the motion in the passenger compartment based on the radar data in the frequency domain.

[0094] In one aspect of the invention, the method includes commanding a user interface to output an alert to an occupant of the passenger compartment based on the identification of the motion.

Claims

1. A method comprising: receiving radar data from an ultra-wideband radar in a passenger compartment of a vehicle; determining a start time of a time window based on a second derivative of the radar data; identifying motion inside the passenger cabin based on the radar data received during the time window; as well as A component of the vehicle is actuated based on the identification of the motion. 2 . The method of claim 1 , wherein the second derivative of the radar data is a second derivative of fast time with respect to slow time.

3. The method of claim 1 , further comprising refraining from using the radar data received before the start time to identify the motion within the passenger compartment.

4. The method of claim 1, further comprising: converting the radar data received during the time window into a frequency domain; and identifying the motion in the passenger compartment based on the radar data in the frequency domain. The method of claim 1 , wherein the time window has a preset duration. 6 . The method of claim 1 , further comprising applying a filter to the radar data, wherein the second-order derivative is a second-order derivative of the radar data after applying the filter. The method of claim 6 , wherein the filter is a smoothing filter.

8. The method of claim 6, wherein the filter is a bandpass filter that isolates frequencies of human breathing.

9. The method of claim 1 , wherein the time window is a first time window, and the start time is a first start time, the method further comprising: The motion in the passenger compartment is identified based on the radar data received during a second time window, the second time window beginning at the end time of the first time window.

10. The method of claim 9, further comprising refraining from using the radar data received after a current time window to identify the motion inside the passenger cabin in response to the second order derivative being below a threshold amount for at least a threshold duration.

11. The method of claim 1 , further comprising classifying the motion as a type of animate, integral motion based on the radar data received during the time window.

12. The method of claim 11, further comprising classifying the motion as a type of animate whole body motion by executing a neural network classifier using the radar data received during the time window as input.

13. The method of claim 1, further comprising identifying a plurality of breathing individuals based on the radar data received during the time window.

14. The method of claim 1, further comprising commanding a user interface to output an alert to an occupant of the passenger compartment based on the identification of the motion.

15. A computer comprising a processor and a memory, the memory storing instructions executable by the processor to perform the method of one of claims 1 to 14.

Images