System and method for monitoring a vehicle cabin

Abstract

The present invention provides a system and method for monitoring a vehicle cabin. Vehicle cabin monitoring uses a radar unit positioned in the center of the cabin to obtain image data of the vehicle cabin and a processor to detect occupancy of seats in the vehicle cabin, classify the occupants, detect postures, determine seat belt status, and monitor the vital signs of the occupants. An output unit may perform responses suitable for the status of the occupants of the vehicle cabin.

Description

[0001] Cross - reference to related applications

[0002] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 020,691, filed on May 6, 2020; U.S. Provisional Patent Application No. 63 / 016,314, filed on April 28, 2020; U.S. Provisional Patent Application No. 63 / 056,629, filed on July 26, 2020; U.S. Provisional Patent Application No. 63 / 049,647, filed on July 9, 2020; and U.S. Provisional Patent Application No. 63 / 135,782, filed on January 11, 2021, the contents of which are incorporated herein by reference in their entirety. Technical field

[0003] The present disclosure relates to systems and methods for monitoring a vehicle's cabin based on radar. Specifically, but not exclusively, the present disclosure relates to detecting the occupancy, posture, and classification of a vehicle's occupants, and controlling vehicle systems based on monitored parameters such as the mass, body type, or orientation of the occupancy object. Background art

[0004] It is important to know how many occupants are in a vehicle and where they are sitting.

[0005] For various reasons, it is useful to know whether a vehicle's seat is occupied.

[0006] Modern vehicle implementations include an excessive number of sensors for determining vehicle occupancy. These sensors include sensors for identifying whether each seat is occupied, whether an infant is in the front seat to disable the airbag, etc.

[0007] There are limitations on the number of passengers a vehicle is allowed to carry. There are also size and weight limitations regarding different seats in a vehicle. For example, passengers below a certain age are not allowed to sit in the front seat.

[0008] In some jurisdictions, to encourage carpooling, private vehicles are allowed to use preferential roads such as bus lanes if they carry at least a certain number of passengers in addition to the driver.

[0009] Broadband MIMO radar devices based on compact antenna arrays are currently used in various imaging applications to visualize near - field and far - field objects and characterize these objects based on their reflection characteristics.

[0010] The current state-of-the-art technology uses MIMO radar signals to create 3D images. However, current MIMO imaging techniques lack the ability to achieve sufficient resolution when identifying targets that are very close to each other or targets found in a mobile environment.

[0011] Complex target objects with different parts, each having its own movement pattern, also complicate imaging with sufficient resolution. Therefore, it is necessary to advance current MIMO radar imaging techniques to achieve a higher degree of resolution.

[0012] Having a large number of sensors to monitor the interior of a vehicle leads to cost issues and reliability issues. A simpler system is needed to obtain the same information. This disclosure addresses this need.

[0013] Automobile manufacturers use a large number of sensors to monitor vehicle performance and safety purposes. Monitoring the occupancy of the cabin and especially whether the seats of the vehicle are occupied and whether there are young children in the vehicle is also useful. For example, pressure sensors are used to detect occupancy within the vehicle, position detectors detect the configuration and position of the vehicle seats, and detectors are used to detect infants left in the vehicle, etc.

[0014] Having a large number of sensors to monitor the interior of a vehicle leads to cost issues and reliability issues. In fact, some modern vehicles include an excessive number of sensors for determining the occupancy of the vehicle. Additionally, these sensors do not identify and distinguish passengers based on their age and body structure. These sensors include sensors for identifying whether each seat is occupied, whether an infant is in the front seat to disable the airbag, etc. It is desirable to obtain the same information with a simpler system.

[0015] Therefore, a simpler system is needed to replace vehicle sensors and operate the safety system based on the seat position of the passengers. The present invention described herein addresses the above needs. SUMMARY OF THE INVENTION

[0016] A first aspect of an embodiment relates to a radar sensor array that is mounted in a position that allows monitoring of the cabin of a vehicle and its occupants.

[0017] The sensors can be located in a slightly central position on the roof of the vehicle to monitor the driver and the front seats, as well as the rear seats.

[0018] The radar sensor array can be positioned behind the roof interior trim or in a box below the roof.

[0019] Alternatively, the sensors can be mounted at the height of the windshield or the rear window.

[0020] In other embodiments, the radar sensor can be incorporated into the headrest, for example, at the central position of the headrest, such as the headrest of the driver's seat. In appropriate cases, the dual-sided sensor can include transceiver arrays facing forward and backward, thus providing 360-degree coverage of the entire vehicle.

[0021] The radar sensor array can be part of a radar-on-chip device that includes a processor, a memory, and data output to the vehicle.

[0022] The radar sensor array is configured to monitor the cabin and the objects and passengers within the cabin, and can distinguish different types of passengers, such as adults, children, infants, pets, and inanimate objects.

[0023] In addition, the detected data includes macroscopic and slight movements over time, and can monitor posture, gestures, breathing, and heart rate.

[0024] In some embodiments, the radar sensor array can also be operated only every few minutes, such as in response to vehicle movement or stop, or door opening or closing.

[0025] Additionally, it can be configured to communicate with a remote server via a communication network, communicate via the cloud or the Internet of Things, to provide detailed information about the occupants and their behaviors to fleet operators, rental vehicle providers, the police, emergency rescue services, etc.

[0026] In some embodiments, the radar sensor array operates continuously, at least when the vehicle is in use, and experiences active periods and idle periods in continuous cycles. In other embodiments, the vehicle's central computer wakes up the sensor under specific conditions, such as in response to door opening or closing, speed changes, detected road conditions, etc.

[0027] By operating in pulse mode, with a data processing period and an idle period following the pulse signaling period, the generation of heat can be controlled.

[0028] In the case of being connected to the vehicle via a data conduit line, the data conduit line can be used to extract heat from the radar sensor array.

[0029] A metal radiator can also be attached to the sensor array, which extends to the outside of the vehicle and is cooled by air passing over the radiator, or it can be connected to the vehicle's metal frame to extract heat.

[0030] Some embodiments relate to sensors being embedded in glass components such as sunroofs, or alternatively being embedded in the windshield or rear window of a vehicle. This allows for a single mounting location, maximizing performance and minimizing the installation costs incurred across various vehicle models.

[0031] As used herein, the term sunroof includes partial glass roofs, panoramic glass roofs, and window panels in a roof.

[0032] A radar sensor array can be integrated into a sunroof and embedded within the material of the sunroof, or can be sandwiched between layers or between laminated sunroofs having at least an upper layer and a lower layer.

[0033] Alternatively, the radar sensor array can be attached to the underside of the sunroof.

[0034] Also alternatively, the radar sensor array can be embedded within a chamber in the sunroof, possibly embedded in a thermoplastic or epoxy resin preferably having high thermal conductivity.

[0035] Such embodiments are characterized in that the sunroof is made of a glass material having good heat dissipation characteristics. The sunroof conducts heat away from the sensor, and with the large surface area of the sunroof, it is easily cooled. In fact, the outer surface of the sunroof is convectively cooled by the movement of the vehicle.

[0036] One aspect of the present invention relates to a sunroof of a vehicle having an integrated radar sensor array for monitoring passengers in the vehicle.

[0037] Other embodiments relate to the radar sensor array of the present invention being attached within glass headlight units, mirrors, windshields, and rear windows for monitoring the exterior of the vehicle. Here too, large glass or other thermally conductive surfaces can act as large heat sinks, preventing the radar unit from overheating. Also, the radar sensor array can be integrated into a chip together with a memory and a digital signal processor, and the chip can be integrated into a headlight, taillight, or indicator unit by being embedded within a chamber or laminated between an inner layer and an outer layer, or simply adhered to the inner surface in a desired position and orientation, or can be integrated into a window such as the windshield, rear window, or side window of a vehicle.

[0038] In one aspect of the present invention, a skeletal key point detection system for detecting occupancy information in a vehicle is disclosed. Seat occupancy provides information on the age category of the passengers on each seat as well as whether they are seated or out of position. The system includes a radar unit, a preprocessor unit, a database, a processing unit, and one or more output units.

[0039] In another aspect of the present invention, the radar unit includes a transmitter array and a receiver array, which are configured to emit electromagnetic radiation beams to vehicle passengers and receive electromagnetic waves reflected by the passengers, respectively. The preprocessing unit receives electromagnetic signals from the radar receiver and extracts human key points using a trained deep neural network (DNN). The extracted PKPs are used to identify the skeletal points of the passengers at each seat of the vehicle. The identified key skeletal points at each seat of the vehicle are sent to the processing unit.

[0040] In another aspect of the present invention, the processing unit includes a matching unit, a rule database, and a communicator. The matching unit compares the key skeletal points received from the preprocessing unit with the standard passenger parameters received from the database and determines the occupancy information of each seat of the vehicle based on the comparison. Then the occupancy information is transmitted to the rule database, which determines the actions for each seat based on the received information. Then the determined actions are transmitted to one or more output units through the communicator.

[0041] In another aspect of the present invention, the human key points (PKPs) may include information on the passenger's head, left and right shoulders, left and right outer points of the lower abdomen or pelvis, and points on the left and right knees or thighs.

[0042] In yet another aspect of the present invention, the occupancy information includes the occupancy of each seat, the age category of each occupant, and the detection of whether each occupant is properly seated or misaligned. Description of the Drawings

[0043] To better understand the embodiments and to show how these embodiments may be implemented, reference will now be made to the drawings by way of example only.

[0044] Now referring specifically to the drawings, it is emphasized that the details shown are by way of example only and are for illustrative discussion of the selected embodiments and for the purpose of presenting what is considered to be the most useful and readily understood description of the principles and conceptual aspects. In this regard, no attempt is made to show structural details in more detail than is necessary for a basic understanding; it will be apparent to those skilled in the art from the description how the various selected embodiments may be put into practice. In the drawings:

[0045] Figure 1 A schematic diagram of a vehicle cabin is shown, which shows where the radar sensor array may be located to track the position and movement of passengers;

[0046] Figure 2 Is a schematic flowchart showing an exemplary method for determining the seat occupancy information of a vehicle according to an aspect of the present invention;

[0047] Figure 3 is a schematic block diagram of an element of an embodiment of the present invention;

[0048] Figure 4A is a schematic view of a ceiling of a vehicle having a sunroof, to which a radar sensor array (typically radar-on-chip) is attached;

[0049] Figure 4B is a schematic side view of the sunroof, which shows the radar chip embedded within the material of the sunroof;

[0050] Figure 4C is a schematic side view of the sunroof, which includes an upper layer and a lower layer and has a radar chip encapsulated between the upper layer and the lower layer;

[0051] Figure 4D is a schematic side view of the sunroof, which shows the radar chip attached to the lower surface of the sunroof;

[0052] Figure 4E is a schematic side view of the sunroof, which shows the radar chip embedded within a chamber within the sunroof;

[0053] Figure 5 is a schematic representation of the radar chip being embedded within a headrest;

[0054] Figure 6A is a flowchart showing the general operation of an embodiment of the system;

[0055] Figure 6B and Figure 6C show exemplary seating of a passenger in a vehicle; and

[0056] Figure 6D 、 Figure 6E 、 Figure 6F 、 Figure 6G and Figure 6H show exemplary misalignment seating of a passenger in a vehicle;

[0057] Figure 7 is a flowchart showing a method for determining seat occupancy information of a vehicle;

[0058] Figure 8 is a schematic diagram of the hardware employed in a MIMO detection system according to an embodiment of the present invention;

[0059] Figure 9A is a general flowchart showing the processing steps employed according to an embodiment of the present invention;

[0060] Figure 9B is a general flowchart showing the general processing steps employed according to an embodiment of the present invention;

[0061] Figure 9C is a flowchart showing the processing steps employed in the first embodiment of the radar signal processing stage according to an embodiment of the present invention;

[0062] Figure 9D is a flowchart showing the processing steps employed in the second embodiment of the radar signal processing stage according to an embodiment of the present invention;

[0063] Figure 9E is a flowchart showing the processing steps employed in the third embodiment of the radar signal processing stage according to an embodiment of the present invention;

[0064] Figure 9F is a flowchart showing the processing steps employed in the target processing stage according to an embodiment of the present invention;

[0065] Figure 10A is a plot of the radial displacement as a function of time for the measurement of a human object according to an embodiment of the present invention;

[0066] Figure 10B depicts two plots of the spectral power density for two identified elements according to an embodiment of the present invention;

[0067] Figures 11A to 11E depicts image products at various stages of processing a passenger sitting in an automotive interior environment according to an embodiment of the present invention; and

[0068] Figure 12 is a diagram depicting the operating phases of an operating cycle employed during an activity detection mode according to an embodiment of the present invention;

[0069] Figure 13 is a flowchart showing how to use a 3D complex radar image of a vehicle's cabin to extract data on which seats are occupied and to classify the occupants of each occupied seat;

[0070] Figure 14A is a flowchart showing singular value decomposition filtering;

[0071] Figure 14B is a flowchart showing continuous spatio-temporal filtering;

[0072] Figure 14C represents a schematic diagram of a filtering step;

[0073] Figure 15A is a two-dimensional map of the area around a central radar sensor showing SVD components;

[0074] Figure 15BShows a two-dimensional map of the area around the central radar sensor after performing DBSCAN clustering;

[0075] Figure 16 Shows a two-dimensional map of the area around the central radar sensor after performing spectral clustering;

[0076] Figure 17 Represents the clustering of points as a Gaussian distribution in three-dimensional space;

[0077] Figure 18 Shows clusters that clearly represent different occupants, with the positions of the seats in the vehicle cabin superimposed thereon;

[0078] Figure 19 Shows a seat arrangement that shows the arrangement of seats corresponding to the Figure 18 seat arrangement;

[0079] Figure 20 Shows the intermediate position between seats 3 and 4 and the intermediate position between seats 5 and 6;

[0080] Figure 21 Is a conversion model that shows the effective state transitions between the rear seats of the vehicle;

[0081] Figure 22 Is a flowchart showing how occupants can be classified;

[0082] Figure 23 Shows the positive cov in the xy plane;

[0083] Figure 24 Shows the negative cov in the xy plane; and

[0084] Figure 25 Is a side view of seat 5 that shows the upper and lower boundaries of the signal cluster and the forward and backward majority boundaries interpreted as occupants. Detailed Description

[0085] Embodiments of the present invention use a single sensor to track both occupancy and movement within the vehicle cabin. To monitor passengers within the vehicle cabin using a single sensor, the possible location of the sensor can be in the center of the cabin such that as many positions of the cabin as possible are within the sensor's target range. Alternatively, this can be located in the ceiling, within the seat, within the headrest, embedded in the window, embedded in the sunroof, embedded in the lighting unit, etc., as described herein.

[0086] For example, the sensor can be hidden behind the fabric of the cabin lining or within the upholstery of the seat. Additionally or alternatively, the sensor can be positioned outside the fabric within the housing.

[0087] The sensor unit can detect presence in the vehicle cabin, covering both the front and rear seats. Using a single sensor unit in this way eliminates the need to use separate sensor units in each row of seats or each seat and replace other sensors, saving labor and wiring costs during manufacturing.

[0088] Aspects of the present disclosure relate to systems and methods for determining seat occupancy information in a vehicle using radar sensors. Seat occupancy can provide information on the age category and seated or misplaced position of passengers in each seat. This information can help operate safety equipment and track any passengers left in the vehicle.

[0089] In various embodiments, the sensors and supporting data analysis determine the shape, (volume and size), position, posture of each occupant, track their movement within the vehicle's cabin and monitor vital signs. Thus, a single sensor arrangement provides multi-functional monitoring of the vehicle cabin and replaces the excess of sensors required heretofore.

[0090] As required, detailed embodiments of the present invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely examples of the present invention, which may be embodied in various forms and alternative forms. The drawings are not necessarily drawn to scale; certain features may be exaggerated or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to use the present invention in various ways.

[0091] In various embodiments of the present disclosure, one or more tasks as described herein may be performed by a data processor such as a computing platform or distributed computing system for executing multiple instructions. Optionally, the data processor includes or accesses a volatile memory for storing instructions, data, etc. Additionally or alternatively, the data processor may access a non-volatile storage device, such as a magnetic hard disk, a flash drive, a removable medium, etc., for storing instructions and / or data.

[0092] It is particularly noted that the systems and methods disclosed herein may not be limited in their application to the details of construction and arrangement of components or methods set forth in the description or shown in the drawings and examples. The systems and methods disclosed herein are capable of other embodiments or can be practiced and performed in various ways and technologies.

[0093] Alternative methods and materials similar or equivalent to those described herein may be used in the practice or testing of the embodiments of the present disclosure. However, the specific methods and materials described herein are for illustrative purposes only. Materials, methods, and examples are not intended to be necessarily limited.

[0094] Reference Figure 1 , shows a schematic diagram of a vehicle 150. Inside the vehicle, there is a driver 152, a child 154 sitting in the front passenger seat, a passenger 156 traveling in the rear seat behind the front passenger seat, and an empty seat 158 behind the driver.

[0095] Also shown is a radar sensor array 160 that is located inside the passenger compartment 165 of the vehicle and is positioned centrally, such as on the ceiling of the vehicle 160, or alternatively within seats, headrests, etc. The radar sensor array 160 monitors changes within the passenger compartment 165 of the vehicle 150. As shown, the radar sensor 160 is positioned centrally. This is a preferred location because it can provide a view that includes the passenger compartment and can cover all seats well. Alternatively, the sensor can be positioned slightly off-center and will still provide the information it needs to replace sensors that are currently dedicated to specific individuals. Thus, the sensor array 160 can be mounted, for example, at the top of the windshield or rear window. In appropriate cases, a dual-sided sensor can include transceiver arrays facing forward and backward, thereby providing a 360-degree coverage of the entire vehicle.

[0096] The internal radar sensor array 160 can be an integrated system, such as the chip 160 described above. However, it should be understood that in various embodiments, one or more of the tasks described herein can be performed by an external data processor, such as a vehicle's computing platform or a distributed computing system, for executing multiple instructions. Optionally, the data processor includes or accesses a volatile memory for storing instructions, data, etc. Additionally or alternatively, the data processor can access a non-volatile storage device, such as a magnetic hard disk, a flash drive, a removable medium, etc., for storing instructions and / or data.

[0097] A preferred embodiment uses a radar sensor array integrated into a chip together with a digital signal processor (DSP) and a memory. For example, one embodiment uses a 4D imaging MIMO radar chip that has a global frequency band (60Ghz or 79GHz), thousands of virtual channels, a wide field of view - both angular and in distance - for both on-axis and high resolution. The radar is set on a chip (ROC), and the preferred embodiment covers a dual-band range, supporting both 60GHz and 79GHz frequency bands.

[0098] Another embodiment uses a sensor array that creates high-resolution images in real time based on advanced RF technology with a radar band from 3 GHz to 81 GHz. The sensor array has 72 transmitters and 72 receivers, and is integrated with a high-performance DSP with a large memory, which is capable of performing complex imaging algorithms without an external CPU.

[0099] Yet another embodiment only has a range from 60 GHz to 81 GHz, and 24 transmitters and 24 receivers.

[0100] With the development of more sophisticated data analysis tools, it is expected that the number of transceivers may be reduced, thereby reducing the unit cost without reducing functionality.

[0101] Due to the integration of multiple transceivers and the transmission, reception, and analysis of a large number of signals using an advanced DSP, high-resolution 4D images are obtained, and these 4D images track the contours with high precision.

[0102] Many applications are designed to alert the driver about an occupant not wearing a seatbelt, an unlocked door, and the inhibition of airbag deployment across from a seat with a young child. It is also possible to provide the driver with alerts about changes in the passenger's breathing or heart rate, or if the driver encounters difficulties, to alert the driver to pull over, or in extreme cases, automatic control may take over.

[0103] In some embodiments, a radar system or an in-vehicle computing system of a vehicle, with which the radar system communicates data, and which is configured to transmit information to a remote database supported by a cloud such as the Internet of Things through a data communication network such as a cellular network. This enables fleet operators and car rental companies to monitor usage, or enables police and emergency services to remotely monitor occupancy and passenger status.

[0104] Therefore, referring Figure 2 , a general embodiment 200 of the present invention includes a radar transceiver array 210, which is powered by a power supply 215 typically from the electronic system of a vehicle. In a vehicle designed to carry three passengers in the rear seats, the radar transceiver array 210 sends radar signals in all directions into the vehicle's cabin 225 and receives radar signals in all directions from the vehicle's cabin 225, and in particular sends and receives radar signals in all directions towards the driver's seat 222, the front passenger seat 224, the right rear seat 226, the left rear seat 228, and the middle rear seat(s) 227.

[0105] The radar transceiver array 210 detects elements in all directions within the cabin 165 of the vehicle 20, and each detected element has a spatial component and a temporal component, and signals by pulsed radar to detect changes in position over time.

[0106] The radar transceiver array 210 is coupled to a memory 214 for storing previous readings and by the spatial component of the detected signal, and the radar transceiver array 210 varies over time, i.e., their temporal component, and / or may also include a library of standard responses that indicate drivers and passengers of various body sizes sitting in the respective seats. The processing unit 212 is thus able to determine the presence of passengers in each direction and distinguish between adults, children and infants, pets, and inanimate objects by determining the body size, height of the occupant, whether the occupant is breathing or showing a heartbeat, etc. Since passengers are expected to be present in specific locations, i.e., seats, responses indicating adults, children, and toddlers or infants from various seat orientations can be usefully stored, and this limits the amount of processing required to obtain useful results.

[0107] The functions described are exemplary. The sensor array collects a large amount of data representative of the vehicle's cabin and its contents and monitors changes over time. The processor can be provided with additional algorithms and programs to analyze the image batches in different ways and add additional functions over time. Thus, since additional variables need to be monitored for legal, insurance, or other purposes, existing, installed sensors and processors may be further programmed to extract relevant parameters from the data.

[0108] The advantage of using radar signals for this purpose is that they are not blocked by most fabrics and are blocked by many non-fabric materials. However, it is worth noting that although passengers can be classified as adults or children and vital signs can be monitored, unlike optical cameras, the resolution is not sufficient to invade privacy.

[0109] Now refer to Figure 3 , which Figure 3 is a schematic block diagram of a system for radar-based monitoring in a vehicle. The system 300 includes a radar unit 304, a preprocessor unit 312, a database 314, a processing unit 316, and output units 324a and 324b.

[0110] The radar unit 304 is installed in a vehicle, such as an automobile. To monitor objects such as passengers within the vehicle cabin, the radar unit 304 should have a direct line of sight to the passengers, and in many cases, the optimal location can be the ceiling area, preferably the center of the ceiling.

[0111] The radar unit 304 includes a transmitter array 306 and a receiver array 310. The transmitter array 306 may include an oscillator 308 connected to at least one transmitter antenna or a transmitter antenna array. Accordingly, the transmitter 306 may be configured to generate a beam of electromagnetic radiation, such as microwave radiation, etc., and the beam of electromagnetic radiation is directed in all directions in the cabin 165 of the vehicle 150, including Figure 1 the driver's seat 153 and the passenger seats 155A-C as shown. Figure 3 The emitted electromagnetic waves towards the exemplary passengers 302a and 302b are shown. The receiver 110 may include a receiver antenna array that is configured and operable to receive the electromagnetic waves reflected from the bodies of the passengers 302a and 302b.

[0112] The radar receiver array 310 is coupled to a memory 326 that stores the signals received by the receiver 310. The memory 326 also stores the previous readings and spatial components of the detected signals, and varies over time, i.e., their time components, and / or may also include a standard response library that indicates drivers and passengers of various body types sitting in the respective seats. Additionally or alternatively, a neural network may be trained to identify and classify the passengers, mapping them to classifications such as age categories and in-seat / out-of-seat status.

[0113] The previous information stored in the memory 326 is transmitted to the preprocessing unit 312. The preprocessing unit 312 is thus able to determine the presence of passengers in each direction and distinguish between adults, children and infants, pets, and inanimate objects by determining the body type, height of the occupant, whether the occupant is breathing or showing a heartbeat, etc. Since passengers are expected to be found in specific locations, i.e., seats, it may be useful to store the responses indicating adults, children, and toddlers or infants from various seat directions, and this will limit the amount of processing required to obtain useful results. It should also be noted that in the case where multiple heartbeat or breathing frequencies are detected in a single location, this may be an indication, for example, that an infant or toddler is being held by an adult, even if the image of the toddler may be masked.

[0114] The processing unit 312 of system 300, whether integrated with the radar transceiver array 310 or in data communication with the radar transceiver array 310 and possibly part of an in-vehicle computer of a vehicle, can interact with in-vehicle output devices 318 such as warning lights or audible signals such as an alarm beep or verbal message. For example, it is noted that a passenger is not wearing a seatbelt, or that a passenger in the front seat 324 is too small to be safely belted. Additionally, the processing unit 312 can interact with an in-vehicle override device 316, for example, to deactivate a position where an airbag might be dangerous, such as the position 324 of an infant in the front passenger seat, or to adjust the tension of a seatbelt.

[0115] Additionally, the processing unit 312 can be coupled to a data transmitter 330 for transmitting occupancy-related data to the cloud. This data can be used by a fleet operator to monitor the number of passengers in a vehicle, and this data can be used by an emergency rescue system, etc.

[0116] The movement, breathing, and heart rate of each passenger can also be monitored.

[0117] Various systems and methods can be used to monitor vital signs such as a passenger's breathing rate and heart rate. As an example, a radar sensor can receive an energy signal reflected from an object within a target area such as a vehicle cabin, identify an oscillation pattern indicative of vital signs within a target area such as a vehicle cabin, and process the oscillation signal to isolate a breathing signal, a heart rate signal, etc.

[0118] In some embodiments, the processor unit collates a series of complex values for each voxel, which represent the reflected radiation of the associated voxel in multiple frames. Accordingly, for each voxel, the central point in the complex plane and the phase value for each voxel in each frame can be determined. In this way, a smooth waveform representing the phase change of each voxel over time can be generated, and a subset of voxels indicating periodic biological parameters can be selected such that desired vital sign metrics such as heart rate, heart rate variability, breathing pattern, etc. can be obtained.

[0119] An example of such a system is described in the applicant's co-pending international patent application with application number PCT / IB2021 / 051380, the entire content of which is incorporated herein by reference in its entirety.

[0120] It should also be noted that a radar sensor in the headrest of the driver's seat can be well positioned to monitor the vital signs of the driver by measuring the reflected radiation from the back of the driver's neck. In this way, the health and alertness of the driver can be monitored in a continuous manner.

[0121] The sensor chip 160 and the processing can operate only for a short period after an event such as closing the door, vehicle acceleration or stop. Where applicable, the system can operate in a pulse mode with short bursts, perform time-related sensing within a few milliseconds, and then perform calculations and analysis on the data for perhaps 10 milliseconds, but then the idle time may be three or four times the period during which the processor is actively sensing or calculating. This may be useful for saving power and preventing overheating by facilitating heat dissipation.

[0122] Wherein, a four-dimensional (4D) imaging MIMO radar is provided on the chip 10, the chip 10 uses a sensor array, the sensor array creates high-resolution images in real time based on advanced RF technology with a radar frequency band from 3 GHz to 81 GHz, the sensor array has 72 transmitters and 72 receivers, a high-performance DSP integrated with a large memory, and can execute complex imaging algorithms without an external CPU. It is understood that this processing generates heat that must be dissipated.

[0123] By operating in a pulse mode and having intermittent downtime, such as when enabling for dozens or hundreds of milliseconds to transmit and receive radar signals, and then processing the data, and then entering the idle mode for dozens or hundreds of milliseconds, heat can be dissipated to some extent, but heat dissipation may still be required.

[0124] In many vehicles, the optimal location of the sensor chip 160 is on the ceiling, but slightly forward to better monitor the driver's arms and feet without being obstructed by the driver's seat. However, in some vehicle models, there is a sunroof at this location. Other models have a panoramic glass ceiling.

[0125] For economic reasons, OEMs prefer to use the same sensor positioning for all vehicle variants to simplify manufacturing and reduce installation costs. This may result in the sensor being positioned in a less than ideal location, and as a result, some data cannot be collected. Or, to achieve full coverage, the manufacturer may choose to install two sensors, which doubles the component and installation costs and increases complexity and thus adverse reliability.

[0126] Embodiments relate to a sensor embedded in a glass component such as a sunroof or panoramic roof, or alternatively, a sensor embedded in the windshield or rear window of a vehicle. This allows for a single installation location, maximizing performance and minimizing the costs associated with installation for various vehicle models.

[0127] Reference Figure 4A , it has been unexpectedly found that the radar chip 410 (the radar chip 410 can be Figure 1a chip 160 and generally includes a radar transceiver array 412, a memory 414, a processing unit 412, and data input, and as Figure 2 shown, a data transmitter 230 to the cloud) embedded or attached to a vehicle 460, such as a skylight 450 in the roof of an automobile or car, helps with heat dissipation because the large glass object conducts heat away from the chip.

[0128] Manufacturing a skylight 450 with an embedded radar chip 410 makes the installation of the radar chip 410 particularly easy. Wires 412 that connect the sensor to the vehicle to provide data and power can also conduct heat away from the radar chip 410.

[0129] It should be understood that in some embodiments, the chip 410 is powered by a long - term button magnet and does not require wiring at all. The chip 410 can be configured to transmit signals to the vehicle using very little power because the transmission is over a short distance. Additionally, a solar panel can be provided on the chip 410 to recharge during daylight hours.

[0130] The skylight can be made of glass or a transparent polymer.

[0131] Reference Figure 4B , the chip 410 can be embedded in the skylight 450A by pouring the skylight 450A around the chip.

[0132] Alternatively, referring to Figure 4C , the skylight 450B can be made of an upper layer 440U and a lower layer 440L of glass or other materials, and the chip 410 can be positioned between the two layers of glass, such as the upper layer 440U and the lower layer 440L.

[0133] Again alternatively, as Figure 4D shown, the chip 410 can be adhered to the underside of the skylight 450C.

[0134] As Figure 4E shown, a chamber 413 can be provided in the skylight 450D, and the chip 410 can be positioned in the chamber 410 and optionally held in place with a thermally conductive epoxy or thermoplastic that is penetrable for radar frequencies and preferably is penetrable.

[0135] The method of attaching or embedding the radar transceiver sensor array described herein or the radar integrated on the chip to the skylight can be used to attach the radar transceiver sensor array and the radar chip to a vehicle, thereby also monitoring the near and far regions outside the vehicle.

[0136] For example, radar chips can be integrated into the vehicle's headlights, ambient lights, or indicator lights. These can be cast in glass or plastic and placed at the front and rear of the vehicle respectively.

[0137] Rearward radars can be useful when reversing or parking. Forward radars can be useful when driving.

[0138] Modern automotive headlight units and rear light units are typically large glass or plastic units that include various lights of different powers and uses, such as indicator lights, headlights, low beam lights, fog lights, etc. By integrating a radar sensor array into this unit, wires can be easily connected to the radar unit and unsightly attachments can be avoided.

[0139] Therefore, the radar sensor array of the present invention can be attached within glass headlight units, mirrors, windshields, and rear windows for monitoring the exterior of the vehicle. Here too, large glass or other thermally conductive surfaces can be used as large heat sinks to prevent the radar unit from overheating. Similarly, the radar sensor array can be integrated into a chip together with a memory and a digital signal processor, and the chip can be integrated into the headlight, rear light, or indicator light unit by being embedded in a cavity or laminated between an inner layer and an outer layer, or into a window such as the windshield, rear window, or side window of a vehicle, or the chip can simply be adhered to the inner surface in a desired position and orientation.

[0140] Now referring Figure 5 to, it should also be noted that another central location within the vehicle can be the seat itself. In particular, the driver's seat or passenger seat in the front row of a two-row cabin is roughly located in the center of the cabin. Even in larger vehicles, the front headrest can provide a good central location for mounting a radar sensor.

[0141] Accordingly, a radar sensing device incorporated into the headrest can be configured to transmit and receive towards the front and rear of the cabin. Various systems and methods can be used to provide a radar sensor with 360-degree coverage.

[0142] Specifically, it has been found that good heart rate signals have been observed when at least one sensor is directed towards the neck and upper back 505 of an object. This may be due to strong pulses passing through the carotid artery 506. Accordingly, as Figure 5 shown, it should be noted that a sensor device 517 located in the headrest of an automobile can be well positioned to monitor the vital signs of the occupant of the automobile seat. Such a sensor can also monitor the health and alertness of the driver.

[0143] For example, a bidirectional radar sensor may include a radar system mounted on a printed circuit board (PCB), the radar system having an array of transmit and receive antennas mounted on the front surface of the PCB to transmit and receive electromagnetic signals with a communication device on the same side as the front surface. The system may also include an array of receive antennas mounted at the edge of the PCB to receive electromagnetic signals reflected from an object within a target area to the side of the PCB.

[0144] In other systems, a reflector mounted on the PCB may be oriented such that an array of transmitter antennas emits waves perpendicular to the surface of the board, the waves impinging on the reflector surface, and the array of transmitter antennas is oriented radially away from the board, while waves received from an object within a target area and radially towards the reflector are oriented towards a receive antenna in a direction perpendicular to the PCB board. When necessary, a phase shifter may be used to compensate for different path lengths caused by the reflected waves.

[0145] A control chip may be configured and operable to control all active components, such as antennas, reflectors, and phase shifters of the PCB including transmit and receive antennas. When using bidirectional antenna elements, methods may be used to distinguish targets on both sides of the PCB.

[0146] An example of such a system is described in the co-pending international patent application with application number PCT / IB2021 / 051380 of the applicant, the entire content of which is incorporated herein by reference in its entirety.

[0147] It should also be noted that a headphone unit incorporating a radar sensor may be provided to be retrofitted to an automotive seat. Such a standalone module may include a communication unit for providing an interface with other modules such as a computing unit, a mobile phone, an in-vehicle infotainment system, etc. When needed, the headrest unit may also include a standalone power source such as an electrochemical cell, a solar panel, an inductive power receiver, etc., or may be configured to receive power from a vehicle power source.

[0148] Reference Figure 6A , shows a general method for using the system of the embodiment. First, a central radar transceiver array is set in the cabin of a vehicle - 602. The central radar transceiver array transmits radar signals 604 in all directions and receives reflections 606 from all directions, reflections from the walls, seats, and floors of the cabin, as well as reflections from the occupants, etc.

[0149] If a significant difference 608 is detected between the received signal from the seat direction and the signal expected from an empty seat, the signal may be analyzed or compared with signals for various targets such as adults, children, pets, infants, and inanimate objects 610.

[0150] The system is also capable of determining and classifying the movement of each occupant, such as including heartbeat and respiration.

[0151] In this way, the occupancy type 612 of each seat can be determined, so that the occupancy of each seat can be determined and classified, and appropriate actions 614 can be taken, such as warning the driver with a warning light or a sound signal, overriding, enabling or disabling certain components, such as airbags, and / or transmitting signals via a data network such as a cellular network or the Internet for use by fleet operators, emergency responders, the driver's family members, etc.

[0152] However, it should be noted in particular that the systems and methods disclosed herein may not be limited in their application to the details of the construction and arrangement of the components or methods set forth in the description or shown in the drawings and examples. The systems and methods in this disclosure are capable of having other embodiments, or capable of being practiced and carried out in various ways and techniques.

[0153] Alternative methods and materials similar or equivalent to those described herein can be used in the practice or testing of the embodiments of this disclosure. However, the specific methods and materials described herein are for illustrative purposes only. The materials, methods, and examples are not intended to necessarily be limiting. Accordingly, various embodiments may appropriately omit, replace, or add various procedures or components. For example, these methods can be performed in a different order than described, and various steps can be added, omitted, or combined. In addition, aspects and components described with respect to certain embodiments can be combined in various other embodiments.

[0154] The movement, respiration, and heart rate of each passenger can also be monitored, because the periodicity of the movement can be recognized as the respiration rate or heart rate.

[0155] Various systems and methods can be used to monitor vital signs, such as the respiration rate and heart rate of a passenger. For example, a radar sensor can receive an energy signal reflected from an object within a target area such as a vehicle cabin, identify an oscillation pattern within the target area such as a vehicle cabin that indicates vital signs, and process the oscillation signal to isolate respiration signals, heart rate signals, etc.

[0156] In certain embodiments, the processor unit collates a series of complex values for each voxel, which represent the reflected radiation of the relevant voxel in multiple frames. Accordingly, for each voxel, the central point in the complex plane and the phase value for each voxel in each frame can be determined. In this way, a smooth waveform representing the phase change of each voxel over time can be generated, and a subset of voxels indicating periodic biological parameters can be selected, so that desired vital sign indicators, such as heart rate, heart rate variability, respiration pattern, etc., can be obtained.

[0157] Examples of such systems are described in the applicant's co-pending international patent application PCT / IB2021 / 051380, the entire content of which is incorporated herein by reference in its entirety.

[0158] It should also be noted that a radar sensor in the headrest of the driver's seat can be well positioned to monitor the driver's vital signs by measuring the reflected radiation from the back of the driver's neck. In this way, the driver's health and alertness can be continuously monitored.

[0159] The background signal indicating an empty compartment is known. This corresponds to the average signal and can be subtracted from the detected signal to simplify the analysis.

[0160] Signals can be clustered by synchronous movement to identify individual passengers and detect gestures, etc. One aspect of the present invention is to provide a radar system that includes a central sensor that provides multi-dimensional time-related tracking within the vehicle compartment to determine the passengers within the compartment. Accordingly, the RADAR sensor unit is located in the ceiling of the vehicle to monitor the passengers in the front seats and rear seats of the vehicle. Generally, the sensor operates in a pulsed mode.

[0161] By clustering the response signals and mapping the response signals to known seat positions in the vehicle, the amount of processing required to detect and classify passengers can be limited.

[0162] The sensor is coupled to a processing unit and knows the response from an empty seat, which is the background response. By monitoring the difference between the response from the empty seat and the detected signal, and by comparing with a response library for infants, children, and adults, the occupant of the seat can be classified as an infant, a child, an adult, a pet, or an inanimate object.

[0163] Knowing the position of the seat, by applying a clustering algorithm and comparing with the stored data, passengers can be detected and classified more efficiently with less processing. Boxes can be quickly drawn around each passenger and used to determine how the passengers are seated.

[0164] Movements of the chest indicating breathing or detected heartbeats can be used to distinguish between living and inanimate objects that sometimes get placed and reflect signals different from those of an empty seat.

[0165] This information can be used to detect when a passenger is in distress or an infant has been left in the vehicle and, after a collision, to inform emergency responders that there is life in the vehicle and in which seat. After a collision, if the breathing or heartbeat of one or more passengers is detected intermittently or with difficulty, emergency responders can use this information to decide which passenger should be rescued and evacuated first.

[0166] The processing unit can use neural networks or fuzzy logic to determine the nature of the occupant.

[0167] The processing unit can be coupled to an indicator that is used to indicate to the driver that a seat is occupied, to deploy an airbag, to enable remote tracking of the passenger, and to alert emergency responders in the event of an accident.

[0168] In addition to the position of the passenger, the central radar sensor can also determine changes in the cabin. For example, if a door is opened.

[0169] The processing unit can be coupled to a database via cloud computing technology and is configured to update the database related to seat occupancy.

[0170] Details of seat occupancy may trigger an alarm if the seat belt is not fastened.

[0171] If the occupant of a seat is indicated as being too light or too short to sit safely in the seat, an appropriate notification can be issued.

[0172] This information may be sensitive enough to enable an intelligent seat belt pretensioner to adapt the seat belt to the passenger.

[0173] Detection of an infant can cause an in-vehicle notification to be displayed or the rear of the vehicle to be operated to notify other drivers to keep their distance.

[0174] By knowing the seating position of the passenger as well as the passenger's body size and shape, the deployment of the airbag in the event of an accident can be customized according to the position of the passenger. For example, if an infant is being carried, the airbag will not be deployed in the front passenger seat. Additionally, an empty seat does not require the deployment of an airbag. Knowing the positions of the heads of the passenger and the driver can enable the deployment of the airbag to be optimized, such as selectively deploying the airbag to better cushion the impact on the head.

[0175] In addition, the advantage of the radar solution is that privacy is protected because the identity of the passenger is not detected.

[0176] In addition to the position of the passenger, the central radar sensor can also determine changes in the cabin. For example, if a door is opened.

[0177] The processing unit can be coupled to the database through cloud computing technology and is configured to update the database related to seat occupancy.

[0178] Local laws or fleet owners require some drivers to be accompanied by a supervisor or a companion. For example, some military and police vehicles have regulations prohibiting the drivers of certain vehicles from driving alone. Other vehicles are not allowed to carry passengers. For example, it may be useful to ensure that new drivers are accompanied by an adult or not to transport other people.

[0179] In the past, a large number of individual sensors have been provided, each sensor being configured to detect one thing, such as a pressure sensor in a seat, a head position sensor for deploying an airbag, etc. Advantageously, from the perspective of unit cost as well as installation and wiring costs, the central sensor of the present invention can replace a large number of such dedicated single-task sensors.

[0180] However, it should be understood that the central sensor of the present invention can be used together with other sensors such as biometric sensors to identify the driver of the vehicle. In cases where the identified driver should not drive alone or should only drive with an accompanying adult, the system can generate a warning to the driver, or notify the supervisor, or can be configured to prevent the vehicle from moving.

[0181] To reduce the number of vehicles and encourage carpooling, during peak congestion periods, some transportation systems allow cars with 2, 3, or 4 occupants in addition to the driver to drive in lanes usually reserved for public transportation. Embodiments of the present invention track the occupancy of the vehicle and can be configured to provide occupancy levels to the municipal authorities, traffic police, etc. via the cloud.

[0182] In fact, even without dedicated lanes, embodiments of the present invention can be used to monitor the vehicle occupancy level, and this can be used to adjust the tolls charged for the use of roads or bridges, etc., or to reduce the annual road tax.

[0183] That a baby is known to be being transported by the vehicle can be transmitted to the cloud so that in the event of an accident, emergency responders know that a baby is being transported and know to look for the baby. In this regard, a baby in the footwell or hidden by a blanket can be detected by the baby's breathing or heartbeat.

[0184] In addition to tracking the position and height of the passengers, the preferred embodiments are configured to track the posture, movement, especially the breathing and heartbeat of the occupants. This can be valuable if an occupant encounters some trouble. For example, it can warn the driver to stop the vehicle.

[0185] After an accident occurs, the radar system can call the emergency rescue service, and the embodiment is operable to monitor the vital signs and indicate these to the emergency responders via the cloud.

[0186] Additionally, in the event of an accident, the radar system can activate an audible alarm and a public announcement system.

[0187] By tracking occupancy, taxi services and bus owners can ensure that drivers do not conceal the number of passengers carried in order to illegally obtain profits.

[0188] It is also possible to track the situation where two passengers are sharing a seat or a passenger is standing. This can be used to monitor the number of passengers in a driver's vehicle or an autonomous vehicle and can be used to prevent a vehicle from carrying more passengers than its permitted or legal capacity, or to report such usage.

[0189] In the case of carjacking or abandonment of the vehicle, the occupancy details during the last journey of the vehicle may provide important information for apprehending the perpetrator.

[0190] The system can be activated for a few minutes after the vehicle door is closed to detect a child locked inside the vehicle.

[0191] By monitoring the movement, breathing and heart activity of the baby and alerting the driver when something goes wrong, the driver can focus on the road.

[0192] If the driver's breathing or heart activity shows deviation from the normal values, the driver can be advised to pull over and / or the signal can be transmitted to the cloud, thereby alerting a third party such as the vehicle owner, spouse, parent, child or next of kin, highway police.

[0193] Preferably, the system is capable of detecting gesture signals, enabling the driver and potential passengers to control various systems such as the radio, air conditioner, etc. via the processing unit.

[0194] The system of the present invention can be provided to vehicles driven by humans as well as autonomous and semi-autonomous vehicles, and can be used by the driver to take over control from a semi-autonomous vehicle operating independently or for autonomous control to signal that such a situation has occurred or indicate that he / she is experiencing a certain health crisis such as a heart attack or seizure.

[0195] Now return to reference Figure 3 , the electromagnetic signal received by the receiver 310 is also sent to the preprocessing unit 312. The preprocessing unit 312 can be configured to extract human key points (PKP) from the received signal using a trained deep neural network (DNN). The extracted human key points (PKP) can include:

[0196] Head - the center of the top of the head

[0197] Left shoulder and right shoulder

[0198] Left and right outer points of the lower abdomen or pelvis

[0199] Points on the left and right knees, or thighs

[0200] The extracted PKP is used to identify the skeletal points of the passenger at each seat in the vehicle. In a particular embodiment of the present invention, the extracted key skeletal points can provide information in the following ways:

[0201] The top of the head can give the highest point of the body, which can be used for height calculation and dislocation detection.

[0202] The shoulders can allow determination of the width of the body.

[0203] The lower abdomen and pelvis can also define a square around the upper body, allowing determination of the actual sitting height. In addition, a baseline can be created for dislocation inspection (abdomen / pelvis line and shoulder line)

[0204] In addition to the abdomen and pelvis, the knees and thighs can also be used to assist in determination, for example, when any other points are missing, blurred, or not detected.

[0205] In addition, the knee points can be used to better distinguish children and adults. In particular, it is surprisingly found that since the legs reach the car floor, the adult knees are usually detected above the plane of the pelvis. In contrast, the knees of children are usually detected along or even below the line parallel to the floor from the pelvis.

[0206] The key skeletal points identified at each seat in the vehicle can be sent to the processing unit 316. The processing unit 316 includes a matching unit 318, a rule database 320, and a communicator 322. The matching unit 118 matches the key skeletal points received from the preprocessing unit 312 with the standard passenger parameters received from the database 314. The database 314 includes a standard list of PKP positions, distances, and ranges according to their age categories. Table 1 shows an exemplary category - age - height relationship:

[0207] Table 1

[0208]

[0209]

[0210] As shown in Table 1, child passengers in the MCD "mid-child" category are approximately 6 years old. Such children typically have a height of about 115 cm and a sitting height of about 63.5 cm. Similarly, older passengers in the ADT "adult" category are over 14 years old. Their typical height is over 160 cm and their typical sitting height is over 82 cm. It should be clearly noted that the category-age-height relationships shown in Table 1 are exemplary in nature and should not limit the scope of the present invention. The category-age-height relationships vary according to the demographics of each country and region.

[0211] Artificial key points can also be added by attaching reflective elements to the seat belt or the seat itself. The reflective element can be a retroreflective element, such as a corner reflector, Lunenberg lens, cat's eye retroreflector, or PCB-based equivalent, to enhance the RCS of the reflective element. For example, reflectors on the seat can provide information on whether they are blocked by the human body. Reflections from the seat elements can provide information on the vibrations of the vehicle during a ride. Reflectors on the seat belt can provide information on whether the seat belt is fastened. Additionally, if the seat belt is fastened, the reflective element on the seat belt becomes a PKP, which can be used to better track breathing and heartbeat by providing an enhanced and more stable signal from the reflector that moves in unison with the chest to re-evaluate the signal. The reflector can incorporate a modulation circuit, such as using reflective array technology, to impose features on the reflected signal to distinguish or discriminate between the reflectors from each other and other elements present in the vehicle cabin.

[0212] The matching unit 318 determines the occupancy information of each seat in the vehicle based on the comparison of the key skeletal points with the standard category-age-height relationships as shown in Table 1. The matching unit 318 is essentially a conclusion engine that includes heuristic codes and / or trained machine learning solutions. The conclusions output from the matching unit 318 can include: the occupancy of each seat, the age category of each occupant, or the detection of whether each occupant is properly seated or misaligned.

[0213] The matching unit or conclusion engine 318 determines whether the seats in the vehicle are occupied or empty. As Figure 1 shown, the front seats are occupied by the driver and the passenger. The rear seats are empty, and another seat is occupied by a passenger. In a particular embodiment, the movement of the chest indicating breathing or detected heat beats can be used to distinguish between living and inanimate objects that are sometimes placed and will reflect signals different from those of an empty seat.

[0214] The matching unit or conclusion engine 318 can also determine the age category of each passenger in the vehicle. In Figure 1Among them, the driver's age category can be determined as "ADT", adult, the passenger can be a child in the age category "MCD", and the passenger can be 50% male in the age category "50M".

[0215] The matching unit or the conclusion engine 318 can also assist in detecting the proper or improper seating of each occupant. Proper seating is defined as the normal seating position of a passenger in the vehicle. Figure 6B and Figure 6C An exemplary proper seating of a passenger in the vehicle is shown. In the proper or normal seating position, the passenger sits upright with the back straight and reclines on the seat. The parts of the legs above and below the knees are almost perpendicular to each other. Figures 6D to 6H Various illustrative improper seatings of passengers in the vehicle are shown.

[0216] The information of occupancy, age category, and misalignment thus determined can be transmitted to the rule database 320, which determines actions for each seat based on the received information. Then the determined actions are transmitted to the output units 324a and 324b through the communicator 322.

[0217] Some exemplary actions determined by the rule database 320 for a specific seat can include canceling the airbag operation in situations considered unsafe, for example, a child sitting in the front seat or someone putting their feet on the dashboard, the strength of the seatbelt retraction can be adjusted to suit the body size and posture of the occupant, and an alarm can be issued if a baby is left in the vehicle or the occupant is in an unsafe posture. The alarm can be provided to the in-vehicle output device 324a in the form of a warning light or a sound signal such as an alarm beep or a verbal message.

[0218] The occupancy information from the matching unit 318 can be used to detect when a passenger is in trouble or a baby is left in the vehicle, and after a collision, to notify emergency responders that there is life in the vehicle and in which seat. After a collision, if the breathing or heat beat of one or more passengers is detected intermittently or with difficulty, emergency responders can use this information to decide which passenger should be rescued and evacuated first.

[0219] Knowing that a baby is being transported by the vehicle can be transmitted to the cloud so that in the event of an accident, emergency responders know that a baby is being transported and know where to look for the baby. In this regard, a baby in the footwell or hidden by a blanket can be detected by its breathing or heartbeat.

[0220] In addition to tracking the position and height of the passengers, the preferred embodiment is configured to track the posture, movement, especially the breathing and heartbeat of the occupant. This can be valuable if the occupant encounters some trouble. For example, it can warn the driver to stop the vehicle.

[0221] After an accident occurs, the radar system can trigger the communication system to contact emergency rescue services, and the implementation is operable to monitor vital signs and indicate these to emergency responders via the cloud.

[0222] Additionally, in the event of an accident, the radar system can activate the alarm system to emit an audible alarm and activate the announcement system.

[0223] By tracking occupancy, taxi services and bus owners can ensure that drivers do not conceal the number of passengers carried in order to illegally obtain profits.

[0224] The sharing of seats by two passengers or the standing of passengers can also be tracked. This can be used to monitor the number of passengers in a driver's vehicle or an autonomous vehicle, and can be used to prevent a vehicle from carrying more passengers than its permitted or legal capacity, or to report such usage.

[0225] In the case of carjacking or abandonment of a vehicle, the occupancy details during the vehicle's last journey can provide important information for apprehending the perpetrator.

[0226] The system can be enabled for a few minutes after the vehicle doors are closed to detect children locked inside the vehicle.

[0227] Additionally, by monitoring the movement, breathing, and heart activity of an on-board infant and alerting the driver when something goes wrong, the driver can focus on the road.

[0228] If the driver's breathing or heart activity shows deviation from normal values, the driver can be advised to pull over and / or a signal can be transmitted to the cloud, thereby alerting third parties such as the vehicle owner, spouse, parents, children, or next of kin, highway police.

[0229] Output units 324a and 324b can be in-vehicle output devices such as a display or audio output. Alternatively, output units 324a and 324b can be remotely located in the form of external devices such as client devices, server devices, routing / switching devices, or cloud servers. Communicator 322 can transmit the determined actions to output units 324a and 324b via a network connection, which can be a wired LAN connection, wireless LAN connection, WiFi connection, Bluetooth connection, Zigbee connection, Z-Wave connection, or Ethernet connection. For example, the communicator transmits occupancy information and the determined actions to a cloud server. This data can be used by fleet operators to monitor the number of passengers in the vehicle, and this data can be used by emergency rescue systems, etc.

[0230] One or more of the preprocessing unit 312, the database 314, and the processing unit 316 may be integrated in the vehicle's system to process the information received from the radar unit 304. Alternatively, any one of these units may be integrated with an external device, such as a client device, a server device, a routing / switching device, or a cloud server. These units then communicate with the radar unit 104 via a network connection, which may be a wired LAN connection, a wireless LAN connection, a WiFi connection, a Bluetooth connection, a Zigbee connection, a Z-Wave connection, or an Ethernet connection.

[0231] Now referring Figure 7 , Figure 7 illustrates an exemplary method for determining seat occupancy information of a vehicle 150. The process begins at step 702 and at step 704, electromagnetic waves are transmitted by the transmitter array 306 towards the passenger seats 155A-C. At step 706, the waves reflected from the passenger seats 155A-C at step 704 are received by the receiver array 310. At step 708, the preprocessing unit 312 extracts human key points (PKP) from the received EM waves, and at step 710, key skeletal points are identified for each seat of the vehicle. The extracted human key points (PKP) may include information about the passenger's head, left and right shoulders, lower abdomen or the left and right outer points of the pelvis, and the left and right knees or points on the thighs.

[0232] At step 712, the key skeletal points are sent to the matching unit 318 of the processing unit 316. At step 714, the matching unit 318 compares the key skeletal points received from the preprocessing unit 112 with the standard passenger parameters received from the database 314. The database 314 includes a standard list of the positions and postures of passengers according to their age categories. At step 716, the matching unit 118 determines the occupancy information for each seat of the vehicle based on the comparison of the key skeletal points with the standard category-age-height relationships stored in the database 314. The conclusions output from the matching unit 318 may include the occupancy of each seat, the age category of each occupant, and the detection of each occupant being seated or out of position.

[0233] The information on occupancy, age category, and out-of-position is then transmitted to the rule database 320, which determines the actions for each seat based on the information received at step 718. At step 720, the determined actions are sent via the communicator 322 to the output units 324a and 324b. At step 722, the desired actions are executed by the output units 324a and 324b, and the process stops at step 724.

[0234] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the present invention. Those skilled in the art will understand that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

[0235] Embodiments of the present invention provide RF signal processing to detect and obtain measurements from one or more elements in at least one target object. In related embodiments, detection and measurement are further performed without having to isolate or identify one or more specific parts contributing to the relevant movement.

[0236] The term "complex target" herein refers to a target object having one or more parts that are not necessarily distinguishable only by their respective reflection characteristics (referred to herein as their respective "reflectivities"); rather, they are distinguishable by their associated movement or motion.

[0237] The identification of target objects is based on their elements having different motion patterns. Similarly, the identification of target objects from the background is achieved by contrasting their respective motion patterns.

[0238] For example, some applications provide further classification of human targets into categories such as "adult", "toddler", and the like.

[0239] Other applications provide identification of regions or parts of the body. The human body is modeled as a collection of rigid bodies (bones) connected by joints. The rigid bodies have the property that all points on the surface move in a related manner because they are all combinations of the 6 degrees of freedom of a rigid body. In these embodiments of the present invention, grouping related movements into elements helps in the identification of regions or parts of the body.

[0240] Other applications provide detection and measurement of body activities, which include but are not limited to: walking, running, jumping; coordinated movement of the limbs; carrying an object; turning the head; gestures; posture changes, and the like.

[0241] Further applications of the present invention provide detection of the relevant movement of an individual in a special environment having specific background characteristics and monitoring requirements, which specific environment includes but is not limited to: the interior of a vehicle and other mobile platforms; hospitals and other medical and care facilities; and public places, non-limiting examples of which include airports and other transportation stations; shopping malls, warehouses, and other commercial establishments; residential and office complexes; museums, theaters, and entertainment halls; parks, playgrounds, and stadiums; and institutions such as schools.

[0242] Additional applications of the present invention include: medical and health-related applications; security applications; crowd management applications; and vehicle safety and comfort applications.

[0243] According to various embodiments of the present invention, a complex object may include a human body. In these embodiments, parts of the body include, but are not limited to: the head, neck, individual limbs, and torso. In certain embodiments, physiological activities such as breathing and heartbeat are detectable and measurable without having to isolate or identify the area of the body (i.e., the torso) responsible for breathing and heartbeat.

[0244] The term "relative movement" herein includes the movement of a physical element of a set of objects relative to another physical element, a change in the volume of the element itself, a change in orientation, position, shape, contour, or any combination thereof.

[0245] The term "measurement" and its variants herein not only denote determining quantitative values (including multi-variable values), but also denote analyzing these values, particularly their temporal variations, and qualitatively characterizing them.

[0246] The term "voxel element" refers to an entity decomposed from a series of 3D images, each of which is associated with its respective frame.

[0247] It should be understood that terms are context-dependent. In the context of the physical domain, the same terms are used when referring to the signal or logical representation of the same entity.

[0248] Non-limiting examples of such qualitative characterizations relate to multi-variable physiological data, measurements such as the heartbeat and breathing of an object. These physiological activities can be detected and measured not only as raw data, but also the current physical and mental condition of the object can be qualitatively evaluated based on respiratory rate, heart rate, and heart rate variability as the measurement result. The mental condition is intended to include, among other conditions, the level of consciousness, drowsiness, fatigue, anxiety, stress, and anger.

[0249] Turning now to the drawings, Figure 8 is a schematic block diagram of a MIMO imaging device including an antenna array 2 coupled to a radio frequency (RF) module 1, the radio frequency (RF) module 1 being linked to a processor 6 that communicates with a memory 7 and an output device 9. The output device 9 includes visual, audio, wireless, and printing devices.

[0250] As shown, as the radial distance Dt varies over time, the reflective elements of the faces 4 of the set of target objects 3 provide different reflectivities. Analyzing the reflectivity data in view of the reflectivity data of previous time frames enables the detection of relevant movements, which advantageously provides a discrimination ability not currently available in MIMO imaging systems. This is because traditional MIMO imaging systems would repeatedly construct an image based on the reflectivity data of each time frame; independent of the reflectivity data of the previous time frame. Accordingly, using correlated motion as a discrimination tool constitutes an advancement in MIMO imaging technology.

[0251] Figure 9A is a high-level flowchart showing a general processing scheme according to an embodiment of the present invention. The scheme can be described as a closed loop, where each iteration of the loop includes a series of steps.

[0252] The loop starts at step 10, where the acquisition and processing of a new time frame begins. The frames start at a fixed interval of Δt (meaning the frame rate is equal to ). According to various embodiments of the present invention, Δt is selected such that the target movement ΔD during Δt is small compared to the wavelength of the radar signal (i.e., ) to maintain continuity from one frame to another. For a wave with a center frequency of f, the wavelength is c / f, where c is the speed of light. When detecting and measuring the periodic correlated movement of a target, imaging through a series of frames is a sampling process, such that the frame rate should be set according to the Nyquist criterion to avoid aliasing. The frames are indexed by t = 0, 1, 2... corresponding to time, where consecutive indices represent the respective multiples of Δt.

[0253] In step 20, the radar signals are transmitted, received, and processed to produce complex phasors that represent the amplitude and phase of each received signal relative to each transmitted signal. Step 20 is further detailed in Figure 9B .

[0254] In step 30, several signal processing steps are performed, resulting in a set of components, each component including a spatial pattern and a trajectory (displacement versus time). Step 30 is further detailed in Figure 9C .

[0255] In step 40, the components are used to identify the target, classify the target, and estimate the target parameters of interest. Step 40 is further detailed in Figure 9F .

[0256] In step 50, the identified target and its estimated parameters are used to interact with external systems, including but not limited to vehicle systems (e.g., to enable a horn, turn on the air conditioner, unlock the doors, etc.), to interact with a communication interface (e.g., to alert the user to use his mobile device), or to interact with a user interface (to inform the user and allow them to take action).

[0257] In step 60, end frame processing. In step 70, the enabled mode of the system is adjusted based on a timer, the identified target and its parameters, and user input. The system enabled mode controls parameters including but not limited to the number of frames the system captures per second (which determines Δt) and the transmission power. In some cases, the system is in a standby mode for a period of time. The enabled mode adjustment is made to conserve system power. When the next frame starts, the loop closes (according to the timing indicated by the enabled mode), and the system returns to step 10.

[0258] Figure 9B is a flowchart detailing the radar signal acquisition step (step 20) from Figure 9A In step 21, radar signals are transmitted from one or more antennas. If multiple antennas are used for transmission, the transmissions can be made sequentially (antenna by antenna) or simultaneously. In some embodiments of the present invention, the antennas transmit simultaneously using a coding scheme such as BPSK, QPSK, or other coding schemes known in the art. The transmission can include a single frequency, or the transmission can include multiple frequencies.

[0259] In step 22, radar signals that have been reflected by targets in the physical environment around the antennas are received by one or more antennas. Then in step 23, for each transmit frequency and each pair of transmit and receive antennas, the received signals are processed to generate complex phasors that represent the phase and amplitude of the received signals relative to the transmitted signals (item 24).

[0260] Figure 9C is a flowchart detailing the radar signal processing step (step 30) in an embodiment of the present invention from Figure 9A In step 31a, a 3D image is generated from a set of complex phasors describing the received signals. The image space is conceptually summarized as data block 32a, which contains an image matrix S = [S v，t with a set of voxels V, and the elements of the set of voxels V spatially conform to a coordinate system. The specific coordinate system for the set of voxels can be selected in the most convenient way. Conventional choices include Cartesian coordinates (v x，y，z ) and polar coordinates but any other coordinate system is equally available. Each voxel is associated with a single value where A v，t is the amplitude, and φ v，t is the phase associated with the reflector at voxel V. The phase φ v，t is determined by the radial displacement (designated as D v，t ) of the reflector in voxel V from the center of the voxel. The phase is related to the displacement by the following formula: where f refers to the center frequency. A single period extends over 2π radians, but an additional factor of 2 is needed because the reflection doubles the distance the wave travels.

[0261] In step 33a, the values (S v，t ) associated with each voxel of the current frame and the values (S v，t-1 ) associated with the same voxel in the previous frame are used together to obtain a robust estimate of the radial displacement between the two frames using the following formula:

[0262]

[0263] where λ and ∈ are real scalar parameters that are chosen to minimize the effect of noise on the final value. Typical values of λ and ∈ are small, where a reasonable value for λ is about 0.1, and for ∈ about 1×10 -8 .

[0264] According to another embodiment of the invention, a slightly modified version of the formula is used in order to provide better linearity of the estimated displacement:

[0265]

[0266] According to an embodiment of the invention, the estimated displacement data is recorded (item 34a) using a sliding window (which can be implemented using a circular buffer among other things), and in step 35a, the radial trajectory components are decomposed into independent elements using blind source separation (BSS, also known as "blind source separation"). In related embodiments, the elements of the radial trajectory are separated using independent component analysis (ICA), which is a special case of BSS. In another embodiment, the elements of the radial trajectory are separated by principal component analysis (PCA). In another embodiment, the elements of the radial trajectory are separated by singular value decomposition (SVD).

[0267] In another embodiment of the invention, an online decomposition algorithm is used that avoids using a sliding window and allows the separation of the elements to be performed incrementally frame by frame.

[0268] is a matrix whose rows represent voxels and whose columns represent frames. The decomposition algorithm extracts in the form of factor triples ("elements") Factorization of:

[0269] C k = u v，k , σ k , w k，t )(3)

[0270] where the matrix [w k，t represents the aggregated frame-related (i.e., time-related) incremental radial displacement. And the matrix [u v，k represents the spatial (voxel-related) pattern associated with the component.

[0271] The incremental radial displacements are summed to obtain an estimated radial displacement trajectory as a function of time:

[0272]

[0273] where the values are normalized to the maximum observed incremental movement for the target. According to related embodiments where the trajectory is calculated as an integral, the term "summation" herein not only refers to the discrete representation in equation (4), but also to "integration".

[0274] The spatial pattern [u v，k and the radial displacement trajectory are recorded as item 36a.

[0275] Figure 9D is a flowchart detailing the radar signal processing steps (step 30) in an embodiment of the present invention (separate from the Figure 9C described embodiment) from Figure 9A . In step 31b, a 3D image (item 32b) is generated in a manner similar to that described above. In step 33b, the 3D image is decomposed using an algorithm similar to the algorithm described above, thereby generating a set of elements, each described by a 3D image and a time pattern including complex phasors (item 34b). In step 35b, each time pattern is processed using a phase detection procedure similar to that described above to generate displacement data for each element (item 36b).

[0276] Figure 9E is a flowchart detailing the radar signal processing steps (step 30) in an embodiment of the present invention (separate from the Figure 9C and Figure 9D described embodiments) from Figure 9AFlowchart of the radar signal processing steps (step 30). In step 31c, a complex radar signal is decomposed using an algorithm similar to that described above to produce a set of elements, each described by a complex time-independent signal pattern and a time pattern including complex phasors (item 32c). In step 33c, each time pattern is processed using a phase detection procedure similar to that described above to produce displacement data for each element (item 34c). In step 35c, each time-independent signal pattern is used to produce a 3D image for the corresponding element in a manner similar to that described above (item 36c).

[0277] Figure 9F is a flowchart detailing the target processing steps (step 40) in an embodiment of the present invention. From Figure 9A In step 41, by examining the spatial patterns of each element, the elements are grouped into targets representing detected physical objects, thereby generating a target list (item 42). In step 43, the targets are classified and each target is given a label such as "background" (e.g., parts inside a car), "adult", "toddler", "pet", etc. (item 44). This classification is done by examining the spatial patterns and time displacement data of each element within the target.

[0278] In step 45, the time displacement data of the elements within each human target are used to generate a spectral power distribution model describing the periodicity of the target's movement. In an embodiment of the present invention, Welch's method is used to generate the spectral power density model (non-parametric spectral model). In another embodiment, an (Autoregressive Moving Average) ARMA model (parametric spectral model) is used to generate the spectral power density model. Physiological parameters are estimated for human targets, and the physiological parameters include respiratory rate, heart rate, and heart rate variability. The respiratory rate and heart rate are estimated from the peak positions in the spectral power distribution. In an embodiment, using Welch's method, the heart rate variability is estimated from the width of the spectral peak corresponding to the heart rate. In another embodiment, using the ARMA model, the heart rate variability is estimated from the parameter representation of the ARMA model itself.

[0279] In step 47, changes in the respiratory rate, heart rate, and heart rate variability are monitored to indicate changes in health or mental state.

[0280] In step 48, the 3D images associated with each element of the human target are used to identify elements having one or more human body parts. Then this identification is used to generate additional data such as human postures and activity types (sitting, standing, running, etc.), as described above.

[0281] Figure 10AA graph showing radial displacement versus time, as measured for a human subject by a method and apparatus according to an embodiment of the present invention. Portion 201 shows the detected heartbeat, and portion 202 shows the detected respiration. Note that according to this embodiment of the present invention, it is not necessary to isolate the individual regions of the body responsible for the heartbeat and respiration.

[0282] Figure 10B Depicts a spectral power density graph of two elements identified by a method and apparatus according to an embodiment of the present invention.

[0283] In this embodiment, the sensors have been positioned close to the human subject, and the two elements represent two motion patterns, one originating from the breathing motion of the human subject and the other from the heartbeat motion of the human subject. It can be seen that these elements represent motions with different periodicities from each other. Then each element is used to calculate the corresponding rate parameters: the respiration rate (measured in RPM - breaths per minute) and the heart rate (measured in BPM - beats per minute).

[0284] Figures 11A to 11E Depicts an image product at various stages of processing a passenger sitting in a car environment.

[0285] As an introduction, the car interior environment has several factors that make it difficult to identify passengers and distinguish them from each other and from the car interior background during imaging; passenger proximity, differences in passenger reflectivity, and car vibrations.

[0286] Passenger proximity refers to passengers sitting very close to each other, even in contact with each other, which is common in the rear seats. Thus, when considering the reflection data for each frame separately, these rear seat passengers can appear as a single target object.

[0287] Due to differences in body size (e.g., adult vs. toddler), positioning, and orientation, the differences in passenger reflectivity can be quite large. The differences in passenger reflectivity can degrade the detection performance (false positives and false negative rates).

[0288] Car vibrations also pose a significant challenge to current state-of-the-art MIMO imaging techniques. As the passenger background (the car interior itself) vibrates and changes its reflection characteristics, the difficulty of detecting position changes becomes even more severe. As described above, these imaging obstacles are addressed by using the correlated motion as a discrimination parameter.

[0289] Figure 11ADepicts a 2D top-down projection of a 3D image, generated by a MIMO radar mounted on the roof of the passenger compartment of a vehicle. The image represents a single captured frame. White rectangles are added to indicate the boundaries of the vehicle interior. The specific scenario shown is an adult sitting in the driver's seat (upper left corner of the white rectangle), a young child sitting in the front passenger seat (upper right corner of the white rectangle), and another adult sitting in the right rear seat (lower right corner of the white rectangle). It can be seen that it is very difficult to identify the young child because of the lower reflectivity of the young child compared to the adult passengers. Objects associated with the adult passengers mask the signal reflections from the young child.

[0290] Figures 11B to 11D By identifying the relevant movements of each individual passenger, spatial patterns associated with three elements that have been decomposed from a sequence of frames are shown. These spatial patterns allow for easy identification of the three passengers.

[0291] Figure 11E A screenshot of the user interface is shown and is used as the output of the system. On the left is an image generated by filtering the spatial patterns shown in Figure 11B 、 Figure 11C 、 Figure 11D and then recombining the spatial patterns shown in Figure 11B 、 Figure 11C 、 Figure 11D On the right is a graphical summary of the occupancy status reported by the system, correctly identifying the two adults and the young child in their correct positions. By examining the spatial patterns for each detected element, the passengers are classified as adults and young children.

[0292] The separated components characterize the spatial motion patterns associated with each type of movement, such as the spatial motion pattern associated with breathing and the spatial motion pattern associated with a heartbeat.

[0293] The set of voxels that characterizes the movement can be derived from a target tracking function, or the set of voxels that characterizes the movement can be derived from prior knowledge, such as the candidate seating positions of people in a vehicle. The set of voxels can include multiple people, where the set of movement patterns will include, for example, the breathing patterns of these multiple people. In the case of a moving vehicle, the spatial motion pattern can include movement caused by the vibration of the vehicle, and the measured voxels can include reference objects, such as an empty seat. In other examples, the measurements can include moving objects in the environment, such as a ceiling fan, in order to distinguish the fluctuations caused by these objects from the movement caused by the person of interest.

[0294] According to some embodiments, the system can be configured to operate in various detection or enablement modes: high detection mode, intermediate mode, or standby mode, where the fps and the corresponding duration are set by the user or the manufacturer. The following are examples of enablement modes:

[0295] High activation mode: The capture rate is 30 frames per second (fps) for 12 seconds, followed by an 18 - second capture hiatus, and these two steps are repeated 6 times consecutively (3 minutes in total).

[0296] Medium activation mode: The capture rate is 10 fps for 9 seconds, followed by a 51 - second capture hiatus, and these two steps are repeated 10 times consecutively (10 minutes in total).

[0297] Standby mode: No capture for 10 minutes while saving the previously captured and processed data for future analysis and comparison.

[0298] The system provides further flexibility by providing configuration provisions to be enabled in various activation modes, each activation mode lasting for a predetermined time or a predetermined number of cycles, or being enabled in a combination of predetermined time periods and cycles.

[0299] In addition, according to some embodiments, the system can automatically change from one activation mode to another in response to the collected data.

[0300] According to some embodiments, the system can be manually enabled (turned “ON”), and according to some embodiments, the system can be automatically enabled in response to a predetermined instruction (e.g., during a specific time period) and / or a predetermined condition of another system.

[0301] Additional energy - saving measures include enabling measures to reduce the number of radars, transmit / receive modules, and processors according to different power consumptions or activation modes.

[0302] According to some embodiments, the system can be temporarily triggered to shut down or be temporarily enabled in the “standby” mode to save power resources.

[0303] Figure 12 Depicts the operation phases of an operation cycle employed during the activation detection mode according to an embodiment. As shown, the operation cycle includes a measurement phase, a calculation phase, and an idle phase.

[0304] In the measurement phase, the capture of complex target objects is at full - frame rate using a current of 450 mA at a voltage of 12 v for a time slot of 10 msec, e.g., 15 to 25 frames per second (fps).

[0305] During the calculation phase, calculations are performed according to at least some of the above - mentioned methods for identifying motion, using a current of 150 mA at a voltage of 12 v for a time slot of 50 msec;

[0306] During the idle phase, a current of 30 mA at a voltage of 12 V is utilized for a time slot of 140 msec to ensure that the memory retains the previously captured or computed data.

[0307] According to some embodiments, the above methods and systems can be implemented for various monitoring and alerting purposes. In a particular application, for example, a baby / toddler sitting in the rear seat of a vehicle or in a crib is monitored. The device is configured to enable an alert in response to detecting a threshold change in the breathing rate or heart rate.

[0308] Another vehicle monitoring application is in the area of detecting a baby or toddler left in a vehicle after a threshold amount of time after the engine is disengaged and the doors are locked.

[0309] In one embodiment, the monitoring device is implemented inside the vehicle to monitor the occupants.

[0310] In one embodiment, the vehicle monitoring device is configured to be enabled for a predetermined number of cycles and / or time period according to at least one of the above high enable mode and intermediate enable mode when the engine is turned off and / or the doors are locked. The device is linked to the engine and the locking system to provide such a drive function. In the absence of observed movement, the monitoring device adopts a standby mode within a configurable time period.

[0311] According to some embodiments, the alerts are selected from enabling the vehicle horn, enabling the vehicle air ventilation system, opening the vehicle windows, unlocking the vehicle, sending an alert to the application user, sending an alert to the emergency rescue service, and any combination thereof. According to some embodiments, the alert is repeated until the system is manually switched off.

[0312] The monitoring device is configured to sequentially repeat the "monitoring" and "standby" operation modes until: the vehicle is switched to "on", where the system is either manually turned off, automatically turned off, or continues to monitor according to the elapse of a threshold time period or the achievement of a threshold number of repetitions.

[0313] Similarly, the device can be employed to monitor an elderly or sick person lying in bed and enable an alert in response to a threshold change in the breathing rate, heart rate, or heart rate variability.

[0314] Alerts linked to the device include audible alerts, visual alerts, or a combination of both, and in some embodiments, the alerts are remotely enabled via any of a variety of wireless technologies.

[0315] It should be understood that embodiments formed by the combination of feature sets set forth in separate embodiments are also within the scope of the present invention. In addition, although certain features of the present invention have been illustrated and described herein, modifications, substitutions, and equivalents are included within the scope of the present invention.

[0316] Aspects of the present disclosure relate to systems and methods for classifying vehicle occupants in various seats of a vehicle.

[0317] The systems and methods required for such tasks are required to operate around the clock, including in the dark, and be able to detect and classify the occupant even when hidden (such as being covered by a blanket for example). Reliable classification should be provided regardless of the vehicle's state, whether the ignition switch is on, the air conditioner is working, the vehicle is stationary or moving, or even moving on a rough road.

[0318] The classification of the occupant can include various categories, such as but not limited to age group, weight, body type, indication of the presence of a child seat, the position of the occupant, animal versus human, child versus adult, male versus female, and objects such as water bottles and hanging shirts (which are prone to movement while driving).

[0319] Although it is important to classify the occupant, it is also often necessary to respect their privacy and avoid detecting and recording identifying details at the same time.

[0320] Possible techniques for classifying the occupant can include pressure sensors under the seat, cameras assisted by any type of depth camera, stand-alone depth cameras, ultrasonic imaging, and radar imaging.

[0321] The main drawbacks of cameras are the need for an external light source and the inability to penetrate non-transparent materials. Additionally, the image resolution of cameras may violate privacy.

[0322] For example, depth cameras that emit their own light at infrared frequencies are able to work in the dark but may saturate during the day. Additionally, they generally do not penetrate seats and blankets.

[0323] Pressure sensors provide information about weight but do not provide any information about the shape or body type of the occupant, and thus they are usually insufficient.

[0324] On the other hand, radar and ultrasonic imaging systems can be implemented using waves with a wavelength of approximately 1 cm. Such systems can operate in the dark and can penetrate objects that are opaque to visible light. A wavelength of 1 cm is sufficient to group occupants, such as age groups, weight, body type, indication of the presence of a child seat, position of the occupant, etc. as described above, to comply with technical and legal requirements as well as safety rules, but not sufficient to identify them. Radar and ultrasonic sensors can be used to identify passengers under a blanket and are not saturated by natural light sources and sounds.

[0325] Reference Figure 13 , describes a method for converting a 3D complex image obtained by a coordinate grid inside a vehicle into a list of occupants of the vehicle with associated classes.

[0326] The method includes obtaining a 3D complex image of the occupants of the vehicle cabin - step 110.

[0327] Image accumulation is required to obtain a dynamic model of the occupants. The three-dimensional image (complex values of a predefined set of coordinates) is stored as a row vector in a matrix. The matrix can store a predefined number of images (frames), or the number of frames to be stored can be variable.

[0328] An array of transmitting and receiving elements is used to generate a set of complex values associated with coordinates in a predefined region or volume inside and possibly around the vehicle. These values and the associated coordinates are called a complex 3D image. The magnitude of the complex value can indicate the probability that a reflecting object is located at that coordinate.

[0329] U.S. Patent Publication 2019 / 0254544, titled "DETECTING AND MEASURING CORRELATED MOVEMENT BY ULTRA-WIDEBAND MIMO RADAR", incorporated herein by reference, provides an exemplary method for obtaining a 3D complex image of a moving occupant. Another method is described in "3-D Radar Imaging Using Range Migration Techniques" by J.M. Lopez-Sanchez, J. Fortuny-Guasch, published in IEEE Transactions on Antennas and Propagation, Vol. 48, No. 5, May 2000, pp. 728 - 737, which is incorporated herein by reference.

[0330] As a particular case, the image can store only real values, such as representing the magnitude in each voxel.

[0331] The known algorithm for constructing such a complex image for an array of transmit and receive elements is the Delay and Sum algorithm (DAS). Variants of the DAS algorithm can be found in “The Delay Multiply and Sum Algorithm in Ultrasound B-Mode Medical Imaging” by Giulia Matrone, Allesandro Stuart, Giosue Caliano, Giuvanni Magenes, published in IEEE Transactions on Medical Imaging, April 2015, Vol. 34, No. 4, which is incorporated herein by reference. More complex algorithms include algorithms for solving inverse problems. A review of solving inverse problems in imaging can be found in “Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods” by Alice Lucas, Michael Iliadis, Rafael Molina, Aggelos K. Katsaggelos, published in IEEE Signal Processing Magazine, January 2018, Vol. 35, No. 1, pp. 20-36, and in “Convolutional Neural Networks for Inverse Problems in Imaging: A Review” by Michael T. McCann, Kyong Hwan Jin, Michael Unse, published in IEEE Signal Processing Magazine, November 2017, Vol. 34, No. 6, pp. 85-95, both of which are also incorporated by reference.

[0332] When classifying an occupant, it can be assumed that they will exhibit at least slight movement over time, such as chest movement and breathing, so the phase change over time for a given coordinate can indicate the movement of the object. This can be detected when viewing multiple frames. The images can be accumulated in buffer 114 (step 112).

[0333] The walls, seats, and other constant and stationary features of the cabin can be subtracted from the detected signal through a background removal algorithm - step 116.

[0334] Background removal can be achieved by subtracting the average value for each coordinate, for example, in one or both of the following ways:

[0335] Apply a high-pass filter to each coordinate

[0336] For each column in the image matrix, subtract the average value of that column.

[0337] Filtering - Perform step 118 to remove the effects of sidelobes, multipath, thermal noise, and clutter. Figure 13 The filtering step is performed based on dynamic behavior and is extended in Figure 14A Then these points are clustered into groups, and each group in the group is associated with an occupant (step 120).

[0338] Provide data corresponding to the vehicle / geometry model, dimensions, and seat positions - step 121. Associate each cluster with a seat - step 122, and generate occupancy likelihood statistics - step 124, such that a threshold is used to decide whether a seat is occupied - step 126. This decision can be supplemented by the results of an occupant dynamic model - step 128.

[0339] Calculate the characteristics of each cluster based on the vehicle geometry and the point distribution for each cluster (possibly over several frames) - step 130.

[0340] Apply model 132 to the feature classification of step 130 to create classification 134, which evaluates the likelihood that an occupied seat is assigned to a specific category, and this can be smoothed - step 138 to assign the occupant of the seat to a specific category.

[0341] The occupancy determination and classification procedure can involve various methods, especially machine learning. For example, a neural network can be trained to determine occupancy or perform classification into the required category groups, but other classification methods can be used. Learning algorithms can be used to optimize the parameters of the function. In the case of implementing the function in the form of a neural network, a feedforward neural network can be used where appropriate. Additionally or alternatively, a network with feedback can be used to consider historical features, such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory) network, etc.

[0342] The occupancy determination and classification procedure can involve various methods, especially machine learning. For example, a neural network can be trained to determine occupancy or perform classification into the required category groups, but other classification methods can be used. Learning algorithms can be used to optimize the parameters of the function. In the case of implementing the function in the form of a neural network, a feedforward neural network can be used where appropriate. Additionally or alternatively, a network with feedback can be used to consider historical features, such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory) network, etc.

[0343] Alternatively, or in addition, the coordinate values of each box around the seat can be used as the input to the network, rather than a list of specific features. Accordingly, a Convolutional Neural Network (CNN) may be suitable. It should be understood that any combination of the above, such as a combined CNN and RNN may be preferred. The coordinate values within each box can be used to determine whether the seat associated with a particular box is occupied.

[0344] Although neural networks have been described above for illustrative purposes, other classification algorithms can be used additionally or alternatively to provide the required classification, such as SVM (Support Vector Machine), decision trees, ensembles of decision trees, also known as decision forests. Other classification algorithms will occur to those skilled in the art.

[0345] Singular Value Decomposition (SVD)

[0346] Multipaths, grating lobes, and sidelobes have similar dynamic behavior. Thus, singular value decomposition (SVD) tends to represent them using a single vector with time-varying coefficients.

[0347] Reference Figure 14A , shows the time evolution of the signal representing an image. The image set can be decomposed into two components, one plotted with a solid line and the other with a dashed line. The magnitude of one component (solid line) decreases with time, while the magnitude of the other component increases with time. SVD decomposition can provide these components.

[0348] The mathematical formula is as follows: The matrix H stores the image set. Each row represents an image. For example, a standard algorithm called singular value decomposition can be used to decompose the matrix H. This matrix is decomposed into three matrices.

[0349] H = U·D·V H

[0350] In this decomposition, U represents a rotation matrix, D is a diagonal matrix (not necessarily square), and V H is a matrix with the same dimensions as H and with orthogonal rows. The rows of V H contain components such as those shown in the figure above.

[0351] Determining the number of components

[0352] Criteria based on the distribution of singular values (the values on the main diagonal of matrix D) can be used to determine the number of required components. One way is to select the components corresponding to the largest singular values, where these singular values add up to a percentage of the total value, for example, 95%.

[0353] Another method is to search for corner points in a plot of the sorted singular values.

[0354] Both methods are known in the art.

[0355] Alternative decomposition

[0356] Although SVD can be used for decomposition, this should be regarded as only an example of a more general class of decomposition options.

[0357] Alternative decompositions include, for example, the following:

[0358] Independent Component Analysis (ICA)

[0359] This decomposition assumes that the observations are a linear mixture of statistically independent sources. The goal of the decomposition is to find the independent sources.

[0360] The formula is as follows:

[0361]

[0362] where M is the unknown mixing matrix. ICA provides sources that can be regarded as components associated with the occupants.

[0363] The inventors have noted that at high SNR levels, the performance of ICA generally exceeds that of SVD in separating different occupants into different components or sources.

[0364] Yet another example of an alternative decomposition method is sequential spatio-temporal filtering decomposition, as shown in the block diagram of Figure 14B as follows.

[0365] The radar system collects image data as a series of frames 1402. The image frames can be stored in a memory unit of the processor as a buffer 1404 including multiple frames. As an example, the buffer can be a matrix including a set of 15 frames representing data captured for 3 seconds. The strongest peak 1406 in each frame can be determined, for example, the voxel having the strongest variance or root mean square value. Thus, the temporal component 1408 of the strongest peak can be determined for the strongest peak in the buffer, and each voxel of each frame can be projected onto the temporal component 1410. Then the temporal component can be removed from the buffer 1412. This method can be repeated as more image data is obtained.

[0366] Component filtering

[0367] Filtering each component is based on the assumption that components in the form of describing temporal domain movement should have local energy. The filtering operation should preferably preserve local values. Two filtering methods are described below.

[0368] Method 1: Divide the image into high-energy patches and only retain the patch with the highest energy.

[0369] Method 2: Only retain the coordinates with energy higher than a threshold. The threshold can be relative to the peak or absolute value.

[0370] These two methods are described in Figure 14C which indicates that the filtering operation zeros many coordinates. Typically, a mask can be defined that reduces unwanted values rather than zeroing them. Figure 14C Combining the filtered components into a filtered image

[0371] The filtered components can be combined into an image on a component-by-component basis.

[0372] In this article, the following notation is used to describe the method.

[0373] where all elements except (n, n) are set to 0.

[0374] The filtered component of

[0375] The nth row of

[0376] The

[0377] of

[0378] As described below, one or more of several different methods can be used to generate the final image.

[0379] The first method involves averaging the absolute values of several rows of :

[0380]

[0381] In the above expression, C represents a constant, such as the number of rows in the summation operation. An alternative method would be to average the powers of the absolute values, such as averaging the square of the absolute values. Another alternative method is to assign different weights to different rows of Typically, the rows corresponding to the most recently captured image are selected.

[0382] An alternative method is to use the components directly without multiplying by U:

[0383]

[0384] Multiplying by U provides temporal information related to the contribution of the components.

[0385] It is useful to associate each non-zero coordinate with a component. The association can be, for example, a "hard" association, which means it is associated with a single component, or a "soft" association, which means a probability is assigned to the association of the coordinate with the component.

[0386] The hard association can be expressed by the following formula: coordinate i is associated with the component n that maximizes the image at that coordinate.

[0387]

[0388] The soft association can be expressed by the following formula:

[0389]

[0390] As performed in step 120, soft association often helps with clustering. However, the association step is not mandatory. This association step can be skipped.

[0391] It should be understood that the positions of the seats and the walls of the cabin are known. These fixed elements are the background and can be removed. After removing the background, the signal represents the occupant.

[0392] The purpose of clustering is to split non-zero coordinates among the occupants of the vehicle. This can be achieved simply by clustering the coordinates using a standard clustering algorithm. An alternative approach is to utilize the prior knowledge that different coordinates are associated with different components, where different coordinates are associated with different SVD (Singular Value Decomposition) components through hard association. For clarity, in the image, each component has a different color.

[0393] It is expected that different occupants in the vehicle will be divided into different SVD components because they move at different times. However, a single SVD component will be associated with different occupants. In this case, some of the coordinates within the component will form one cluster, while other coordinates will form another cluster. Applying a clustering algorithm can be used to split the component into clusters.

[0394] In Figure 15A , two sets of coordinates associated with the same SVD component are circled. The clustering application for each SVD component generates Figure 15B , in Figure 15B , the clusters circled in Figure 15B shows the clustering of each component using the DBSCAN algorithm, which can also be able to remove outliers. However, other clustering algorithms in the art can be used.

[0395] Returning to reference Figure 14C , two filtering methods are described. Method 1 involves dividing the image into high-energy spots and only retaining the spots with the highest energy. In contrast, Method 2 only retains the coordinates with energy higher than a threshold, which can be relative to the peak or relative to the absolute value.

[0396] Only method 2 requires the use of the DBSCAN algorithm for clustering because method 1 leaves only one cluster for each component.

[0397] Now, the clusters can be applied to the clusters themselves. Each cluster is represented as a Gaussian distribution with a mean and covariance that match the distribution of the points within the cluster. A distance metric can then be defined between these Gaussian distributions.

[0398] The distance between distributions can take many forms. For example, the Kullback Leibler divergence, the Bhattacharyya distance, the Hellinger distance, and the L2 norm of the difference.

[0399] For example, spectral clustering, which applies algorithms known in the art, can be used to generate Figure 16 the results shown in

[0400] For example, a Gaussian distribution can be fitted to a set of coordinates with a given intensity using the following equation. The symbols are as follows.

[0401] The coordinates of the points in the cluster can be represented by the column vector r =(r x , r y , r z ) T However, these coordinates are not necessarily Cartesian coordinates, nor are they necessarily three-dimensional coordinates. Each point in the cluster is associated with an amplitude, denoted by m. The amplitude of the cluster points is a real and positive number, i.e., m i > 0 for each point i.

[0402] For example, the relative weights of each point in the cluster are defined as follows:

[0403]

[0404] The center of the cluster is reduced to:

[0405]

[0406] The matrix is defined as:

[0407]

[0408] where p represents the number of points in the cluster, and

[0409]

[0410] The covariance of the cluster is defined as

[0411]

[0412] Finally, the covariance matrix c and the center μ are used to define a Gaussian distribution:

[0413]

[0414] Other distributions such as the t-distribution, uniform distribution, Gaussian mixture, etc. can be used to describe the clustering of points.

[0415] Figure 17 shows the clustering of points represented as a Gaussian in three-dimensional space.

[0416] It should be understood that each occupant of the vehicle can be associated with a seat. Seat association is the process of associating the seats in the vehicle with each occupant to determine whether a seat is occupied.

[0417] p k (r) = Pr{ r ∈ seat k}

[0418] In Figure 18 , the rectangles define the regions associated with specific seats within the vehicle interior. Typically, these regions can overlap, and the distribution does not need to be uniform. However, as can be seen, the various clusters do align well with the boxes, and thus it is possible to determine whether each seat is occupied.

[0419]

[0420]

[0421] In the second distance metric, it can be used as described in the previous section on clustering.

[0422] d k (cluster q ) = the distance from the distribution of cluster q to p k ( r )

[0423]

[0424] When the probability that cluster q is associated with seat k is determined for each pair {k, q}, each cluster (q) can be associated with the seat (k) for which p k (cluster q ) is the largest.

[0425] We make a distinction between "hard" occupancy and "soft" occupancy. We will now describe how to calculate occupancy.

[0426] For "hard" occupancy, the following rules are used:

[0427] A seat with no cluster associated with it is considered an empty seat.

[0428] A seat with at least one cluster associated with it is considered occupied.

[0429] For "soft" occupancy, the following expression can be used to determine, based on the probability matrix p between each cluster q and each seat k′ k′ (cluster q ) the probability that seat k is occupied:

[0430]

[0431] As an example of a model for valid transitions within a vehicle, consider a vehicle with two rows. The front row seat 1 is the driver, and next to him is seat 2. In the second row are seats 3, 4, and 5, where seat 3 is behind the driver. We can also define positions between two seats, such as seats 3.5 and 4.5. These arrangements are shown in Figure 19 and Figure 20 .

[0432] Figure 21 Shows Figure 13 the occupancy transition diagram for the second row. Each circle represents an occupancy state, with the occupied seat and the state number indicated on the circle.

[0433] The occupancy probability Pr{Seat k is occupied} can be combined with the transition model in the following way.

[0434] Transition probabilities are assigned to each transition in the transitions in the diagram. For example, for state 1, there is a probability of transitioning to state 2, a probability of transitioning to state 3, and a probability of remaining in state 1. These probabilities can be arbitrarily defined, or they can be based on statistical data. In the diagram, wherever there is no connection between states, the transition probability is assumed to be 0. For example, the probability of transitioning from state 1 to state 5 is 0.

[0435] The transition probability from state s 1 to state s 2 is represented by p t (s 1 , s 2 ).

[0436] Each state S has a seat associated with that state. The occupied seat associated with state S is represented by o s . For example, state 4 has seats 3 and 5 occupied: o4 = {3, 5}

[0437] The probability that the update system is in state S can be done as follows:

[0438]

[0439] In other words, the probability of occupancy in state S is updated to the probability that the seat is in state S multiplied by the sum of the transition probabilities from state S' to state S multiplied by the probability of occupancy in these states.

[0440] In some embodiments, instead of calculating the exact probability, a number proportional to the probability is calculated.

[0441] In some embodiments, the sum can be replaced by a single term representing the most likely transition to state S, as follows:

[0442]

[0443] In some embodiments, the logarithm of the probability is used, and the above multiplication can be replaced by the following sum:

[0444]

[0445] In some embodiments, the most likely state can be selected and a few steps can be traced back according to the most likely consecutive states that led to it. Indicating the previous state that led to the most likely current state stabilizes the system by reducing the sensitivity to errors in the seat occupancy probability.

[0446] The implementation of the last equation lends itself to linear programming methods.

[0447] Reference Figure 22 , for occupant classification, when needed, an LP (low-pass) filter can optionally be applied to the filtered 3D image for a configurable time constant (typically a few seconds). The LP filter can be a moving average or can be an IIR (infinite impulse response) exponent of a moving average filter. In other configurations, the low-pass filter may not be applied.

[0448] Using the LP filter tends to preserve useful volume information (preserving voxels that account for reflections of the body). Different body parts may move at different times, and the LP filter can facilitate the accumulation of information over time, resulting in a single 3D image.

[0449] The image is then divided into 3D regions, where each region contains a single object.

[0450] In the case of a vehicle, given its dimensions and geometry, it is possible to derive a 3D region using some measurements of the vehicle cockpit.

[0451] For example, for a 5-seater car, the following measurements are constant for each car model with respect to the sensor origin and can be collected:

[0452] FRF - Front row, furthest forward position (measured with reference to the "centroid" of the seat - see image)

[0453] FRR - Front row, furthest rearward position (measured with reference to the "centroid" of the seat - see image)

[0454] RRC - Rear bench "centroid"

[0455] BNCW - Rear bench width

[0456] STH - Seat height from the ground to the edge

[0457] SSH - Sensor height from the ground to the sensor

[0458] Another option is to use the decision on the upper layer to obtain the number and position of the occupants (providing [x, y] coordinates for each occupant and opening a box around the position [x - dx, x + dx, y - dy, y + dy, 0, SSH]).

[0459] Yet another option is to use SVD decomposition to "color" each voxel by component number, giving an initial guess for the voxel clusters. Then each component can be subjected to DBSCAN - Density-Based Spatial Clustering of Applications with Noise (geometric) clustering to remove outliers and separate geometrically distinct clusters into separate components. From this step on, each component (cluster) cannot be separated any further, and additional clustering is done to separate into groups of clusters.

[0460] For each 3D region, volume- and intensity-based features can be extracted.

[0461] Manually crafted cluster features can be used to evaluate the identity of the occupants.

[0462] Notation:

[0463] Given a 3D image region, all voxels with intensity higher than a specific level are extracted. This gives a list of the occupied voxels of the 3D region.

[0464] voxel i = [x i , y i , z i , I i , i = 1...N

[0465] PC = {voxel 1 ...voxel N}

[0466] Calculate the following features from this list of points. The coordinates are relative to a defined central point of the 3D region. In the case of a car seat, the central point is defined as the region directly above the seat (z - coordinate) and at the center of the expected adult strength (x - coordinate, y - coordinate).

[0467] The number of occupied voxels

[0468] The number of occupied voxels can indicate the volume of the occupied area.

[0469] The number of occupied voxels = N

[0470] The center of intensity

[0471] The center of intensity is the average position of the voxels weighted by their intensity

[0472]

[0473] Covariance and weighted covariance

[0474] Covariance gives a measure of how the points are distributed in space. In the case of occupant classification, it indicates the person's pose (e.g., a forward - leaning adult / a baby sitting in a toddler seat).

[0475] Figure 23 and Figure 24 illustrates this principle.

[0476] The following algorithm shows a code example for calculating the weighted covariance.

[0477] numDimensions = 3;

[0478] gmm_params.mean = zeros(numDimensions, 1);

[0479] gmm_params.cov = zeros(numDimensions, numDimensions, 1);

[0480] weights = iSeat_pts_lntensity / sum(iSeat_pts_Intensity);

[0481] gmm_params.mean = sum(weights.*iSeat_pts_XYZ)';

[0482] zero_mean_weighted_points = sqrt(weights).*(iSeat_ts_XYZ - gmm_params.mean');

[0483] gmm_params.cov = (zero_mean_weighted_points' * zero_mean_weighted_points);

[0484]

[0485]

[0486] cov = zmwp T ·zmmwp

[0487] Extrema of coordinates in the 3D region

[0488] The extreme (maximum / minimum) coordinates of all 3 Cartesian axes give a measure of the occupied region in the 3D region.

[0489] Reference Figure 25 , an indication of the occupant's body size and shape can be obtained by drawing a rectangle around the cluster.

[0490] X min = min(X), X = x 1 ...x N

[0491] X max = max(X), X = x 1 …x N

[0492] Y min = min(Y), Y = y 1 …y N

[0493] Y max = max(Y), Y = y 1 …y N

[0494] Z min = min(Z), Z = z 1 ...z N

[0495] Z max = max(Z), Z = z 1 ...z N

[0496] Center of intensity for a defined z-value

[0497] For adults, the center of intensity on the z-slice above the seat height is typically near the backrest of the seat. For toddlers sitting in a rear-facing infant seat, the center of gravity in some slices is typically shifted more forward.

[0498]

[0499]

[0500] cov = zmwp T ·zmwp

[0501] Average intensity

[0502]

[0503] Maximum intensity

[0504] I max = max(I), I = I 1 ...I N

[0505] Energy below the seat level

[0506] Energy below the seat level can indicate whether a toddler is in the seat. If the toddler is sitting in the seat, there will be no reflections below the expected seating height.

[0507]

[0508] To stabilize the classification output, the classification can be saved in a buffer for a few seconds, and a majority vote or a stabilizer with hysteresis can be used to determine the final classification decision.

[0509] Technical description

[0510] The technical and scientific terms used herein shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. However, it is expected that during the life cycle of the patents maturing from this application, many related systems and methods will be developed. Accordingly, the scope of terms such as computing unit, network, display, memory, server, etc. is intended to include all such prior new technologies.

[0511] As used herein, the term "about" means at least ±10%.

[0512] The terms "comprises," "comprising," "includes," "including," "has," and conjugates thereof mean "including but not limited to," and indicate that the listed components are included, but generally do not exclude other components. Such terms subsume the terms "consisting of" and "consisting essentially of."

[0513] The phrase "consisting essentially of" means that a composition or method may include additional elements and / or steps, provided that the additional elements and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0514] As used herein, unless the context clearly dictates otherwise, the singular forms "a," "an," and "the" may include plural referents. For example, the term "compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

[0515] The word "exemplary" as used herein means "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred over or better than other embodiments or to exclude the combination of features with other embodiments.

[0516] The word "optionally" as used herein means "provided in some embodiments but not in other embodiments." Any particular embodiment of the present disclosure may include a plurality of "optional" features, unless such features conflict.

[0517] Whenever a numerical range is indicated herein, it is intended to include any recited number (fractional or integral) within the indicated range. The phrases "ranging / between" a first recited number and a second recited number and "ranging / from" a first recited number "to" a second recited number are used interchangeably herein and are intended to include the first and second recited numbers and all the fractional and integral numerals therebetween. Accordingly, it should be understood that the description of a range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the present disclosure. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as the individual numerical values within that range. For example, a description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., as well as the individual numbers within that range, for example 1, 2, 3, 4, 5, and 6, and non-integer intermediate values. This applies regardless of the breadth of the range.

[0518] It should be understood that, for clarity, certain features of the present disclosure described in the context of separate embodiments may also be provided combinatorially in a single embodiment. Conversely, for brevity, the various features of the present disclosure described in the context of a single embodiment may also be provided separately or in any suitable sub-combination or in any other described embodiment suitable for the present disclosure. Certain features described in the context of the various embodiments should not be considered essential features of those embodiments unless the embodiment is inoperative without those elements.

[0519] Although the present disclosure has been described in connection with its specific embodiments, it is apparent that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, all such alternatives, modifications, and variations are intended to be embraced within the spirit and broad scope of the appended claims.

[0520] All publications, patents, and patent applications mentioned in this specification are hereby incorporated by reference in their entirety into the specification to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference herein. Additionally, the citation or identification of any reference in this application should not be construed as an admission that the reference is available as prior art to the present disclosure. With respect to the use of section headings, they should not be construed as necessarily limiting.

[0521] The scope of the disclosed subject matter is defined by the appended claims and includes combinations and sub-combinations of the various features described above, as well as variations and modifications thereof, which will occur to those skilled in the art upon reading the foregoing description.

Claims

1. A vehicle cabin monitoring system, the vehicle cabin monitoring system comprises: a radar unit, the radar unit comprising: at least one transmitter antenna, the at least one transmitter antenna being connected to an oscillator and the at least one transmitter antenna being configured to transmit electromagnetic waves into the vehicle cabin, and at least one receiver antenna, the at least one receiver antenna being configured to receive electromagnetic waves reflected by an object within the vehicle cabin and the at least one receiver antenna being operable to generate raw data; a processor unit, the processor unit being configured to receive the raw data from the radar unit and the processor unit being operable to generate image data based on the received data; a memory unit, the memory unit being configured to and operable to store the image data; and at least one output unit; wherein, the processor unit is configured to and operable to: generate at least one 3D image based on the radio frequency response; process one or more consecutive 3D images obtained from the 3D images by removing the background therefrom; filter the 3D image by removing the contribution of at least one of sidelobes, multipath, thermal noise and clutter; detect the occupancy of seats within the vehicle cabin; classify at least one occupant of the seats within the vehicle cabin; detect the posture of at least one occupant of the seats within the vehicle cabin; and determine the seat belt status of at least one occupant of the seats within the vehicle cabin; and wherein, the output unit is configured to and operable to: cancel the airbag operation if the posture of at least one occupant indicates that the airbag operation is unsafe.

2. The vehicle cabin monitoring system according to claim 1, wherein, the radar unit is located at a central position in the roof of the vehicle cabin.

3. The vehicle cabin monitoring system according to claim 1, wherein, the radar unit is embedded between two layers of glass.

4. The vehicle cabin monitoring system according to claim 1, wherein, the radar unit is attached to the glass surface by thermally conductive epoxy resin.

5. The vehicle cabin monitoring system according to claim 1, wherein, the radar unit is incorporated into a sunroof.

6. The vehicle cabin monitoring system according to claim 1, wherein, the radar unit is incorporated into a headrest.

7. The vehicle cabin monitoring system according to claim 1, wherein, the processor unit is operable to: cluster the filtered 3D image; associate at least one seat with at least one occupant; and classify one or more of the occupants based on the distribution of points for each cluster in the 3D image and according to the vehicle geometry.

8. The vehicle cabin monitoring system according to claim 1, wherein, the processor unit is operable to: extract human key points from the received signals; and identify the skeletal points of at least one occupant of the seats in the vehicle cabin.

9. The vehicle cabin monitoring system according to claim 8, wherein, The human key points are selected from at least one of the following: head, left shoulder, right shoulder, left pelvis, right pelvis, left knee, right knee, left thigh, right thigh, and combinations thereof.

10. The vehicle cabin monitoring system according to claim 1, wherein, the processor unit is operable to: monitor the vital signs of at least one occupant of the vehicle cabin.

11. The vehicle cabin monitoring system according to claim 10, wherein, the vital signs include at least one of the occupant's heat rate and respiratory rate.

12. The vehicle cabin monitoring system according to claim 1, wherein, the output unit is configured to: generate an alarm in case of an anomaly.

13. The vehicle cabin monitoring system according to claim 12, wherein, the anomalies are selected from: occupant not wearing a seatbelt, child sitting in the front seat, occupant in an unsafe posture, driver showing low alertness, and combinations thereof.

14. The vehicle cabin monitoring system according to claim 1, wherein, the output unit is configured and operable to: communicate with emergency rescue services in case of an accident.

15. The vehicle cabin monitoring system according to claim 1, wherein, the output unit is further configured to: generate an alarm if a young child is left in the vehicle cabin.

16. The vehicle cabin monitoring system according to claim 1, wherein, the output unit is configured and operable to: trigger a communication system to contact emergency rescue services and indicate vital signs to emergency rescue personnel.

17. A method for monitoring a vehicle cabin: Provide a radar unit, the radar unit comprising: at least one transmitter antenna connected to an oscillator and configured to transmit electromagnetic waves into the vehicle cabin, and at least one receiver antenna configured to: receive electromagnetic waves reflected by an object in the vehicle cabin, and the at least one receiver antenna is operable to generate raw data; Provide a processor unit configured to: receive raw data from the radar unit, and the processor unit is operable to generate image data based on the received data; Provide a memory unit configured and operable to: store the image data; and Provide at least one output unit; Transmit electromagnetic waves into the vehicle cabin; Receive electromagnetic waves reflected by an object in the vehicle cabin; Generate a set of complex values associated with voxels in the vehicle cabin; Convert the set of complex values into a 3D complex image; Filter the 3D complex image by removing the contribution of at least one of sidelobes, multipath, thermal noise, and clutter; Detect the occupancy of seats in the vehicle cabin; Classify at least one occupant of seats in the vehicle cabin; Detect the posture of at least one occupant of seats in the vehicle cabin; Determine the seat belt status of at least one occupant of a seat in the vehicle cabin, and Cancel the airbag operation if the posture of at least one occupant indicates that the airbag operation is unsafe.

18. The method according to claim 17, wherein, the method further comprises: Clustering the filtered 3D complex images; Associating at least one seat with at least one occupant; and Classifying one or more of the occupants based on the distribution of points for each cluster in the 3D complex image and according to the vehicle geometry.

19. The method according to claim 17, wherein, the method further comprises: Extracting human key points from the received signals; and Identifying the skeletal points of at least one occupant of a seat in the vehicle cabin.

20. The method according to claim 19, wherein, the human key points are selected from at least one of the following: head, left shoulder, right shoulder, left pelvis, right pelvis, left knee, right knee, left thigh, right thigh, and combinations thereof.

21. The method according to claim 17, wherein, the method further comprises: Monitoring the vital signs of at least one occupant of the vehicle cabin.

22. The method according to claim 21, wherein, the vital signs include at least one of the occupant's heart rate and respiratory rate.

23. The method according to claim 17, wherein, the method further comprises: In case of an abnormality, the output unit generates an alarm.

24. The method according to claim 23, wherein, the abnormality is selected from: occupant not wearing a seat belt, child sitting in the front seat, occupant in an unsafe posture, driver showing low alertness, and combinations thereof.

25. The method according to claim 17, wherein, the method further comprises: In case of an accident, the output unit communicates with the emergency rescue service.

26. The method according to claim 17, wherein, the method further comprises: If a young child is left in the vehicle cabin, the output unit generates an alarm.

27. The method according to claim 17, wherein, the method further comprises: The output unit triggers the communication system to contact the emergency rescue service and indicate the vital signs to the emergency rescue personnel.

Images