Time-of-flight circuitry, vehicle safety system, and methods

Abstract

The present disclosure generally pertains to Time-of-flight circuitry for a vehicle safety system, the circuitry being configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system.

Description

[0001] TIME-OF-FLIGHT CIRCUITRY, VEHICLE SAFETY SYSTEM, AND

[0002] METHODS

[0003] TECHNICAL FIELD

[0004] The present disclosure generally pertains to time-of-flight circuitry for a vehicle safety system, a vehicle safety system, a method for time-of-flight circuitry for a vehicle safety system, and a method for a vehicle safety system.

[0005] TECHNICAL BACKGROUND

[0006] In the context of mobility, it is generally known to surveil or monitor a driver or other occupants of a vehicle. For example, if it is determined that a driver is about to fall asleep, an alert may be generated to keep the driver from falling asleep.

[0007] Also, time-of-flight (ToF) technology is known. In ToF, a roundtrip delay of emitted light may be determined, which is indicative of a distance (or depth) from an emission source to a scene (e.g., an object)and back to a light receiver .

[0008] In direct time-of-flight (dToF), the roundtrip delay may be determined by measuring the time from emission to re-reception. In indirect time-of-flight (iToF), the roundtrip delay may be determined by synchronizing a modulation frequency of the emitted light with a light detector.

[0009] Although there exist techniques for vehicle occupant monitoring, it is generally desirable to provide time-of-flight circuitry for a vehicle safety system, a vehicle safety system, a method for time-of-flight circuitry for a vehicle safety system, and a method for a vehicle safety system.

[0010] SUMMARY

[0011] According to a first aspect, the disclosure provides time-of-flight circuitry for a vehicle safety system, the circuitry being configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system.

[0012] According to a second aspect, the disclosure provides a vehicle safety system comprising: time-of-flight circuitry configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system.

[0013] According to a third aspect, the disclosure provides a method for time-of-flight circuitry for a vehicle safety system, the method comprising: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system.

[0014] According to a fourth aspect, the disclosure provides a method for a vehicle safety system, the vehicle safety system comprising time-of-flight circuitry, the method comprising: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system.

[0015] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0016] BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0018] Fig. 1 depicts a occupant monitoring system according to the present disclosure;

[0019] Fig. 2 depicts a method for generating body feature information according to the present disclosure;

[0020] Fig. 3 depicts an illustrational result of a method according to Fig. 2;

[0021] Fig. 4 depicts a skeleton in a depth image and a result of segmentation in a confidence image;

[0022] Fig. 5 depicts a depth image, a confidence image, and an image in which distances between a driver and a seat are depicted;

[0023] Fig. 6 depicts a method for ToF circuitry and / or for a vehicle safety system according to the present disclosure in a block diagram; Fig. 7 depicts a method for a vehicle safety system according to the present disclosure in a block diagram;

[0024] Fig. 8 depicts a flow chart of a method for deciding whether a more precise occupant tracking mode is applied; and

[0025] Fig. 9 depicts a flow chart of a method that is carried out in the more precise occupant tracking mode.

[0026] DETAILED DESCRIPTION OF EMBODIMENTS

[0027] Before a detailed description of the embodiments starting with Fig. 1, general explanations are made.

[0028] As mentioned in the outset, occupant monitoring in vehicles is generally known. It has been recognized that known approaches which include ToF for in-cabin monitoring may achieve inferior results compared to infrared (IR) alone. For example, IR imaging may have at least twice as good an SNR than ToF. Also, a contrast of IR imaging may be better. A resolution of ToF may be lower than of IR imaging.

[0029] However, it has been recognized that depth provided by ToF may be useful for in-cabin monitoring.

[0030] Moreover, it has been recognized that an IR image may be obtained from a ToF measurement too, e.g., by using a single tap component of iToF as an approximation of the image content included in the IR image. To improve the SNR of the IR image, all taps and components of iToF may be added together. It has further been recognized that such information may be used for exposure control, e.g., based on a region of interest (ROI). For example, the single tap component may be used for exposure control since it may be indicative of ambient light.

[0031] It has further been recognized that it may be possible to improve the ToF confidence signal to get closer to IR imaging performance in terms of resolution. For example, spatial resolution may be increased with super resolution with two (or more) taps and depth information. For example, the increased spatial resolution may be used for more precise monitoring of body parts, such as eye ROI (which might not be possible with the original resolution).

[0032] Also, a modulation frequency may be tuned to get more precise depth and increase accuracy. A more accurate depth may be combined with confidence images for super resolution. Noise removal algorithms may be used to further increase the SNR in confidence images. Also, exposure and gain control may be used for comparative contrast in different ROIs. It has been recognized that the following aspects may be useful to implement in order to increase a spatial resolution with two (or more) taps:

[0033] • Using confidence image with depth to detect salient features in the depth image out of which one can perform salient body segmentation, thereby obtaining reliable body segments present in the depth images.

[0034] • Using the confidence and clean depth for detection of a head of the occupant(s) along with other available body segments.

[0035] • Using head IR information to adapt the gain to obtain higher dynamic range / contrast for confidence / IR component.

[0036] • Based on depth ROI, adapting a modulation frequency to get higher precision depth component for recognition of body part position for active / passive safety.

[0037] • The adapted depth and confidence / IR may then be used with segmentation to aid IR part to get higher resolution image of eye regions (super resolution with machine learning approach).

[0038] In such embodiments, as mentioned above, a machine-learning approach may be utilized, wherein the present disclosure is not limited to such embodiments.

[0039] In a non-machine learning approach according to the prior art, one would make a clean mesh, map confidence values, interpolate and then project the values to a 2D coordinate system. In such a case, there would be multi-samples (multi -tap in iToF) in time as well, which may be used for a 3D trajectory estimation which may then be filtered in time domain.

[0040] According to the present disclosure, higher depth accuracy may be achieved and may be used for salient features to detect a distance between an occupant to points of interest in the cabin.

[0041] Therefore, some embodiments pertain to time-of-flight circuitry for a vehicle safety system, the circuitry being configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system, e.g., adaptive restraint system.

[0042] The circuitry may be any entity or multitude of entities configurable to process time-of-flight data according to the present disclosure, such as a processor (CPU (central processing unit), GPU (graphics processing unit)), an FPGA (field-programmable gate array), a computer, a server, a camera, a driver monitoring system (DMS), or the like. Also, combinations of the above-mentioned entities may be envisaged, in some embodiments.

[0043] Time-of-flight data may be obtained based on a time-of-flight (ToF) measurement, such as in indirect measurement according to iToF (indirect time-of-flight), dToF (direct time-of-flight), spot ToF, or the like.

[0044] For example, the ToF circuitry may communicate with a ToF system or may be included in a system, such that the ToF circuitry may be configured to process the ToF data. For example, the ToF system may be positioned roughly around a rear-mirror of an in-cabin of the vehicle and thereby having a large field of view covering the whole cabin. It will be appreciated that some positions in the cabin and therefore the field of view may be occluded by certain fixed fittings, moveable fittings, occupants or articles introduced into the cabin by occupants (e.g., car fixtures, bag of an occupant, or the like), such that also embodiments may be envisaged in which the ToF system may be partly occluded.

[0045] Also, it should be noted that other positions of the ToF systems are envisaged according to the present disclosure depending on what ROI should be monitored, e.g., on a roof lining, a lighting console, behind a steering wheel, or the like.

[0046] The ToF data may be indicative of an occupant of a vehicle. For example, a corresponding ToF sensor / camera may be directed towards a position / location of an occupant (e.g., a driver) of the vehicle such that, when the occupant is in the vehicle, a ToF measurement may be used to acquire parameters (e.g., body features, as discussed herein) of an occupant.

[0047] Based on the ToF data, body feature information regarding the occupant may be generated. A body feature may include one or more body parts (such as for example a head, a forearm, a thigh, a torso, a hand, a forearm connected to a hand) that are recognized, a distance of a specific body part (e.g., from an exterior border of the body, such as a back of a head) to a location (or a point) of (or within) the vehicle, a posture of the occupant, body segmentation information, depth information of a body part, or the like. Body features may include different parameters such as upper body size of the occupant, width, age, gender, body volume, etc.

[0048] The body feature information may be generated based on a predetermined algorithm, which, for example, may be based on a data driven approach, such as an artificial intelligence, a neural network, a machine-learning approach, or the like. The predetermined algorithm may be a hard- coded algorithm, or it may include a learning algorithm which may further be developed / trained, e.g., when an (end) user uses the vehicle. The body feature information may be provided to a safety function of a vehicle safety system, such as an Occupant Status Monitoring (OSM), adaptive restraint system or a driver monitoring system (DMS). The safety function may process the body feature information together with other information provided from different sensors, such as features from an RGB or IR camera, sound information from a microphone, temperature characteristics from a temperature sensor, breathing or pulse information for example from a radar sensor, or the like.

[0049] According to the present disclosure, a position of a body feature in steady state (e.g., a state in which no safety measure may be necessary) may be determined. Also a temporal dynamic behavior of the body feature(s) may be determined which may be indicative of information whether or not the occupant is responsive to a system notification (e.g., a warning), or whether the occupant returns to a nominal seating position from an irregular position.

[0050] However, such temporal behaviors of an occupant may be too abruptto be captured in a rough detection and may require a precise capturing in high resolution as well as a high frame rate. According to the present disclosure, such predictive behavior of occupants may be achieved. However, such monitoring may have a high power consumption, such that it may not be operated continuously, in some embodiments, such that, in such embodiments, selective operation may be carried out based on a selected ROI, as described herein.

[0051] Accordingly, the ToF circuitry may be part of an OSM and / or DMS or it may be a separate unit in the vehicle.

[0052] In some embodiments, the body feature information is generated based on depth information of the time-of-flight data.

[0053] As generally known, ToF data may include depth information and confidence information. If (only) the depth information is used, a depth of different body parts of the occupant (with respect to the ToF sensor / camera) may be determined. Thereby, for example, a posture of the occupant may be determined.

[0054] Accordingly, in some embodiments, the body feature information is indicative of a posture of the occupant.

[0055] In some embodiments, the body feature information is generated based on confidence information of the time-of-flight data, such that segmentation of the occupant’s body may be carried out, as will be discussed further below. Accordingly, in some embodiments, the body feature information includes body segmentation information of the occupant.

[0056] Both the depth and the confidence information may be used. Hence, in some embodiments, the body feature information is generated based on depth information of the ToF data and confidence information of the ToF data.

[0057] Although a distance between a predetermined body feature may be determined based on depth alone or confidence alone, it may be more effective (e.g., more exact) when both types of information are used.

[0058] Accordingly, in some embodiments, the body feature information is indicative of a distance between a body feature of the occupant and a predetermined location with respect to (or “of’ or “within”) the vehicle. Furthermore, according to the present disclosure, a predictive decisionmaking may be enabled causing a system to perform an adaptive safety control operation. Body features according to the present disclosure are expected to move smoothly and not likely to have an abrupt movement, in some embodiment, thereby enabling trajectory estimation. Also, a Kalman filter or the like may be provided to further improve occupant motion estimation.

[0059] Some embodiments pertain to a vehicle safety system including: time-of-flight circuitry configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system, as discussed herein.

[0060] The vehicle safety system may include or be combined with at least a part of an OSM, restraint system, DMS, or the like, as discussed herein.

[0061] In some embodiments, the safety function included in the vehicle safety system is configured to: obtain the body feature information.

[0062] In some embodiments, the safety function is configured to obtain intention estimation based on a temporal behavior analysis.

[0063] For example, by tracking a (detailed) behavior of an occupant, an anticipation / prediction of a body feature may be carried out before the occupant actually behaves according to the prediction. Hence, it may be possible to anticipate a position of a body feature before it reaches a destination area for activating a warning or an alarm, thereby enabling a counter-measure before it might be too late. Moreover, false alarms may be reduced. In some embodiments, the safety function is further configured to: generate an alert based on a distance of a body feature of the occupant to a predetermined location with respect to the vehicle.

[0064] Some embodiments pertain to a method for time-of-flight circuitry for a vehicle safety system, the method including: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system, as discussed herein.

[0065] In some embodiments, body feature information is generated based on depth information of the time-of-flight data, as discussed herein. In some embodiments, the body feature information is indicative of a posture of the occupant, as discussed herein. In some embodiments, the body feature information is generated based on confidence information of the time-of-flight data, as discussed herein. In some embodiments, the body feature information includes body segmentation information of the occupant, as discussed herein. In some embodiments, the body feature information is generated based on depth information of the time-of-flight data and confidence information of the time-of-flight data, as discussed herein. In some embodiments, the body feature information is indicative of a distance between a body feature of the occupant and a predetermined location with respect to the vehicle.

[0066] Some embodiments pertain to a method for a vehicle safety system, the vehicle safety system including time-of-flight circuitry, the method including: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system, as discussed herein.

[0067] In some embodiments, the method further includes: obtaining, by the safety function, the body feature information, as discussed herein. In some embodiments, the safety function is further configured to: generating an alert based on a distance of a body feature of the occupant to a predetermined location with respect to the vehicle, as discussed herein.

[0068] From application performance perspective, the present disclosure may achieve precise tracking of an occupant’s behavior, e.g., by analyzing a temporal transition and further enabling to anticipate a near future position of a body feature. For example, based on an initial (small) motion that may be recognized, a prediction may be carried out, as discussed herein. Thereby, false alarms and incorrect operations due to inappropriate estimation may be reduced. If a system repeatedly sends false preventive alarm, for example to return to an appropriate seating position, the system may be perceived to be annoying to an occupant who may ignore the alarms.

[0069] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0070] Returning to Fig. 1, there is depicted a Occupant Status Monitoring (OSM) 1 as an embodiment of a vehicle safety system.

[0071] The OSM 1 includes a ToF sensor 2, in this embodiment an iToF camera configured to acquire time-of-flight data indicative of a driver of a vehicle, without limiting the present disclosure in that regard since any other occupant may be monitored accordingly. The ToF data includes depth and confidence information as discussed herein.

[0072] The OSM 1 further includes ToF circuitry 3 according to the present disclosure.

[0073] Based on obtained ToF data, the ToF circuitry 3 is further configured to generate body segmentation information and to provide the body segmentation information to a safety function 4 included in the OSM 1.

[0074] In this embodiment, the safety function 4 also obtains an RGB camera output for the surveillance of the driver. It should be noted that the present disclosure is not limited to the case described herein since in some embodiments, an IR camera or RGB-IR camera may be used accordingly.

[0075] Fig. 2 depicts a schematic block diagram of a method 10 for generating body feature information 11.

[0076] ToF data including confidence 12 and depth 13 are fed into a confidence-based depth filtering algorithm 14 configured to generated different outputs, based on a coarse determination whether the obtained result has a sufficient quality:

[0077] One output is statistics for gain control 15 which is used for refining the confidence 12, i.e., a gain of the iToF camera is adapted based on this information. The confidence 12 is again used for refining the statistics for gain control 15. Another output is statistics for modulation frequency 16 which is used to refine the depth measurement. For example, the modulation frequency may be adapted such that the depth measurement becomes more precise, as generally known to the skilled person.

[0078] Also, based on the confidence 12, segmentation labels 17 are generated based on a machinelearning algorithm. Based on the labels 17, the confidence 12, as well as the confidence filtered depth 14, the body feature information is generated. The labels, the confidence, and the depth may allow the system to estimate the 3D positions and orientations of predetermined body parts. Thereby, a corresponding skeleton fitting may be carried out based on which the body features are determined.

[0079] Fig. 3 depicts an illustrational result of a method according to Fig. 2. An interior of a vehicle is shown captured with an iToF camera, wherein the ToF data are only pre-processed at that point. In the interior, a driver 31 and a passenger 32 are shown.

[0080] Based on the pre-processed depth, certain points 33 (white rectangles) on the occupants are identified which are indicative of their respective postures (indicated with white lines in Fig. 3).

[0081] Based on the identified points, a distance to a predetermined location in the vehicle (or with respect to the vehicle) is determined. In this embodiment, a distance between the driver’s 31 head and a steering wheel 34 is determined. In some embodiments predetermined locations may be fixed positions, such as fixed armrests, corners of windows, or positions on the roof lining, where adaptive restraint system might be placed It should be noted that any distance between identified points and predetermined locations may be determined, in some embodiments. Also, multiple distances may be determined in some embodiments.

[0082] It should also be noted that a distance to any point (or location) of the vehicle may be determined and the present disclosure is not limited to points of the in-cabin. For example, a distance of a body feature to a windshield (inside or outside or intermediate point) may be determined. Also, a distance of a body feature between an intermediate layer of a car roof (e.g., a stabilizing metal layer) may be determined, in some embodiments.

[0083] Fig. 4 depicts a skeleton in a filtered depth image (top image) and the result of a segmentation is shown in a confidence image (bottom image). As in Fig. 3, certain points and a skeleton of the driver are identified in the depth image.

[0084] In the confidence image, body parts 41 are identified. Similarly, Fig. 5 depicts a depth image (top left) and a confidence image (bottom left), wherein in the confidence image, points on the occupant’s body are identified based on the depth image with a high precision. Higher confidence values can be indicative of a front of the occupant’s body parts used for segmentation with depth information, while low confidence values indicate areas of depth with lower reliability to be discarded for further processing. For example, fingers of the driver can be recognized in this manner (indicated with white circles).

[0085] On the right of Fig. 5, it is shown that distances of the driver’s head and back are determined based on a method discussed herein and are used as an indicator of the driver’s position or posture. For example, in case it is determined, based on the determined distances, that the driver is bent forward, an alert is generated that he is out of position. Additionally or alternatively, an airbag restraint system is adapted or suppressed based on the driver’s position.

[0086] Fig. 6 depicts a method 50 according to the present disclosure in a block diagram. The method may be carried out by ToF circuitry according to the present disclosure and / or by a vehicle safety system according to the present disclosure.

[0087] At 51, ToF data are obtained, as discussed herein.

[0088] At 52, body feature information is generated and, at 53, provided to a safety function, as discussed herein.

[0089] Fig. 7 depicts a method 60 for a vehicle safety system according to the present disclosure in a block diagram.

[0090] At 61, body feature information is obtained, as discussed herein.

[0091] At 62, an alert is generated based on a distance of a body feature of an occupant to a predetermined location with respect to the vehicle, as discussed herein.

[0092] In some embodiments, a method described under reference of Fig. 7 may further include learning (e.g., by repeated operation) the occupant’s personal behavior characteristics including his body posture and / or temporal dynamical movements indicative of whether the occupant is expected to move in order to determine whether an alarm and / or a further safety relevant countermeasure is required. A simple threshold-based detection operation is also envisaged, in some embodiments, but it may exhibit inferior performance to the adaptive operation explained herein.

[0093] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. Please note that the division of the OSM 1 into units 2 to 4 is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, the OSM 1 could be implemented by a respective programmed processor, field programmable gate array (FPGA), or the like. It may further contain a personalization feature with a learning function and a dictionary, parametrization by individual users with exportable parameter sets, and the like.

[0094] The methods described herein can also be implemented as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the method described to be performed.

[0095] Some embodiments pertain to a method for in-cabin monitoring (ICM) including at least one of the following aspects: obtaining wide field of view iToF images of people (e.g., occupants of a vehicle) and high confidence areas (where depth is reliable); fusing pre-processed (removed low confidence areas) iToF images; detection of occupants and measuring people parameters and their position in the vehicle; categorizing people and postures as a function of the measured parameters measuring the distance between specific body parts and the key in-cabin 3D points of interest; supplementing driver monitoring system related to head orientation and eye gaze

[0096] A wide field, as mentioned above, may be defined as an angle of view such that at least eighty percent of a width of a (front) cockpit area (e.g., bordered by the exterior dimensions of seats or by the windows, or the like) may be captured.

[0097] The iToF images may be obtained from an iToF ICM sensor, wherein each image may correspond to a region of interest (ROI) of the iToF sensor. For example, different ROIs may have different resolutions, different frames per second (FPS), different size, or the like.

[0098] For example, in an ROI in which the head of the driver is included, higher resolution, FPS, or the like may be needed than in an ROI where no body part of the driver is identified. An ROI may be a rectangular region including the body part, but the present disclosure is not limited in that regard since any shape of the ROI may be envisaged. Also, it should be noted that an overlap of two body parts (of the same occupant or of different occupants) may be included in an ROI.

[0099] Accordingly, one or more ROIs may include a part or the whole head of the occupant with gain adaptation to obtain a higher dynamic range and / or contrast for confidence and / or IR component.

[0100] In some embodiments, one or more ROIs may include an eye region of an occupant with a resolution enhancement compared to other ROIs (e.g., super-resolution, such as two to four times higher). Such data may be used to supplement an OSM / DMS for head orientation and gaze monitoring.

[0101] For example, the super-resolution may be achieved with a machine-learning approach using a single reflection image and enhanced depth image.

[0102] In some embodiments, based on the ROI, a modulation frequency is adapted, thereby achieving a higher precision depth component for passive safety.

[0103] In some embodiments, an ROI includes a predetermined body region depth enhancement by use of an adaptive modulation frequency when a rough position and a 3D bounding box is assessed.

[0104] In some embodiments, an adapted depth and confidence / IR is used with segmentation to aid an IR part to get a higher resolution image of eye regions (super resolution with machine learning approach).

[0105] In some embodiments, a confidence image with depth to detect salient features in the depth image based on which salient body segmentation is performed, thereby obtaining reliable body segments present in the depth image.

[0106] In some embodiments, categories of people are used for occupant status monitoring (OSM).

[0107] In some embodiments, categories of people are used for passive safety features related to adaptive restraint systems. The categories may include a body size, such as small, medium, large, a gender, an age, and the like. Additionally or alternatively, weight may be a category. For example, a seating arrangement may adapt to an occupant’s weight, and so, known weight measurements may be re-purposed. Weight sensors may be used to determine where airbags should trigger, for example. Such weight measurements may be performed by using strain gauge layout under the occupant’s seat. Posture / position detection according to the present disclosure may additionally be used to determine whether it is safe to deploy an airbag or to suppress it depending on the detected occupant in irregular seating condition. However, the estimation based on strain gauge detection alone (without the determination according to the present disclosure) may have a limited detection accuracy and may not suffice to perform an intelligent adaptive airbag deployment control.

[0108] In some embodiments, besides parameters about the size, volume, weight, etc., seating positions, hands, leg, postures, and predicted movements based on intention detection as described herein may be envisaged.

[0109] In some embodiments, body segmentation may be carried out and a distance between specific segmentation body parts (head, chest, torso, or the like) to specific 3D points of interest, in vehicle for restraint system adaptation may be determined, e.g., to adapt a seat belt, airbag, or the like.

[0110] Assessing a driver’s or any other occupant’s physical status and its anticipated movements (or movement intention) are may be further relevant when combined with the increased use of autonomous driving, e.g., when occupants depart from a traditional seating position while in drive and thus, in case of an accident, an inappropriate deployment of an airbag may cause severe injuries.

[0111] In some embodiments, posture may include at least one angle at which an occupant is seated, for example slouching or leaning towards the center console and armrest, or conversely towards the door, or twisting themselves around to look at what is happening in the passenger area or reaching for an object that has slipped under a seat.

[0112] Passive safety devices may be configured to act differently dependent on detected postures, e.g., based on a control signal. For example, airbags may inflate differently depending on the detected posture or behavior of the occupant. For example, airbags may contain different pockets or volumes which can be differently controlled, such as individually. For example, airbags may be inflated to different pressures. For example, airbags may be inflated from different inflation sources causing an airbag to be harder or softer on one side than another or fully inflated on one side before another is fully inflated.

[0113] Some safety features may, in the past, have been designed based on worst case scenario impacts with an emphasis on saving life. The disclosure envisages that safety features can be deployed in a manner such as to mitigate harm to the occupant. For example, it may not be appropriate to deploy an airbag in the same manner, if they are facing to the left, rather than facing forward. Therefore, an airbag may deploy in a first mode when the occupant’s head is detected to be facing in the direction of travel, and the airbag may deploy a second, different, mode when an occupant’s head is detected to be facing a direction which is greater than a first threshold angle. The airbag may deploy in a third mode, differently again, when an occupant’s head is detected to be facing a direction which is greater than a second threshold angle. It will be appreciated that head is merely an example of a body part and that the disclosure may apply to other body parts or combinations of multiple body parts. It will be appreciated that an airbag may be a system of multiple airbags or airbag pockets. The disclosure provides advantageous adaptability of safety features based on occupant posture.

[0114] It will be appreciated that the above description, for clarity, has described embodiments with reference to different functional units, circuitry and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuitry and / or processors may be used without detracting from the embodiments.

[0115] Described embodiments may be implemented in any suitable form including hardware, software, firmware or any combination of these. Described embodiments may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuitry and / or processors.

[0116] Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in any manner suitable to implement the technique.

[0117] Furthermore, the invention is described taking the conventional seating equipment as reference but it obviously could be further expanded to largely flexible equipment to be introduced in automated vehicles when driver and passenger are no longer restrained to a specific position within the vehicle, requiring adaptation of the detection to best fit these new needs. Some examples are reclined seat, turned seat accommodating the occupants, and the like.

[0118] Fig. 8 depicts an embodiment of a method 70 according to the present disclosure for performing a steady state rough operation mode before defining a specific ROI and performing to a fine operation mode. The operation of the device in high frame rate and high-resolution mode means the device needs to be operated at high operation frequency and therefore becomes less power efficient and dissipates a part of the energy as heat which may further deteriorate sensor performance due to noise induced by the temperature.

[0119] To save energy and suppress thermal noise, the device is operated at a moderated operation mode at a steady state operation mode (low frame rate, low resolution) when precise occupant evaluation is not required. In this steady state operation mode, it is still possible to perform occupant rough estimation based on an analysis of captured data.

[0120] At 71, the sensor is operated in nominal operation mode to capture the occupant’s status in rough mode and to check for irregularities, without running any further detailed analyses.

[0121] At 72, the captured data is analyzed to check whether the occupant might be out of an expected position and / or posture to self-trigger a need for detailed precision operation to estimate the precise driver status.

[0122] At 73, the system (OSM) further checks an external request flag whether or not to shift the operation to high precision (“FINE”) operation mode. The switching to this FINE operation mode can be determined by detecting “irregular occupant posture”, or the like, but as well taking into consideration the vehicle status while in drive (e.g., speed) and its risk prevailing with possibility for the need to deploy an airbag, or the like. For example, if the vehicle status is determined to be in an empty country road at low speed with negligible risk, it may not be necessary to assume that the airbag is required. Similarly, such an assumption may be made while maneuvering the vehicle in a garage or a parking spot (such that the airbag may be deactivated, in such cases). On the other hand, if a situation is identified in which an accident is imminent and the driver not seated in a secure seating position, an anticipation / prediction / estimation of a driver status may be required. In which condition set the device is required to switch to FINE precision operation could be parametrized to best fit the intended safety operation to the type of vehicle in interest.

[0123] At 74, the system makes the decision whether to shift the sensor operation to the FINE mode to enable better decision taking for the operation of passive safety system without harming the occupant with inappropriate deployment of the airbag(without limiting the present disclosure in that regard). If a FINE operation is required, it will exit the loop and go to a precise operation flow, such as described under reference of Fig. 9. If the shift of the operation mode to the precise (FINE) mode is not necessary, the method will maintain the operation loop to resample the driver status in the rough operation mode until the monitoring operation in no longer needed due to termination of the trip not illustrated in this flow.

[0124] Analyses and detection based on personalized learned characteristics are not given in this flow chart for simplicity, but it can well be adapted for rough occupant estimator. Operation flow described herein is an exemplary embodiment and it may be rearranged, e.g., by intermittent or occasional transition to fine operation to confirm driver ambiguously detected states, or the like.

[0125] There is a limited space available within a vehicle and thus, it may be necessary to minimize the number of equipment taking space of the passenger compartment. Therefore, in some embodiments, a driver status detector for passive safety is integrated or combined with equipment for the purpose of OSM / DMS.

[0126] Fig 9 provides an operation sequence flow chart when the FINE precision operation according to Fig. 8 is activated. According to the embodiment of Fig. 9, the same equipment is used for occupant behaviour and intention estimation as the equipment used in Fig. 8 (thereby not occupying more space, as discussed above).

[0127] In this embodiment, an anticipation / prediction / estimation of the driver’s behavior is carried out based on an initial detection of movement. This is made possible by a personalized action classification based on historical data of the occupant that is used to learn a behavior dictionary with DNN machine learning (without limiting the present disclosure in that regard).

[0128] At 81, an operation control unit (not illustrated) determines an ROI for FINE precision tracking when indicated as described above. Also, other relevant parameters requiring precision detection are determined to enable improved high resolution and precise motion tracking required to achieve higher accuracy of estimation.

[0129] Once the operation of the image sensor is adapted to track a particular ROI with higher accuracy, at 82, it becomes possible to obtain posture details and precise motion enabling to conduct anticipation / prediction / estimation of the occupant’s future behavior by analyzing the driver’s time-sequential behavior with high accuracy. In other words, at 82, a detailed tracking is carried out to estimate an expected future motion (e.g., of at least one of body, upper posture, hand, foot, and the like).

[0130] Based on the anticipation / prediction / estimation, it can be determined that the occupant intends to return to a regular seating position, aborting any NDRA (non-driving related activities) and returning his hands and feet back to their normal operating position, or the like, However, just tracking the motion does not necessarily provide the correct information necessary for a later system decision, e.g., the decision to emit an alarm, or the like.

[0131] At 83, the detected detailed tracking information is compared to the historical occupant behavior data to enable a behaviour analysis based on personal behavior characteristics. For example, some drivers may be slow and other drivers may be quicker in reacting, thereby exhibiting different behaviors when compared to each other.

[0132] This evaluation is built upon cumulative personal behavior data analyses and based on a learning algorithm, stored and maintained as a baseline of a reference dictionary, which, in some embodiments may further include a health condition, a body feature, or the like, thereby adding biasing on a decision , e.g., based on a health factor (e.g., fatigue, drowsiness), and thus modifying the behavior characteristics.

[0133] At 84, the analysis system uses the detected motion estimation results to score the driver / occupant willingness to react by analyzing the detected motion and comparing it to the personal behavior tendency dictionary data.

[0134] The estimator uses the cumulative historical results learned based on the occupant’s historical record, as later described at 89. Thus, it is possible to predict the driver’ s / occupant’s expected behavior from the initial and detailed tracked cue, even when the driver / occupant has not reached an expected target position yet.

[0135] This anticipation / prediction is important since there may be a limited time budget in activating a pre-deployment system of an airbag, for example. In case of providing cues to the DMS / OSM system, the anticipation can improve the estimation of the driver’ s / occupant’s intention to intervene. If a decision is only made based on a final posture and / or position, there might be a chance that it is decided that an airbag is inflated too late or that it is not inflated at all due to an incorrect decision (or that it is not inflated although necessary).

[0136] In another example, if it fails to detect the willingness of the driver to take control of the vehicle in a critical situation, thereby causing the vehicle to initiate an evasive maneuver even in situation the driver is properly interacting with the system, a high risk of an accident may be at hand. Reducing false alarm or inappropriate intervention is crucial in achieving higher trust of the occupant in the system. Otherwise, the occupant may disable such a function.

[0137] At 85, the detected occupant / driver status is evaluated and the remaining time budget for taking a decision is estimated. For taking the decision, it may need to be balanced between remaining time versus running a further additional round of detailed occupant / driver tracking detection, as at 87. If not enough time is left for another round, it may be decided to directly start the preparation for the activation of an adaptive smart airbag according to the detected status or based on anticipated expectation of the targeted occupant position. In the case the detection results are fed to the OSM / DMS system, the detected results will serve to achieve improved accuracy on its estimation, especially in deriving a confidence level of the occupant’s intention to intervene.

[0138] At 87, another round of notification escalation is carried out, as discussed above, e.g., to find out whether the occupant reaches a safe seating position and / or to determine an optimal deployment of the safety system and / or to determine a counter-measure to be taken, if the driver does not intervene. Also, a notification is issued to the occupant / driver to return to a safe seating position. Also, in some embodiments, warnings may be escalated, if necessary (e.g., louder or more frequent) and the occupant’s behavior may be recorded and handed over to be stored as personal historical record for the learned occupant behavior dictionary, thereby updating the dictionary.

[0139] At 86, a decision is taken, as described above. If there is not enough time left, at 88, a passive safety system is started since a risk is too imminent

[0140] It should be noted that if it is determined that no risk is imminent, the FINE mode is terminated and the ROUGH operation mode described in Fig. 8 is carried out.

[0141] The accumulated temporal behavior tracking data is analyzed and fed, at 89 into the learned occupant behavior dictionary.

[0142] The termination of the loop at 86, i.e., to move to 88 instead of repeating and proceeding to 87 is made when at least one of following conditions are met with further needs to continue the behavior monitoring of the occupant according to the prompt initiated at 71.

[0143] 1. Irreversible activation of the detonation of airbag determined

[0144] 2. Situation requiring pre-activation to prepare the airbag release re-establishing lock that avoids incorrect airbag detonation

[0145] 3. Occupant / driver back to the expected “rest” position detected and confirmed that the situation does no longer require further behavior tracking and release from alert state (in vehicle design, “rest” position (or another “irregular seating position) may be expected to not require an adaptive deployment of the airbag)

[0146] 4. In operation functioning as a component of OSM / DMS to track driver behavior or the anticipated behavior detection, it may temporarily leave the FINE precision operation mode if MRM (minimum risk maneuver) operation is performed under safe conditions. However, termination of the loop does not apply if the MRM determines that there is an imminent risk during the MRM maneuver, occupant / driver behavior tracking continues to monitor and estimate the occupant / driver and provide necessary posture and related detail to prepare for the adaptive deployment of the airbag or otherwise reaches a safe stop without having to detonate the airbag, e.g., by independent triggering mechanism built in the airbag system itself.

[0147] Each of incident tracking of the detailed occupant behavior which had been temporally recorded stored as cumulated data is fed to behavior learning and used to refine the accuracy of the detector in 89 (occupant behavior dictionary) at exiting the loop.

[0148] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure. The present disclosure is explained based of time-of-flight technologies, but it can also be achieved with a combination of 2D image sensor technologies, radio wave, millimeter radar sensor, IR imaging, NIR, imaging, a combination of NIR and RGB or any other combination, or the like, further including complementary use to achieve high precision operation and the overall energy efficiency.

[0149] Note that the present technology can also be configured as described below.

[0150] (1) Time-of-flight circuitry for a vehicle safety system, the circuitry being configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system.

[0151] (2) The time-of-flight circuitry of (1), wherein body feature information is generated based on depth information of the time-of-flight data.

[0152] (3) The time-of-flight circuitry of (2), wherein the body feature information is indicative of a posture of the occupant.

[0153] (4) The time-of-flight circuitry of anyone of (1) to (3), wherein the body feature information is generated based on confidence information of the time-of-flight data. (5) The time-of-flight circuitry of (4), wherein the body feature information includes body segmentation information of the occupant.

[0154] (6) The time-of-flight circuitry of anyone of (1) to (5), wherein the body feature information is generated based on depth information of the time-of-flight data and confidence information of the time-of-flight data.

[0155] (7) The time-of-flight circuitry of (6), wherein the body feature information is indicative of a distance between a body feature of the occupant and a predetermined location with respect to the vehicle.

[0156] (8) Vehicle safety system comprising: time-of-flight circuitry configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system.

[0157] (9) The vehicle safety system of (8), further comprising: the safety function configured to: obtain the body feature information.

[0158] (10) The vehicle safety system of (9), wherein the safety function is further configured to: generate an alert based on a distance of a body feature of the occupant to a predetermined location with respect to the vehicle.

[0159] (11) A method for time-of-flight circuitry for a vehicle safety system, the method comprising: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system.

[0160] (12) The method of (11), wherein body feature information is generated based on depth information of the time-of-flight data.

[0161] (13) The method of (12), wherein the body feature information is indicative of a posture of the occupant. (14) The method of anyone of (11) to (13), wherein the body feature information is generated based on confidence information of the time-of-flight data.

[0162] (15) The method of (14), wherein the body feature information includes body segmentation information of the occupant.

[0163] (16) The method of anyone of (11) to (15), wherein the body feature information is generated based on depth information of the time-of-flight data and confidence information of the time-of- flight data.

[0164] (17) The method of (16), wherein the body feature information is indicative of a distance between a body feature of the occupant and a predetermined location with respect to the vehicle.

[0165] (18) A method for a vehicle safety system, the vehicle safety system comprising time-of-flight circuitry, the method comprising: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system.

[0166] (19) The method of (18), further comprising: obtaining, by the safety function, the body feature information.

[0167] (20) The method of (19), wherein the safety function is further configured to: generating an alert based on a distance of a body feature of the occupant to a predetermined location with respect to the vehicle.

[0168] (21) A computer program comprising program code causing a computer to perform the method according to anyone of (11) to (20), when being carried out on a computer.

[0169] (22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (11) to (20) to be performed.

Claims

CLAIMS1. Time-of-flight circuitry for a vehicle safety system, the circuitry being configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system.

2. The time-of-flight circuitry of claim 1, wherein body feature information is generated based on depth information of the time-of-flight data.

3. The time-of-flight circuitry of claim 2, wherein the body feature information is indicative of a posture of the occupant.

4. The time-of-flight circuitry of claim 1, wherein the body feature information is generated based on confidence information of the time-of-flight data.

5. The time-of-flight circuitry of claim 4, wherein the body feature information includes body segmentation information of the occupant.

6. The time-of-flight circuitry of claim 1, wherein the body feature information is generated based on depth information of the time-of-flight data and confidence information of the time-of- flight data.

7. The time-of-flight circuitry of claim 6, wherein the body feature information is indicative of a distance between a body feature of the occupant and a predetermined location with respect to the vehicle.

8. Vehicle safety system comprising: time-of-flight circuitry configured to: obtain time-of-flight data indicative of an occupant of a vehicle; generate, based on the time-of-flight data, body feature information regarding the occupant; and provide the body feature information to a safety function of the vehicle safety system.

9. The vehicle safety system of claim 8, further comprising: the safety function configured to: obtain the body feature information.

10. The vehicle safety system of claim 9, wherein the safety function is further configured to: generate an alert based on a distance of a body feature of the occupant to a predetermined location with respect to the vehicle.

11. A method for time-of-flight circuitry for a vehicle safety system, the method comprising: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system.

12. The method of claim 11, wherein body feature information is generated based on depth information of the time-of-flight data.

13. The method of claim 12, wherein the body feature information is indicative of a posture of the occupant.

14. The method of claim 11, wherein the body feature information is generated based on confidence information of the time-of-flight data.

15. The method of claim 14, wherein the body feature information includes body segmentation information of the occupant.

16. The method of claim 11, wherein the body feature information is generated based on depth information of the time-of-flight data and confidence information of the time-of-flight data.

17. The method of claim 16, wherein the body feature information is indicative of a distance between a body feature of the occupant and a predetermined location with respect to the vehicle.

18. A method for a vehicle safety system, the vehicle safety system comprising time-of-flight circuitry, the method comprising: obtaining time-of-flight data indicative of an occupant of a vehicle; generating, based on the time-of-flight data, body feature information regarding the occupant; and providing the body feature information to a safety function of the vehicle safety system.

19. The method of claim 18, further comprising: obtaining, by the safety function, the body feature information.

20. The method of claim 19, wherein the safety function is further configured to:generating an alert based on a distance of a body feature of the occupant to a predetermined location with respect to the vehicle.

Images