Method and system for determining a status indicative of whether a safety belt of a vehicle is used

Abstract

The present disclosure provides methods and systems for determining a state indicative of whether a safety belt of a vehicle is used. A computer-implemented method for determining a state indicative of whether a safety belt of a vehicle is used, the method comprising the steps of: acquiring at least one image of a portion of an interior of the vehicle; determining whether the at least one image includes a buckle receiver; if it is determined that the at least one image includes the buckle receiver, determining the state based on the image, otherwise performing: extracting information related to a user of the safety belt and / or information related to a buckle of the safety belt from the acquired image; determining a probability of a state change based on the extracted information; and updating the state based on the determined probability.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a method and a system for determining a state indicative of whether a safety belt of a vehicle is used. BACKGROUND

[0002] The safety belt reminder function, which is partly required by law, can consist of two elements: a sensing unit that detects the presence of a person at a given seat, and a second sensing unit that detects whether the safety belt is used.

[0003] To improve the cost of the overall system, many Original Equipment Manufacturers (OEM) are interested in alternative solutions to replace the multiple sensing units. One sensing unit that can cover both elements can be a vision sensor, such as a camera.

[0004] Currently, cameras are introduced in many vehicle cabins. Camera sensors are used, for example, for driver state sensing, such as drowsiness / distraction, but can also be used for other tasks if the camera is positioned in such a way that it can see the relevant part of the cabin.

[0005] However, it is difficult for the camera to observe the occupant in all situations.

[0006] Therefore, there is a need to provide an improved method and system for observing the occupant. SUMMARY

[0007] The present disclosure provides a computer-implemented method, a computer system and a non-transitory computer readable medium. Exemplary embodiments are given in the dependent claims, the description and the drawings.

[0008] In one aspect, the present disclosure is directed to a computer-implemented method for determining a state indicative of whether a safety belt of a vehicle is used, the method comprising the steps of: acquiring at least one image of a portion of the interior of the vehicle; determining whether the at least one image comprises a buckle receiver; if it is determined that the at least one image comprises the buckle receiver, determining the state based on the image, otherwise performing: - extracting information related to a user of the safety belt and / or information related to a buckle of the safety belt from the acquired image; - determining a probability of a state change based on the extracted information; and - updating the state based on the determined probability. The probability can be determined based on an analysis of the at least one image, for example based on the position of the hand of the user or the trajectory of the hand of the user.

[0009] The initial state can be predefined as not wearing a safety belt.

[0010] In other words, if the state cannot be estimated directly based on the image (i.e. without knowing the previous state), the probability of at least a change of state can be estimated based on the image, e.g. based on information extracted from the image. Thus, given the current state, the probability of an updated state can be determined.

[0011] According to various embodiments, the information can be extracted using image processing methods, e.g. using machine learning methods, e.g. artificial neural networks.

[0012] According to various embodiments, the extracted information can comprise at least one keypoint of at least one body part of the user. For example, the keypoint can be a coordinate (e.g. a three-dimensional coordinate in a world coordinate system) of a joint of the user (e.g. a shoulder joint of the user or an elbow joint of the user or a hand of the user).

[0013] According to various embodiments, the extracted information can comprise information about the position of the buckle. For example, the information about the position of the buckle can comprise a static position or a time series of positions (in other words: a trajectory of the buckle).

[0014] According to various embodiments, a plurality of images, e.g. a time series of images, can be captured and can be subject to subsequent processing. With a time series of images, more information can be conveyed, so that the probability can be estimated more accurately.

[0015] The state can be or can indicate a probability of whether the seat belt is used.

[0016] The state can be a "seat belt used" or a "seat belt not used" or a probability distribution of both states. The probability distribution can provide a probability for each of "seat belt used" and "seat belt not used", e.g. a value between 0 and 1 or between 0% and 100%, wherein the probabilities can add up to 1 or 100%.

[0017] According to embodiments, the portion of the interior of the vehicle comprises at least one of a portion near the buckle of the seat belt or a portion near the buckle receiver of the seat belt. However, even if the portion of the interior comprises the buckle or the buckle receiver, some portions of the image can be obstructed, so that the image does not necessarily comprise or show the buckle or the buckle receiver in reality.

[0018] According to embodiments, the computer-implemented method can further comprise the step of: determining a trajectory of a hand of the user relative to the buckle if it is determined that the at least one image does not comprise the buckle receiver; wherein the probability is determined based on the trajectory. Even in case the camera cannot see the actual buckling or unbuckling, e.g. due to an obstruction of a portion of the buckle receiver, the trajectory of the hand can be used to determine the probability and then update the state.

[0019] According to embodiments, the computer-implemented method can further comprise the step of setting the probability of changing the state from "unbuckled" to "buckled" higher than the probability of changing the state from "buckled" to "unbuckled" if it is determined that the hand is on a trajectory towards the belt buckle receptacle without determining that the at least one image comprises the belt buckle receptacle. While trajectories of the hand towards the belt buckle receptacle can be observed for buckling and unbuckling, typically longer trajectories can be observed when buckling, as the hand has to grasp the belt buckle and move it all the way to the belt buckle receptacle. In contrast to this, when unbuckling, shorter trajectories of the hand can be observed, e.g. from a stationary position near the leg to the belt buckle receptacle.

[0020] According to embodiments, the computer-implemented method can further comprise the step of setting the probability of changing the state from "unbuckled" to "buckled" lower than the probability of changing the state from "buckled" to "unbuckled" if it is determined that the hand is on a trajectory away from the belt buckle receptacle without determining that the at least one image comprises the belt buckle receptacle.

[0021] According to embodiments, the computer-implemented method can further comprise the step of classifying a posture of the hand of the user of the safety belt near the belt buckle if it is determined that the at least one image does not comprise the belt buckle receptacle; wherein the probability is determined based on the posture. The posture of the hand can be classified using any suitable classification method, e.g. a binary classification method that can distinguish between two possible postures, or a multi-class classification method that can distinguish between multiple possible postures. Possible classes can for example include a "grasping" posture, an "open hand" posture, or a "hand holding belt buckle" posture.

[0022] According to embodiments, the computer-implemented method can further comprise the step of setting the probability of changing the state from "unbuckled" to "buckled" higher than the probability of changing the state from "unbuckled" to "buckled" if it is determined that the posture of the hand is a grasping posture without determining that the at least one image comprises the belt buckle receptacle.

[0023] According to embodiments, the computer-implemented method can further comprise the step of determining whether the belt buckle is located in the hand of the user if it is determined that the at least one image does not comprise the belt buckle receptacle; wherein the probability is determined based on whether the belt buckle is located in the hand of the user. In addition to the information whether the belt buckle is located in the hand of the user, the trajectory of the hand of the user can be used when updating the state.

[0024] According to embodiments, the computer-implemented method can further comprise the step of: determining a gaze direction of a user of the seatbelt relative to the buckle if it is determined that the at least one image does not comprise the buckle receiver; wherein the probability is determined based on the gaze direction.

[0025] According to embodiments, the status comprises a probability that the seatbelt of the vehicle is used.

[0026] In another aspect, the disclosure relates to a computer system comprising a plurality of computer hardware components configured to perform a plurality or all steps of the computer-implemented method described herein. The computer system can be part of a vehicle.

[0027] The computer system can comprise a plurality of computer hardware components (e.g. a processor, e.g. a processing unit or a processing network), at least one memory (e.g. a memory unit or a memory network), and at least one non-transitory data storage. It will be appreciated that further computer hardware components can be provided and used to perform steps of the computer-implemented method in the computer system. The non-transitory data storage and / or the memory unit can comprise a computer program for instructing the computer, e.g. using the processing unit and the at least one memory unit, to perform a plurality or all steps or aspects of the computer-implemented method described herein.

[0028] In another aspect, the disclosure relates to a vehicle (e.g. a car, a bus, a truck, or a lorry) comprising a computer system as described herein and a sensor configured to acquire images.

[0029] In another aspect, the disclosure is directed to a non-transitory computer-readable medium comprising instructions for performing a plurality or all steps or aspects of the computer-implemented method described herein. The computer-readable medium can be configured as an optical medium, e.g. a compact disc (CD) or a digital versatile disc (DVD), a magnetic medium, e.g. a hard disk drive (HDD), a solid state drive (SSD), a read-only memory (ROM), e.g. a flash memory, or the like. Further, the computer-readable medium can be configured as a data storage accessible via a data connection, such as an internet connection. The computer-readable medium can be, for example, an online data repository or a cloud storage.

[0030] The disclosure is also directed to a computer program for instructing a computer to perform a plurality or all steps or aspects of the computer-implemented method described herein.

[0031] With the method and apparatus according to various embodiments, a visual detection of seatbelt usage of mostly obscured persons in the back seat of a car can be provided.

[0032] With the method and apparatus according to various embodiments, the classification of the seat belt usage by the vehicle occupant can be monitored using the interior sensing system by taking into account predetermined criteria in the interior scene.

[0033] Each time a direct or indirect cue (in other words: criterion) related to the seat belt status is observed, the probability distribution of "on" and "off" can be updated, as described above. In this sense, the relevant observations can potentially be exploited to provide reclassification at individual frames. BRIEF DESCRIPTION OF DRAWINGS

[0034] Exemplary embodiments and functionalities of the present disclosure are described herein in connection with the following illustrative figures:

[0035] Figure 1 is a diagram of a system according to various embodiments;

[0036] Figure 2 is a flowchart illustrating a method for determining a status indicating whether a seat belt of a vehicle is used according to various embodiments; and

[0037] Figure 3 is a computer system having a plurality of computer hardware components configured to perform steps of a computer-implemented method for determining a status indicating whether a seat belt of a vehicle is used according to various embodiments. DETAILED DESCRIPTION

[0038] The safety belt reminder function, which is partially mandated by law, can consist of two elements: a sensing unit that detects the presence of a person at a given seat, and a second sensing unit that detects whether the safety belt is used.

[0039] To improve the cost of the overall system, many original equipment manufacturers (OEMs) are interested in alternative solutions to replace the plurality of sensing units. One sensing unit that can cover both elements can be a vision sensor, such as a camera.

[0040] One or more cameras can be provided around the rearview mirror, in the center console, above the dashboard, or in the overhead console. These locations can be summarized as front-row "center high" locations (as opposed to camera locations above the second or third row of seats, or A- or B-pillars).

[0041] According to various embodiments, the camera can be disposed in such a front-row center high location or any other suitable location.

[0042] From the perspective of the central high camera, a passenger sitting in the left or right seat of the second row can at most be partially visible. To detect whether a passenger is wearing a seat belt, a very specific solution to this problem can be provided which relies heavily on partial and indirect observations.

[0043] According to various embodiments, for the left and right seat of the second row, a probability state is maintained with respect to whether the seat belt is buckled or not, respectively. Since the seats can be considered to be most independent, details of the various embodiments can be described for only one seat.

[0044] First, it can be detected when a person starts to occupy a seat. This can be achieved by a face and / or body tracking based or seat area classifier. The detection can be performed when and shortly after the door corresponding to the seat has been opened and can also cover the case that a person switches the seat from another seat of the second row to the considered seat.

[0045] When the person arrives at the seat, it can initially be determined that the seat belt is off. From this point, cues can be sought that indicate that the seat belt is buckled. One example can be the movement of a hand that can hold the seat belt plug onto the belt buckle. When such a cue is observed, the probability state can be modified to reflect the probability that the seat belt is now on given the observation.

[0046] For a cue that directly observes the state of the seat belt, a likelihood value p on in the range [0, 1] can be defined that the seat belt is worn off and the opposite value p on = 1 - p .

[0047] For a cue that only observes a potential state change (e.g. a buckle / unbuckle event), a similar likelihood function can be defined that relates to the likelihood of a state change to buckle or unbuckle.

[0048] Now, any observed cue can be associated with a conditional probability that indicates, given the observation, what the probability is to stay at p off or to switch to p off if we were in the unbuckled state p on before or to stay at p on or to switch to p on if we were in the buckled state p off before. Thus, with Bayes’ theorem, the probability state with respect to the state of the seat belt can be updated.

[0049] In the following, relevant cues for observations will be described.

[0050] For example, a direct observation of the seatbelt portion can be provided from the acquired image: if the camera has a line of sight to the buckle receiver area and is able to detect the portion of the seatbelt in this area in front of the person’s body, this can significantly increase the probability that the seatbelt is worn. It does not indicate whether the seatbelt is correctly worn, so some misuse cases cannot be distinguished if only the lower part of the seatbelt is visible, for example the shoulder strap passing under the arm or the waist strap behind the body.

[0051] According to various embodiments, if a hand trajectory towards / away from the seatbelt is determined based on the acquired image (or acquired images):

[0052] - a hand detection module (e.g. a body keypoint detection module or other implementation of hand detection) can locate the position of the hand in the image;

[0053] - the output of the hand detection module can be 2D or 3D coordinates (e.g. hand center point), hand region (e.g. bounding box) or pixel-wise segmentation of the hand region;

[0054] The hand can be assigned to a person and seat in an assignment step;

[0055] A hand pose classification module can distinguish different poses of the hand, for example a grasping hand, an open hand, or a pose where the hand holds the seatbelt buckle;

[0056] - a trajectory classification module can distinguish a movement from the shoulder area towards the buckle receiver, a movement from the buckle receiver towards the shoulder area, and other trajectories based on the sequence of images;

[0057] - if the trajectory is “towards the buckle receiver”, the likelihood of the seatbelt being on can be increased;

[0058] - if the trajectory is “away from the buckle receiver”, the likelihood of the seatbelt being off can be increased;

[0059] - if the classified hand pose is “grasping hand”, this can lead to a higher likelihood than “open hand”;

[0060] - if the seatbelt plug is detected in the hand during a movement towards the buckle (hand pose “hand holding seatbelt”), this can lead to a higher likelihood of switching from off to on. Conversely, if it is detected during a movement away from the buckle, the likelihood of switching from on to off can be very high;

[0061] - in another variant, instead of using discrete hand poses from a classification module, the likelihood can be a function of the numerical output of a hand pose classification neural network;

[0062] - if the seatbelt plug is visible in the hand, this can lead to a higher probability of switching from on to off;

[0063] - if the seatbelt plug is in the hand during a movement away from the buckle, this can lead to a higher likelihood of switching from on to off;

[0064] - if the seatbelt plug is visible in the hand, this can lead to a higher probability of switching from on to off;

[0065] - the relevant shoulder region can be selected by the attention region generation model based on body keypoints and vehicle-specific configuration data.

[0066] According to various embodiments, if it is determined, based on the acquired image (or images), that the hand is moved away from the buckle:

[0067] - first, the hand position in the image can be estimated, for example by locating key body points or by an object detection approach that is able to detect the hand region;

[0068] - the static buckle region can be configured in a calibration step, for example in the form of an attention region in the image. Alternatively, the buckle can be located by commonly used object detection methods;

[0069] - the movement away from the buckle can be represented by an increasing relative distance of the detected hand region and the static buckle region.

[0070] According to various embodiments, if it is determined, based on the acquired image (or images), that the hand is present at the buckle receiver region:

[0071] - the hand trajectory classification can provide a hand activity classification (e.g. stationary, moving, buckling action...);

[0072] - if a movement of the hand is observed, this can mean that the buckle button is pressed (activity class buckling action) and this can indicate a switch from on to off;

[0073] - the likelihood of changing the buckle state can be proportional to the confidence of the activity class of the class “buckling action”;

[0074] - if the hand is always stationary, the likelihood of changing the buckle state can be low;

[0075] - generally, each time the hand is at the buckle, the probability of switching the state from on to off or from off to on is small.

[0076] According to various embodiments, looking at the buckle by the (occupant) during any of these activities can slightly increase the probability of a change in the above cases:

[0077] - The head detection module can locate the image region containing the head of the person;

[0078] - The detected image region can be further analyzed to estimate the head orientation (e.g. 3 rotation angles);

[0079] - The estimated head rotation parameters can be used to identify the case where the head is looking at the seatbelt buckle receptacle;

[0080] - Looking at the seatbelt can increase the likelihood of a seatbelt state change.

[0081] According to various embodiments, the combination of the person (in other words; the occupant) looking at the seatbelt and his or her hand being in that region can further increase the overall likelihood of a seatbelt state change.

[0082] Observing the seatbelt plug in the buckle or the seatbelt stretched in a reasonable position and direction can increase the confidence that the seatbelt is being worn.

[0083] According to various embodiments, if it is determined based on the acquired image (or images) that the hand is moved to the shoulder / above the shoulder:

[0084] - If the shoulder region is visible and can be detected via the body keypoint detection module, the shoulder region can be dynamically set in the image relative to the detected shoulder point;

[0085] - If the body keypoint detection module is not available, a configurable static shoulder region can be defined.

[0086] Figure 1 A diagram 100 of a system according to various embodiments is shown. Car specific configuration parameters 102 can be provided. A camera 104 can acquire an image or a series of images. Vehicle state data 106 can include, for example, door state and / or vehicle speed. A body keypoint detection module 110, a person presence module 108, an attention region generator 112, a hand tracking module 114, a seatbelt classifier 116, a seatbelt detector 118, a hand position assignment module 120, a hand trajectory classification module 122, a hand pose classification module 124 can be provided. A hand position likelihood 126, a hand trajectory likelihood 128, a hand pose likelihood 130, a seatbelt region classifier likelihood 132, and a seatbelt detector likelihood 134 can be determined and provided to a fusion module 136.

[0087] More specifically, a shoulder point can be provided to the attention region generator 112, and a hand point can be provided to the hand tracking module 114.

[0088] The camera 104 can provide images (or multiple images) to the person presence module 108, the body keypoint detection module 110, the seatbelt classifier 116, and the seatbelt detector.

[0089] The belt buckle and shoulder regions can be provided from the region of interest generator 112 to the hand position assignment module 120. The hand position assignment module 120 can provide the hand region index and / or the duration of the hand within a given region to the hand position likelihood (function) 126.

[0090] The region of interest generator 112 can provide information about the body regions to the seatbelt classifier 116.

[0091] The hand tracking module 114 can provide information about the tracked hand position and / or bounding box and / or hand velocity and / or hand acceleration to the hand trajectory classification module 122 and / or the hand pose classification module 124.

[0092] The hand trajectory classification module 122 can provide the hand trajectory class (and optionally the corresponding confidence), e.g., from shoulder to buckle or from buckle to shoulder, and / or the hand activity index (and optionally the corresponding confidence), e.g., stationary, moving, buckling activity, to the hand trajectory likelihood (function) 128.

[0093] The hand pose classification module 124 can provide information about the hand pose (and optionally the corresponding confidence) to the hand pose likelihood (function) 130.

[0094] The seatbelt classifier 116 can provide the per-region buckle state (and optionally the corresponding confidence) to the seatbelt region classifier likelihood (function) 132.

[0095] The seatbelt detector 118 can provide the per-pixel classification and / or segmentation of the seatbelt pixels to the seatbelt detector likelihood (function) 134.

[0096] It should be appreciated that the system can include some or all of the components shown in FIG. 1, but can not be limited to these components. Figure 1

[0097] The input data can include data from at least one camera and vehicle sensor data (e.g., door status or speed), as well as some configuration parameters that can be specific to a given car model.

[0098] Figure 1 The likelihoods 126, 128, 130, 132, and 134 shown are the respective likelihood functions.

[0099] According to various embodiments, the hand position likelihood function can have the following properties: ​

[0100] - If a hand is detected in the shoulder region or the buckle region for a period of time, the likelihood of a state change is high;

[0101] - If a hand is detected outside the region for a period of time, the likelihood is low;

[0102] - The hand position likelihood function can be a function of the duration inside the box (e.g. the likelihood is highest for a certain short period of time in which the seatbelt is operating normally, and will decrease for shorter or longer periods of time);

[0103] - The hand position likelihood function can be designed manually based on model knowledge or learned from training data;

[0104] - The hand positioning likelihood can only contribute to the transition events and not provide a direct measure of the seatbelt state.

[0105] According to various embodiments, the hand trajectory likelihood function can have the following properties:

[0106] - If the observed trajectory resembles (observed or modeled) unbuckling or buckling motion, the corresponding likelihood can be high;

[0107] - The similarity can be measured in terms of spatial trajectory as well as velocity, or can be the output of a trained regression model.

[0108] According to various embodiments, the hand pose likelihood can have the following properties:

[0109] - If the pose of the hand during a potential unbuckling or buckling motion indicates that the buckle can be held, this can increase the likelihood of observing a buckling or unbuckling activity;

[0110] - The hand pose can be a discrete hand state determined by a classifier. Individual hand states can be associated with a likelihood of the hand holding the buckle;

[0111] - Optionally, a regression model can be trained to estimate the likelihood of the hand currently holding the buckle.

[0112] According to various embodiments, the seatbelt classifier likelihood can have the following properties:

[0113] - The classifier can classify image regions according to whether they contain a seatbelt or a buckle;

[0114] - This can be particularly relevant for the area around the buckle receiver. If a seatbelt or buckle is detected in this area, this can increase the probability of the seatbelt being "on" over time;

[0115] - Optionally, the belt buckle receiver can be classified as visible or occluded. If it is visible and no seatbelt or belt buckle is detected, this can increase the likelihood of a "seatbelt off" over time.

[0116] According to various embodiments, the seatbelt detector likelihoods can have the following properties:

[0117] - The detector can detect and localize the seatbelt and the seatbelt buckle;

[0118] - If a belt buckle is detected in the buckle receiver, this can increase the likelihood of a "seatbelt on" over time;

[0119] - If a seatbelt is detected in the vicinity of the buckle receiver, this can increase the likelihood of a "seatbelt on" over time.

[0120] - If the belt buckle and / or the seatbelt are detected in or near the hand during a potential buckling or unbuckling motion, this can increase the confidence of the classification output of this motion.

[0121] According to various embodiments, to simplify the detection / observation of the above events and cues, markers (e.g., infrared (IR) markers) can be placed on the seatbelt plug and / or on the seatbelt itself.

[0122] According to different embodiments, a combination with other sensors like a buckle sensor and a seatbelt extension measurement can be provided.

[0123] According to various embodiments, the system can be modeled by a hidden Markov model, where the described likelihood functions correspond to the observation likelihoods for a given state, and the hidden states are at least seatbelt on (buckled) or seatbelt off (unbuckled). Additional states can be added for "buckling motion", "unbuckling motion", and "unknown".

[0124] Figure 2 A flow 200 is shown, which illustrates a method for determining a state indicating whether a seatbelt of a vehicle is used, according to various embodiments. At 202, at least one image of a portion of an interior of the vehicle can be acquired. At 204, it can be determined whether the at least one image includes a belt buckle receiver. If it is determined at 204 that the at least one image includes (or shows or contains) a belt buckle receiver, at 206, a state can be determined based on the image. Otherwise (i.e., if it is determined at 204 that the at least one image does not include a belt buckle receiver), at 208, information related to a user of a seatbelt and / or information related to a buckle of the seatbelt can be extracted from the acquired image. Furthermore, at 210, a probability of a state change can be determined based on the extracted information, and at 212, the state can be updated based on the determined probability.

[0125] According to various embodiments, the extracted information can include or can be at least one key point of at least one body part of the user, and / or the extracted information can include or can be information about a position of the buckle.

[0126] According to various embodiments, the portion of the interior of the vehicle can include or can be at least one of a portion near a buckle of the seat belt or a portion near a buckle receiver of the seat belt.

[0127] According to various embodiments, if it is determined that the at least one image does not include the buckle receiver, a trajectory of a hand of the user with respect to the buckle can be determined, and the probability can be determined based on the trajectory.

[0128] According to various embodiments, if, in a case where it is determined that the at least one image does not include the buckle receiver, it is determined that the hand is heading toward the buckle receiver, the probability that the state is changed from "unbuckled" to "buckled" can be set to be higher than the probability that the state is changed from "buckled" to "unbuckled".

[0129] According to various embodiments, if, in a case where it is determined that the at least one image does not include the buckle receiver, it is determined that the hand is heading away from the buckle receiver, the probability that the state is changed from "unbuckled" to "buckled" can be set to be lower than the probability that the state is changed from "buckled" to "unbuckled".

[0130] According to various embodiments, if it is determined that the at least one image does not include the buckle receiver, a posture of the hand of the user of the seat belt near the buckle can be classified, and the probability can be determined based on the posture.

[0131] According to various embodiments, if, in a case where it is determined that the at least one image does not include the buckle receiver, the posture of the hand is classified as a gripping posture, the probability that the state is changed from "unbuckled" to "buckled" can be set to be higher than the probability that the state is changed from "unbuckled" to "buckled" when an open hand posture is determined as the posture of the hand.

[0132] According to various embodiments, if it is determined that the at least one image does not include the buckle receiver, it can be determined whether the buckle is located in the hand of the user, and the probability can be determined based on whether the buckle is located in the hand of the user.

[0133] According to various implementations, the gaze direction of a seatbelt user relative to the buckle can be determined, and a probability can be determined based on the gaze direction. For example, a camera system can be used to determine the gaze direction. The position of one or more of the user's eyes and / or the user's head can be determined to ascertain the gaze direction. Since the camera's coordinates can be fixed and are known to be related to the vehicle's chassis, and therefore also relative to the buckle, the seatbelt user's gaze direction relative to the buckle can be determined.

[0134] According to various implementations, the state may include or may be the probability that the vehicle's seat belts are used.

[0135] According to various implementation methods, it is determined that the state can be estimated based on the image, and the state can be determined based on the image.

[0136] Each of steps 202, 204, and 206, as well as the further steps described above, can be performed by computer hardware components.

[0137] Figure 3 A computer system 300 with multiple computer hardware components is shown, the multiple computer hardware components being configured to perform steps of a computer-implemented method according to various embodiments for determining a state indicating whether a vehicle's seatbelt is in use. The computer system 300 may include a processor 302, a memory 304, and a non-transitory data storage unit 306. An image sensor 308 (e.g., a camera, time-of-flight camera, infrared camera, lidar sensor, or radar sensor) may be provided as part of the computer system 300 (e.g., ...). Figure 3 (as shown), or it can be provided outside the computer system 300.

[0138] Processor 302 can execute instructions provided in memory 304. Non-transitory data storage unit 306 can store computer programs, including instructions that can be transferred to memory 304 and then executed by processor 302. Image sensor 308 can be used to acquire images of a portion of the interior of the vehicle.

[0139] The processor 302, memory 304, and non-transitory data storage unit 306 may be connected to each other, for example, via electrical connection 310 (e.g., cable or computer bus) or via any other suitable electrical connection to exchange electrical signals. The image sensor 308 may be connected to the computer system 300, for example, via an external interface, or may be provided as part of the computer system (in other words: inside the computer system, for example, via electrical connection 310).

[0140] The terms “coupled” or “connected” are intended to mean either a direct “coupling” (e.g., via a physical link) or a direct “connection” as well as an indirect “coupling” or an indirect “connection” (e.g., via a logical link).

[0141] It should be appreciated that what has been described above for one of the methods can similarly hold for the computer system 300.

[0142] List of reference signs

[0143] 100 diagram of a system according to various embodiments

[0144] 102 car-specific configuration parameters

[0145] 104 camera

[0146] 106 vehicle state data

[0147] 108 person presence module

[0148] 110 body keypoint detection module

[0149] 112 region of interest generator

[0150] 114 hand tracking module

[0151] 116 seat belt classifier

[0152] 118 seat belt detector

[0153] 120 hand position assignment module

[0154] 122 hand trajectory classification module

[0155] 124 hand pose classification module

[0156] 126 hand position likelihood

[0157] 128 hand trajectory likelihood

[0158] 130 hand pose likelihood

[0159] 132 seat belt region classifier likelihood

[0160] 134 seat belt detector likelihood

[0161] 136 fusion module

[0162] 200 flowchart illustrating a method for determining a state indicative of whether a seat belt of a vehicle is used according to various embodiments

[0163] 202 the step of acquiring an image of a portion of the interior of the vehicle 204 determines whether the state can be estimated based on the image 206 if it is determined that the state cannot be estimated based on the image, a step of determining a probability of a change in the state based on the acquired image, and updating the state based on the determined probability 300 computer system according to various embodiments

[0164] 302 processor

[0165] 304 memory

[0166] 306 non-transitory data storage

[0167] 308 image sensor

[0168] 310 connection

Claims

1. A computer-implemented method for determining a state indicative of whether a safety belt of a vehicle is used, the method comprising the steps of: - acquiring (202) at least one image of a portion of an interior of the vehicle; - determining (204) whether the at least one image comprises a belt buckle receiver; - if it is determined that the at least one image comprises the belt buckle receiver, determining (206) the state based on the image, - else performing: - extracting (208) from the acquired image information related to a user of the safety belt and / or information related to a belt buckle of the safety belt; - determining (210) a probability of a state change based on the extracted information; and - updating (212) the state based on the determined probability.

2. The computer-implemented method according to claim 1, wherein, - the extracted information comprises at least one keypoint of at least one body part of the user.

3. The computer-implemented method according to claim 1, wherein - the extracted information comprises information on a position of the belt buckle.

4. The computer-implemented method according to claim 1, wherein - the portion of the interior of the vehicle comprises at least one of a portion near a belt buckle of the safety belt or a portion near a belt buckle receiver of the safety belt.

5. The computer-implemented method according to claim 1, further comprising the step of: - if it is determined that the at least one image does not comprise the belt buckle receiver, determining a trajectory of a hand of a user of the safety belt relative to the belt buckle; wherein the probability is determined based on the trajectory.

6. The computer-implemented method according to claim 5, further comprising the step of: - if it is determined that the hand is on a trajectory towards the belt buckle receiver in the case that it is determined that the at least one image does not comprise the belt buckle receiver, setting a probability of changing the state from “unbuckled” to “buckled” higher than a probability of changing the state from “buckled” to “unbuckled”.

7. The computer-implemented method according to claim 5, further comprising the step of: - if it is determined that the hand is on a trajectory away from the belt buckle receiver in the case that it is determined that the at least one image does not comprise the belt buckle receiver, setting a probability of changing the state from “unbuckled” to “buckled” lower than a probability of changing the state from “buckled” to “unbuckled”.

8. The computer-implemented method according to claim 1, further comprising the step of: - if it is determined that the at least one image does not comprise the belt buckle receiver, classifying a posture of a hand of a user of the safety belt near the belt buckle; wherein the probability is determined based on the posture.

9. The computer-implemented method according to claim 8, further comprising the step of: If the pose of the hand is classified as a grasping pose in the event that it is determined that the at least one image does not include the buckle receiver, the probability of changing the state from "unbuckled" to "buckled" is set to be higher than the probability of changing the state from "unbuckled" to "buckled" when a pose of an open hand is determined as the pose of the hand.

10. The computer-implemented method of claim 1, further comprising: if it is determined that the at least one image does not include the buckle receiver, determining whether the buckle is located in the user's hand; wherein the probability is determined based on whether the buckle is located in the user's hand.

11. The computer-implemented method of claim 1, further comprising: determining a gaze direction of a user of the seat belt relative to the buckle; wherein the probability is determined based on the gaze direction.

12. The computer-implemented method of claim 1, wherein the state comprises a probability that the seat belt of the vehicle is used.

13. A computer system (300) comprising a plurality of computer hardware components configured to perform the steps of the computer-implemented method according to any one of claims 1 to 12.

14. A vehicle comprising the computer system (300) according to claim 13 and a sensor (308) configured to acquire images.

15. A non-transitory computer-readable medium comprising instructions for performing the computer-implemented method according to any one of claims 1 to 12.

Images