System and method for assisting a vehicle occupant

Abstract

A system (102) and a method (500) for assisting a vehicle (100) occupant. The system (102) includes one or more speed sensors (104) configured to detect the speed of the vehicle (100), one or more image capture units (106) configured to capture in real time at least one of one or more images and one or more videos of the occupant, and a control unit (108) configured to determine the posture of the occupant as a normal posture or an abnormal posture based on at least one of the captured one or more images and videos when it is detected that the speed is greater than a predetermined speed, and to execute one or more predetermined operations when it is determined that the posture of the occupant is in an abnormal posture for a time period greater than a predetermined time period.

Description

Technical Field

[0001] The present invention relates to a system and method for assisting a vehicle occupant. More particularly, the present invention relates to a system and method for assisting an occupant based on the occupant's posture.

Background Art

[0002] In the current situation, vehicles do not have a simple and economical means to monitor the posture of an occupant while the occupant is in the vehicle and assist the occupant when the occupant's posture is abnormal / inappropriate. Also, vehicles do not have a simple and economical means to determine the number of occupants on the vehicle for adjustment of the vehicle's suspension system. Further, when the occupant's posture is inappropriate, or when the occupant's posture indicates that the occupant is inattentive, the vehicle does not have a means to control the vehicle, i.e., a means to decelerate the vehicle or perform an emergency braking operation.

[0003] In some vehicles, one or more sensors are disposed on the vehicle and / or on the vehicle's occupant to detect the number of occupants on the vehicle and / or the posture of the occupant. In one example, one or more load sensors are provided on the vehicle's handlebar to determine whether the occupant is holding the handlebar. In another example, one or more load sensors are provided under the vehicle's seat to indicate the load on the vehicle for adjustment of the vehicle's suspension system based on the number of occupants on the vehicle. In another example, one or more sensors such as an inertial measurement unit, an accelerometer, a gyroscope are used to determine the inclination of the occupant and thereby determine the inclination of the vehicle. In another example, one or more sensors are provided on the joints of the occupant and / or on the occupant's clothing including a jacket, a helmet, etc. Using a plurality of special sensors to detect or determine the occupant's posture or the number of occupants on the vehicle increases the overall cost of the vehicle, causes discomfort to the occupant, and requires additional hardware components such as wiring and a controller to communicate the occupant's posture or the number of occupants to the vehicle, which is undesirable.

[0004] Also, in the current situation, instead of generating a warning against the root cause of a trouble or accident, i.e., an inappropriate posture or carelessness of the rider, a warning is generated against the position and / or parameters of the vehicle resulting from the inappropriate or abnormal posture of the rider. For example, in the current situation, a warning is given when the vehicle deviates from the lane by more than a predetermined distance, but not against the cause of such deviation, which may be that the rider is looking to the left or right, or that the rider is driving using only one end of the handlebar. Summary of the Invention Problems to be Solved by the Invention

[0005] From this perspective, there is an urgent need to overcome at least the drawbacks of the above prior art. Means for Solving the Problems

[0006] In one aspect of the present invention, a system for assisting a vehicle occupant is disclosed. The system includes one or more speed sensors, one or more image capture units, and a control unit. The one or more speed sensors are mounted on the vehicle. The one or more speed sensors are configured to detect the speed of the vehicle. The one or more image capture units are mounted on the vehicle. The one or more image capture units are configured to capture in real time one or more images and / or one or more videos of the vehicle occupant. In a saddle-type vehicle, the one or more image capture units are mounted on the vehicle such that images and / or videos of both the occupant and the rear seat occupant can be captured by the one or more image capture units. Thus, the present invention can determine the load on the vehicle without using one or more sensors and can also determine the posture of the rear seat occupant. In one embodiment, the one or more image capture units can be mounted at the front of the vehicle. In another embodiment, the one or more image capture units can be mounted on both the front and the rear of the vehicle.

[0007] The control unit is also mounted on the vehicle and communicates with the one or more speed sensors and the one or more image capture units. When it detects that the speed of the vehicle is greater than a predetermined speed, the control unit determines the posture of the occupant as a normal posture or an abnormal posture based on the captured one or more images and / or videos. In certain situations, if the posture of the occupant is determined to be a normal posture, the control unit takes no further action. In certain situations, when it is determined that the posture of the occupant is an abnormal posture over a time period longer than a predetermined time period, one or more predetermined operations are executed by the control unit to assist the vehicle occupant.

[0008] In one embodiment, the one or more image capture units are mounted on the vehicle dashboard or the vehicle instrument cluster.

[0009] In one embodiment, one or more predefined operations include instructing an acoustic warning device to generate an acoustic warning, instructing a visual warning device to generate a visual warning, instructing a tactile warning device to generate a tactile warning, and / or instructing a speed control device mounted on the vehicle to control the speed of the vehicle.

[0010] In one embodiment, the control unit includes a key feature identification unit and a pose determination unit. The key feature identification unit communicates with one or more image capture units. The key feature identification unit is configured to identify an initial set of key features of the occupant in at least one of the captured one or more images and the captured one or more videos. The key feature identification unit is further configured to assign a confidence score to each of the identified key features in the initial set of key features. The key feature identification unit is further configured to determine a final set of key features. The final set of key features depends on the confidence scores assigned to each of the key features in the initial set of key features. The higher the confidence score, the greater the probability of finding a key feature in the captured image and / or video. In one embodiment, key features having a confidence score of 0.6 or higher are included in the final list of key features. However, this value should not be construed as limiting and may be set to different values by the manufacturer. The pose determination unit communicates with the key feature identification unit. The pose determination unit is configured to determine the angles and distances between the key features in the final set of key features in order to estimate the pose of the occupant of the vehicle. The pose determination unit is further configured to compare the estimated pose of the occupant with at least one of one or more predefined normal poses and abnormal poses. Based on the comparison, the pose determination unit determines the pose of the occupant as a normal pose or an abnormal pose.

[0011] One or more key features of the rider include the rider's nose, the rider's upper lip, the rider's lower lip, the tip of the rider's chin, the rider's jaw, the rider's torso, the rider's neck, the rider's left finger joint, the rider's right finger joint, the rider's left eye, the rider's right eye, the rider's right ear, the rider's left ear, the rider's right shoulder, the rider's left shoulder, the rider's left elbow, the rider's right elbow, the rider's left wrist, the rider's right wrist, the rider's waist, and / or the rider's back.

[0012] One or more predetermined abnormal postures of the rider include one hand being on the vehicle's handlebar while the other hand is away from the vehicle's handlebar, both hands being away from the vehicle's handlebar, the head being partially or fully turned to the right or left direction of the vehicle, the head being partially or fully turned upward or downward of the vehicle, the rider standing on the vehicle, the rider being tilted to the left or right direction of the vehicle, and the rider being tilted forward or backward of the vehicle.

[0013] In one embodiment, the control unit is configured to receive the tilt angle of the vehicle with respect to the ground on which the vehicle is traveling. One or more tilt angle sensors are configured to detect the tilt angle of the vehicle with respect to the ground on which the vehicle is moving. One or more tilt angle sensors communicate with the control unit. The tilt angle of the rider with respect to the vehicle is determined by the control unit based on the captured one or more images and / or videos. The tilt angle of the vehicle with respect to the ground is determined based on the tilt angle of the rider with respect to the vehicle and the tilt angle of the vehicle with respect to the ground. Based on the tilt angle of the rider with respect to the ground, the posture of the rider is determined as a normal posture or an abnormal posture. It should be understood that excessive tilting by the rider while the vehicle is already tilted may cause the vehicle to slip. Therefore, excessive tilting by the rider when the vehicle is already tilted is an abnormal posture.

[0014] In another aspect of the present invention, a method for assisting a vehicle occupant is disclosed. The method includes detecting the speed of the vehicle. The step of detecting the speed of the vehicle is performed by one or more speed sensors mounted on the vehicle. The method further includes capturing in real time at least one of one or more images and one or more videos of the occupant. The step of capturing one or more images and / or videos is performed by one or more image capture units mounted on the vehicle. The method further includes determining the posture of the occupant as a normal posture or an abnormal posture based on at least one of the captured one or more images and videos when the speed of the vehicle is greater than a predetermined speed. The step of determining the posture is performed by a control unit communicating with the one or more speed sensors and the one or more image capture units. The method further includes performing one or more predetermined operations when the posture of the occupant is determined as an abnormal posture over a time period greater than a predetermined time period. The step of performing one or more predetermined operations is performed by the control unit.

[0015] In one embodiment, the step of performing one or more predetermined operations includes instructing an acoustic warning device to generate an acoustic warning, instructing a visual warning device to generate a visual warning, instructing a tactile warning device to generate a tactile warning, and / or instructing a speed control device to control the speed of the vehicle. The term "control" here refers to an operation performed by the control unit to limit the speed of the vehicle, such as adaptive cruise control, or an operation performed by the control unit to reduce the speed of the vehicle, such as an emergency braking operation.

[0016] In one embodiment, the step of determining the rider's posture includes: (i) identifying an initial set of key features of the rider in at least one of the captured one or more images and videos; (ii) assigning a confidence score to each of the identified key features in the initial set of key features; (iii) determining a final set of key features based on the confidence scores; (iv) determining the angles and distances between the key features in the final set of key features to estimate the rider's posture; (v) comparing the estimated posture of the rider with one or more predetermined normal postures and abnormal postures; and (vi) determining the estimated posture of the rider as a normal posture or an abnormal posture based on the comparison. Steps (i), (ii), and (iii) are executed by the key feature identification unit of the control unit. The key feature identification unit communicates with one or more image capture units. Steps (iv), (v), and (vi) are executed by the pose determination unit of the control unit. The pose determination unit communicates with the key feature identification unit.

[0017] In one embodiment, the step of determining the rider's posture further includes: (i) determining the inclination angle of the rider with respect to the vehicle based on at least one of the captured one or more images and videos; (ii) receiving the inclination angle of the vehicle with respect to the ground from one or more inclination angle sensors mounted on the vehicle; (iii) determining the inclination angle of the rider with respect to the ground based on the inclination angle of the rider with respect to the vehicle and the inclination angle of the vehicle with respect to the ground; and (iv) determining the posture of the rider as a normal posture or an abnormal posture based on the inclination angle of the rider with respect to the ground. Steps (i) to (iv) are executed by the control unit of the vehicle. The control unit communicates with one or more inclination angle sensors mounted on the vehicle.

[0018] Reference is made to embodiments of the present invention, examples of which may be shown in the accompanying drawings. These drawings are intended to be illustrative and not limiting. It should be understood that the present invention is described generally in the context of these embodiments, but the scope of the present invention is not intended to be limited to these particular embodiments.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6a

Figure 6b

Figure 6c

Figure 7a

Figure 7b

Modes for Carrying Out the Invention

[0020] Here, various features and embodiments of the present invention will be understood from the following further description of this specification described below.

[0021] FIG. 1 is a block diagram of a system 102 for assisting a rider of a vehicle 100 according to an embodiment of the present invention. The term "vehicle" includes not only passenger transportation vehicles but also saddle-type vehicles. Saddle-type vehicles include two-wheeled vehicles such as bicycles, scooters, and motorcycles, three-wheeled vehicles, and four-wheeled vehicles. Passenger transportation vehicles include three-wheeled and four-wheeled vehicles such as autorickshaws, automobiles, trucks, and lorries. The term "vehicle" also includes conventional internal combustion engine vehicles, electric vehicles, and hybrid vehicles.

[0022] As shown, system 102 includes one or more speed sensors 104, one or more image capture units 106, and a control unit 108. One or more speed sensors 104 are mounted on vehicle 100 and configured to detect the speed of vehicle 100. One or more image capture units 106 are mounted on vehicle 100 and configured to capture one or more images and / or videos of the passenger in real time. The control unit 108 is also mounted on vehicle 100 and communicates with one or more speed sensors 104 and one or more image capture units 106. The speed of vehicle 100 is received by the control unit 108 from one or more speed sensors 104. When it is detected that the speed of vehicle 100 is greater than a predetermined speed, the control unit 108 is configured to determine the posture of the passenger as a normal posture or an abnormal posture. If the abnormal posture is maintained by the passenger for a time period greater than a predetermined duration, the control unit 108 is configured to perform one or more predetermined operations. The one or more predetermined operations include any actions performed by the control unit 108 to warn the passenger and / or to control the speed of the vehicle. The term "control" refers to an operation performed by the control unit 108 to limit the speed of vehicle 100, such as adaptive cruise control, or an operation performed by the control unit 108 to reduce the speed of vehicle 100, such as an emergency braking operation.

[0023] In one embodiment, the image capture unit 106 is a camera. In a vehicle 100 having a camera for authentication of the vehicle's passenger, the same camera may be used to capture one or more images and / or videos of the passenger of vehicle 100. The image capture unit 106 can operate in both a daytime mode and a nighttime mode.

[0024] In one embodiment, one or more image capture units 106 are attached to the front of the vehicle 100, such as the dashboard and / or instrument cluster of the vehicle 100, and / or to the rear of the vehicle 100, such as the grab rail of the vehicle 100.

[0025] In one embodiment, the predetermined speed is 5 km per hour. However, this value should not be construed as limiting and may include any value preset by the manufacturer of the vehicle 100 in the control unit 108.

[0026] In one embodiment, the predetermined duration is 30 seconds. However, this value should not be construed as limiting and may include any value preset by the manufacturer of the vehicle 100 in the control unit 108.

[0027] In a saddle-type vehicle, the term "rider" may include both the rider of the vehicle and the rider of the rear seat sitting behind the rider of the vehicle 100. The control unit 108 may be configured to determine the postures of both the rider and the rider of the rear seat. It should be understood that an abnormal posture of the rider of the rear seat in a saddle-type vehicle can also result in a vehicle accident. In such a scenario, one or more image capture units 106 are attached such that one or more images and / or videos of the rider and the rider of the rear seat can be captured. In a non-limiting example, one or more image capture units 106 may be attached to the front and the rear of the vehicle 100.

[0028] FIG. 2 is a block diagram of a system 202 for assisting a rider of a vehicle 200 according to another embodiment of the present invention.

[0029] As shown, system 202 includes one or more speed sensors 204, one or more image capture units 206, and a control unit 208. The configurations, interrelationships, and functions of the one or more speed sensors 204, the one or more image capture units 206, and the control unit 208 are the same as those defined for the one or more speed sensors 104, the one or more image capture units 106, and the control unit 108 in FIG. 1 of the present invention.

[0030] FIG. 2 further shows one or more devices for performing one or more predetermined operations by the control unit 208. As shown, the control unit 208 communicates with an acoustic warning device 210, a visual warning device 212, a tactile warning device 214, and a speed control device 216. An acoustic warning device 210, such as a buzzer or horn, is mounted on the vehicle 200 and configured to generate an acoustic warning. A visual warning device 212, such as a light-emitting diode, is mounted on the vehicle 200 and configured to generate a visual warning. A tactile warning device 214, such as a vibrator capable of generating vibration, is mounted on the vehicle 200 and configured to generate a tactile warning. The acoustic warning device 210, the visual warning device 212, and the tactile warning device 214 warn the driver to correct an abnormal posture and avoid an accident / collision. The control unit 208 controls the speed of the vehicle 200 to prevent an accident / collision. Here, the term "control" refers to an operation performed by the control unit 208 to limit the speed of the vehicle 200, such as adaptive cruise control, or an operation performed by the control unit 208 to suppress the speed of the vehicle 200, such as an emergency braking operation. It should be understood that the acoustic warning device 210, the visual warning device 212, the tactile warning device 214, and the speed control device 216 may be operated independently or in combination with each other. Also, the acoustic warning device 210, the visual warning device 212, the tactile warning device 214, and the speed control device 216 include devices already known or developed later.

[0031] Figure 3 is a block diagram of a system 302 for assisting a vehicle occupant according to another embodiment of the present invention.

[0032] As shown, the system 302 includes one or more speed sensors 304, one or more image capture units 306, and a control unit 308. The configurations, interrelationships, and functions of the one or more speed sensors 304, the one or more image capture units 306, and the control unit 308 are the same as those defined for the one or more speed sensors 104, the one or more image capture units 106, and the control unit 108 in FIG. 1 of the present invention. Also, the configurations, interrelationships, and functions of the acoustic warning device 310, the visual warning device 312, the tactile warning device 314, and the speed control device 316 are the same as those defined for the acoustic warning device 210, the visual warning device 212, the tactile warning device 214, and the speed control device 216 in FIG. 2 of the present invention.

[0033] FIG. 3 further shows one or more tilt angle sensors 318 mounted on vehicle 300 and communicating with control unit 308 to determine the posture of the rider. The one or more tilt angle sensors 318 are configured to detect the tilt angle of vehicle 300 with respect to the ground on which vehicle 300 is traveling. The tilt angle of vehicle 300 with respect to the ground determined by the one or more tilt angle sensors 318 is transmitted to control unit 308. Control unit 308 is configured to determine the tilt angle of the rider with respect to vehicle 300. Control unit 308 determines the tilt angle of the rider with respect to vehicle 300 by processing one or more images and / or videos captured by one or more image capture units 306. For the purposes of the present invention, the tilt angle of the rider with respect to vehicle 300 can be determined using system 402 shown in FIG. 4 of the present invention and methods 600 shown in FIGS. 6a, 6b, and 6c. However, system 402 and method 600 for determining the tilt angle of the rider with respect to the vehicle should not be construed as limiting, and other currently known or later developed systems and methods can also be used by the present invention to determine the tilt angle of the rider with respect to vehicle 300.

[0034] When receiving the tilt angle of the vehicle 300 from one or more tilt angle sensors 318 and determining the tilt angle of the occupant relative to the vehicle 300, the control unit 308 determines the tilt angle of the occupant relative to the ground using techniques known currently or developed later. Based on the tilt angle of the occupant relative to the ground, the control unit 308 determines whether the occupant's posture is normal or abnormal. In one embodiment, the tilt angle of the occupant is compared with one or more predetermined tilt angles to determine whether the occupant's posture is normal or abnormal. The one or more predetermined angles are preset in the control unit 308 by the manufacturer of the vehicle 300. It should be understood that if the occupant tilts excessively while the vehicle 300 is already tilted, it will cause the vehicle 300 to slip. Therefore, when the vehicle 300 is already tilted and the occupant tilts excessively, it is an abnormal posture. The tilt angle sensor can be an inertial measurement unit (IMU), an accelerometer, a gyroscope, and other sensors known currently or developed later. The tilt angle sensor sends the roll rate and yaw rate of the vehicle to the control unit 308.

[0035] Figure 4 is a block diagram of a system 402 for assisting an occupant of a vehicle 400 according to another embodiment of the present invention.

[0036] As shown, system 402 includes one or more speed sensors 404, one or more image capture units 406, and a control unit 408. The control unit 408 includes a key feature identification unit 408a and a pose determination unit 408b. One or more speed sensors 404 are mounted on vehicle 400 and configured to detect the speed of vehicle 400 and transmit it to the control unit 408. One or more image capture units 406 are configured to capture one or more images and / or videos of the vehicle's 400 occupants and communicate with the key feature identification unit 408a. The key feature identification unit 408a is configured to identify an initial set of key features of the occupant in at least one of the captured one or more images and videos. The key feature identification unit 408a further assigns a confidence score to each of the identified key features in the initial set of key features and is further configured to determine a final set of key features based on the confidence scores. The confidence score indicates the probability that a key feature exists in the captured one or more images and videos. The higher the confidence score, the greater the probability of finding a key feature in the captured one or more images and videos. In one embodiment, key features having a confidence score of 0.6 or higher are included in the final list of key features. However, this value should not be construed as limiting and may be set to different values by the manufacturer. One or more key features of the occupant include the occupant's nose, upper lip, lower lip, tip of the occupant's chin, occupant's jaw, occupant's torso, occupant's neck, occupant's left finger joints, occupant's right finger joints, occupant's left eye, occupant's right eye, occupant's right ear, occupant's left ear, occupant's right shoulder, occupant's left shoulder, occupant's left elbow, occupant's right elbow, occupant's left wrist, occupant's right wrist, occupant's waist, and / or occupant's back. By way of non-limiting example, at least 22 such key features can be identified by the key feature identification unit 408a.

[0037] The pose determination unit 408b communicates with the key feature identification unit 408a. The pose determination unit 408b is configured to determine the angles and distances between key features in the final set of key features to estimate the rider's posture. For example, the distance between both eyes, the distance between the left shoulder and the tip of the chin, the angle of the tip of the chin from the left shoulder, etc. are determined by the pose determination unit 408b. When the head is rotated to the right, the change / increase in the angle and distance of the tip of the chin from the left shoulder gives an indication that the rider is looking to the right while boarding the vehicle 400. In other words, the estimated posture of the rider suggests that the rider is looking to the right while boarding the vehicle 400. The pose determination unit 408b is further configured to compare the estimated posture of the rider with at least one of one or more predetermined normal postures and abnormal postures. One or more predetermined abnormal postures of the rider include that one hand is on the handlebar of the vehicle 400 while the other hand is away from the handlebar of the vehicle 400, both hands are away from the handlebar of the vehicle 400, the head is partially or fully turned to the right or left direction of the vehicle 400, the head is partially or fully turned upward or downward of the vehicle 400, the rider is standing on the vehicle 400, the rider is tilted to the left or right direction of the vehicle 400, and / or the rider is tilted to the forward or backward direction of the vehicle 400. Based on the above comparison, the pose determination unit 408b determines the estimated posture of the rider as a normal posture or an abnormal posture. In the above example, the posture of the rider looking to the right can be determined as an abnormal posture by the control unit 408.

[0038] The pose determination unit 408b and the key feature identification unit 408a can be data-driven models such as trained neural networks, genetic algorithms, and algorithms known currently or developed later. During the training phase of the data-driven model, the ranges of angles and polygons formed between different key features that define different (normal and abnormal) poses are performed. When trained and tested, in real time, the angles and joint parameters between the identified key points are obtained, classified into different poses by the key feature identification unit 408a and the pose determination unit 408b, and normal poses and abnormal poses are identified.

[0039] When determining a pose as an abnormal pose, the control unit 408 determines the time period for which the abnormal pose is maintained by the vehicle 400's occupant. If such a duration of time is greater than a predetermined time period, the control unit performs one or more predetermined operations. The one or more predetermined operations include instructing the acoustic warning device 410 to generate an acoustic warning, instructing the visual warning device 412 to generate a visual warning, instructing the tactile warning device 414 to generate a tactile warning, and / or instructing the speed control device 416 to control the speed of the vehicle 400. The acoustic warning device 410, the visual warning device 412, the tactile warning device 414, and the speed control device 416 are mounted on the vehicle 400 and include devices known currently or developed later.

[0040] FIG. 5 is a flowchart showing a method 500 for assisting an occupant of a vehicle according to an embodiment of the present invention.

[0041] In step 501, the method includes detecting the speeds of the vehicles 100, 200. The step of detecting the speeds of the vehicles 100, 200 is performed by one or more speed sensors 104, 204 mounted on the vehicles 100, 200.

[0042] In step 502, the method includes capturing in real time at least one of one or more images and one or more videos of the passenger. The step of capturing one or more images and / or videos is performed by one or more image capture units 106, 206 mounted on the vehicle 100.

[0043] In step 503, the method includes determining the posture of the passenger as a normal posture or an abnormal posture based on at least one of the captured one or more images and videos when it is detected that the speed is greater than a predetermined speed. The step of determining the posture of the passenger as a normal posture or an abnormal posture is performed by control units 108, 208 that communicate with one or more speed sensors 104, 204 and one or more image capture units 106, 206.

[0044] In step 504, the method further includes performing one or more predetermined operations when it is determined that the posture of the passenger is in an abnormal posture for a time period greater than a predetermined time period. The step of performing one or more predetermined operations is performed by control units 108, 208.

[0045] In one embodiment, step 504 of performing includes instructing the acoustic warning device 210 to generate an acoustic warning, instructing the visual warning device 212 to generate a visual warning, instructing the tactile warning device 214 to generate a tactile warning, and / or instructing the speed control device 216 to control the speed of the vehicle. The term "control" here refers to an operation performed by the control unit to limit the speed of the vehicle, such as adaptive cruise control, or an operation performed by the control unit to reduce the speed of the vehicle, such as an emergency braking operation.

[0046] Figures 6a, 6b, and 6c are flowcharts showing a method 600 for assisting a passenger of a vehicle 400 according to another embodiment of the present invention.

[0047] In step 601, the method includes detecting the speed of the vehicle 400. The step of detecting the speed of the vehicle 400 is performed by one or more speed sensors 404 attached to the vehicle 400.

[0048] In step 602, the method includes capturing in real time at least one of one or more images and one or more videos of the passenger. The step of capturing one or more images and / or videos is performed by one or more image capture units 406 attached to the vehicle 400.

[0049] In step 603, the method includes steps 603a, 603b, 603c, 603d, 603e, and 603f for determining the posture of the passenger as a normal posture or an abnormal posture. Step 603 is performed by a control unit 408 attached to the vehicle 400 and communicating with one or more speed sensors 404 and one or more image capture units 406.

[0050] In step 603a, the method includes identifying an initial set of key features of the passenger in at least one of the captured one or more images and videos. The step of identifying the initial set of key features is performed by a key feature identification unit 408a of the control unit 408. Each frame of the image or video is sent to the key feature identification unit 408a. One or more key features of the passenger may include the passenger's nose, upper lip, lower lip, tip of the jaw, jaw, torso, neck, left finger joints, right finger joints, left eye, right eye, right ear, left ear, right shoulder, left shoulder, left elbow, right elbow, left wrist, right wrist, waist, and / or back.

[0051] In step 603b, the method includes assigning a confidence score to each of the identified key features in the initial set of key features. The assigning step is performed by the key feature identification unit 408a of the control unit 408. The higher the confidence score, the greater the probability of finding a key feature in the captured image and / or video. In one embodiment, key features having a confidence score of 0.6 or higher are included in the final list of key features. However, this value should not be construed as a limitation and may be set to different values by the manufacturer.

[0052] In step 603c, the method includes determining a final set of key features based on the confidence scores. The step of determining the final set of key features is performed by the key feature identification unit 408a of the control unit 408.

[0053] In step 603d, the method includes determining the angles and distances between the key features in the final set of key features to estimate the rider's posture. The step of determining the angles and distances is performed by the pose determination unit 408b of the control unit 408.

[0054] In step 603e, the method includes comparing the estimated posture of the rider with one or more predetermined normal postures and abnormal postures. The step of comparing the estimated posture with one or more predetermined normal postures and abnormal postures is performed by the pose determination unit 408b of the control unit 408. One or more predetermined abnormal postures of the rider include that one hand is on the handlebar of the vehicle while the other hand is away from the handlebar of the vehicle 400, both hands are away from the handlebar of the vehicle 400, the head is partially or fully turned to the right or left direction of the vehicle 400, the head is partially or fully turned upward or downward of the vehicle, the rider is standing on the vehicle 400, the rider is inclined to the left or right direction of the vehicle 400, and / or the rider is inclined to the front or rear direction of the vehicle 400.

[0055] In step 603f, the method includes determining the estimated posture of the rider as a normal posture or an abnormal posture based on the comparison in step 603e. The step of determining the estimated posture as a normal posture or an abnormal posture is performed by the pose determination unit 408b of the control unit 408.

[0056] In step 604, the method includes performing one or more predetermined operations when it is determined that the posture of the rider is an abnormal posture over a time period greater than a predetermined time period. The step of performing one or more predetermined operations is performed by the control unit 408.

[0057] In one embodiment, step 604 of performing includes instructing the acoustic warning device 410 to generate an acoustic warning, instructing the visual warning device 412 to generate a visual warning, instructing the tactile warning device 416 to generate a tactile warning, and / or instructing the speed control device 418 to limit the speed of the vehicle 400.

[0058] Figures 7a and 7b are flowcharts showing a method 700 for assisting a passenger of a vehicle 300 according to another embodiment of the present invention.

[0059] In step 701, the method includes detecting the speed of the vehicle 300. The step of detecting the speed of the vehicle 300 is performed by one or more speed sensors 304 attached to the vehicle 300.

[0060] In step 702, the method includes capturing at least one of one or more images and one or more videos of the passenger in real time. The step of capturing one or more images and / or videos is performed by one or more image capture units 306 attached to the vehicle 300.

[0061] In step 703, the method includes steps 703a, 703b, 703c, and 703d for determining the posture of the passenger as a normal posture or an abnormal posture. Step 703 is performed by a control unit 308 attached to the vehicle 300 and communicating with one or more speed sensors 304 and one or more image capture units 306.

[0062] In step 703a, the method includes determining the inclination angle of the passenger with respect to the vehicle 300 based on at least one of the captured one or more images and videos.

[0063] In step 703b, the method includes receiving the inclination angle of the vehicle 300 with respect to the ground on which the vehicle is traveling. The inclination angle of the vehicle 300 with respect to the ground is detected by one or more inclination angle sensors 318 attached to the vehicle 300 and transmitted to the control unit 308.

[0064] In step 703c, the method includes determining the inclination angle of the passenger with respect to the ground based on the inclination angle of the passenger with respect to the vehicle 300 and the inclination angle of the vehicle 300 with respect to the ground.

[0065] In step 703d, the method includes determining the posture of the rider as a normal posture or an abnormal posture based on the inclination angle of the rider with respect to the ground. In one embodiment, the inclination angle of the rider with respect to the ground is compared with one or more predetermined inclination angle values indicating a normal posture or an abnormal posture. It should be understood that if the rider inclines excessively while the vehicle is already inclined, it will cause the vehicle to slip. Therefore, when the vehicle is already inclined and the rider inclines excessively, it is an abnormal posture.

[0066] In step 704, the method includes performing one or more predetermined operations when it is determined that the posture of the rider is an abnormal posture over a time period greater than a predetermined time period. The step of performing one or more predetermined operations is executed by the control unit 308.

[0067] In one embodiment, the steps of performing include instructing the acoustic warning device 310 to generate an acoustic warning, instructing the visual warning device 312 to generate a visual warning, instructing the tactile warning device 314 to generate a tactile warning, and / or instructing the speed control device 316 to control the vehicle. Here, the term "control" refers to an operation executed by the control unit to limit the speed of the vehicle, such as adaptive cruise control, or an operation executed by the control unit to suppress the speed of the vehicle, such as an emergency braking operation.

[0068] It should be understood that the typical hardware configuration of the control units 108, 208, 308, 408 may include a set of instructions that can be executed to cause the control units 108, 208, 308, 408 to execute the methods disclosed above.

[0069] The control units 108, 208, 308, 408 may include a processor that can be a central processing unit (CPU), a graphics processing unit (GPU), or both. The processor can be one or more general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or other currently known or later developed devices for analyzing and processing data. The processor may implement a software program such as manually generated, i.e., programmed code.

[0070] The control units 108, 208, 308, 408 can include a memory. The memory can be a main memory, static memory, or dynamic memory. The memory can include various types of computer-readable storage media such as volatile and non-volatile storage media including, but not limited to, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media, etc. The memory is operable to store instructions executable by the processor. The illustrated or described functions, operations, or tasks can be executed by a programmed processor that executes instructions stored in the memory.

[0071] The control units 108, 208, 308, 408 may further include a display unit such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a flat panel display, a solid state display, a cathode ray tube, or other currently known or later developed display device for outputting determined information. The display can function as an interface for the user to view the functions of the processor or, specifically, as an interface with the software stored in the memory.

[0072] Furthermore, the control units 108, 208, 308, 408 may include an input device configured to enable a user to interact with any of the components of the control units 108, 208, 308, 408. The input device can be a numeric keypad, a keyboard, or a cursor control device such as a mouse or joystick, a touch screen display, a remote control, or any other device that operates to interact with the control units 108, 208, 308, 408.

[0073] The control units 108, 208, 308, 408 can also include a disk drive unit or an optical drive unit. The disk drive unit can include one or more instruction sets, for example, a computer-readable medium into which software can be incorporated. Further, the instructions can embody one or more of the methods or logics as described. In a particular example, the instructions can be fully or at least partially present in memory or in a processor during execution by the control units 108, 208, 308, 408. The memory and the processor can also include a computer-readable medium as described above. The present invention contemplates a computer-readable medium that includes instructions or receives and executes instructions in response to a propagated signal such that a device connected to a network can communicate data via the network. Further, the instructions can be transmitted or received via the network. The network can include a wired network, a wireless network, an Ethernet AVB network, or a combination thereof. The wireless network can be a cellular phone network. Further, the network can be a public network such as the Internet, a private network such as an intranet, or a combination thereof, and can utilize various networking protocols that are currently available or developed later.

[0074] The claimed features / method steps of the present invention as described above are not routine, not conventional, and not well understood in the art because the claimed steps enable the following solutions to existing problems in the prior art. Specifically, the technical problem of a passenger taking an abnormal posture while riding in vehicles 100, 200, 300, 400 is solved by the present invention.

[0075] The present invention improves the safety of passengers in vehicles 100, 200, 300, 400. The passengers in vehicles 100, 200, 300, 400 receive a warning when it is determined by the present invention that the passenger's posture is abnormal for a time period greater than a predetermined time period. The present invention can also perform one or more control operations on vehicles 100, 200, 300, 400, such as reducing the speed of vehicles 100, 200, 300, 400 or performing an emergency braking operation, to prevent accidents in vehicles 100, 200, 300, 400 when it is determined that the passenger's posture is abnormal.

[0076] The present invention enhances the recognition of the riding patterns of vehicles 100, 200, 300, 400. When a warning is given for the determination of an abnormal posture, the passenger of the vehicle can correct the passenger's posture while riding in vehicles 100, 200, 300, 400. Further, the present invention can monitor the passenger's posture and recommend the best riding posture for the passenger's bad posture to reduce fatigue and enhance safety. The present invention can be used to make the passenger understand the posture in which the passenger drives most of the time and to give the passenger suggestions on how to improve the riding posture and thus feel less fatigue.

[0077] The present invention provides safety-critical functions such as controlling or suppressing the speed of the vehicle when it is determined that an abnormal posture is maintained by the passenger for a time interval greater than a predetermined time interval and further when the passenger does not correct the posture after one or more warnings.

[0078] The present invention does not require a plurality of sensors on the rider's body to determine the rider's posture, which reduces the overall cost of the vehicle.

[0079] In the case of a saddle-type vehicle, the present invention can also determine the presence and posture of the rider in the rear seat.

[0080] The present invention can also be used to identify the load on the vehicle, i.e., the number of riders on the vehicle, without using sensors. The number of riders on the vehicle can be determined by one or more of the image capture units 106, 206, 306, 406 of the present invention.

[0081] Although the present invention has been described with respect to several embodiments, it will be apparent to those skilled in the art that various changes and modifications may be made without departing from the scope of the present invention as defined in the following claims.

Claims

1. A system (102, 202, 302, 402) for assisting a vehicle occupant of a vehicle (100, 200, 300, 400), comprising: One or more speed sensors (104, 204, 304, 404) mounted on the vehicle (100, 200, 300, 400) and configured to detect the speed of the vehicle (100, 200, 300, 400); One or more image capture units (106, 206, 306, 406) mounted on the vehicle (100, 200, 300, 400) and configured to capture in real time at least one of one or more images and one or more videos of the occupant; A control unit (108, 208, 308, 408) mounted on the vehicle (100, 200, 300, 400) and communicating with the one or more speed sensors (104, 204, 304, 404) and the one or more image capture units (106, 206, 306, 406); Wherein the control unit (108, 208, 308, 408) is configured to: When detecting that the speed is greater than a predetermined speed, determine the posture of the occupant as a normal posture or an abnormal posture based on at least one of the captured one or more images and videos; When determining that the posture of the occupant is the abnormal posture over a time period greater than a predetermined time period, perform one or more predetermined operations. A system (102, 202, 302, 402) configured to perform the above.

2. The control unit (408) is configured to: - Communicate with the one or more image capture units (406), - Identify an initial set of key features of the occupant in at least one of the captured one or more images and videos, - Assign a confidence score to each of the identified key features in the initial set of key features, - Determine a final set of key features based on the confidence scores. A key feature identification unit (408a) configured to perform the above; - Communicate with the key feature identification unit (408a), - Determine the angles and distances between the key features in the final set of key features to estimate the posture of the occupant. - Comparing the estimated posture of the rider with at least one of one or more predetermined normal postures and abnormal postures; - Determining the estimated posture of the rider as a normal posture or an abnormal posture based on the comparison A pose determination unit (408b) configured to perform The system (402) according to claim 1, comprising **Claim 3** The one or more predetermined operations include at least one of instructing an acoustic warning device (210, 310, 410) to generate an acoustic warning, instructing a visual warning device (212, 312, 412) to generate a visual warning, instructing a tactile warning device (214, 314, 414) to generate a tactile warning, and instructing a speed control device (216, 316, 416) mounted on the vehicle to control the speed of the vehicle (100, 200, 300, 400). The system (202, 302, 402) according to claim 1. **Claim 4** The one or more image capture units (106, 206, 306, 406) are mounted on at least one of the dashboard of the vehicle (100, 200, 300, 400) and the instrument cluster of the vehicle (100, 200, 300, 400). The system (102, 202, 302, 402) according to claim 1. **Claim 5** One or more key features of the rider include at least one of the rider's nose, the rider's upper lip, the rider's lower lip, the tip of the rider's jaw, the rider's jaw, the rider's torso, the rider's neck, the rider's left finger joints, the rider's right finger joints, the rider's left eye, the rider's right eye, the rider's right ear, the rider's left ear, the rider's right shoulder, the rider's left shoulder, the rider's left elbow, the rider's right elbow, the rider's left wrist, the rider's right wrist, the rider's waist, and the rider's back. The system (402) according to claim 2. **Claim 6** The predetermined abnormal posture of the rider is One hand is on the handlebar of the vehicle (400) while the other hand is away from the handlebar of the vehicle (400); Both hands are away from the handlebar of the vehicle (400); The head is partially or fully turned to the right or left of the vehicle (400). The head is partially or fully turned upward or downward of the vehicle (400). The rider is standing on the vehicle (400). The rider is leaning to the left or right of the vehicle (400). The rider is leaning forward or backward of the vehicle (400). The system (402) according to claim 2, including at least one of the above.

7. The control unit (308) receives the inclination angle of the vehicle (300) with respect to the ground from one or more inclination angle sensors (318) attached on the vehicle (300). determines the inclination angle of the rider with respect to the vehicle (300) based on at least one of the captured one or more images and videos. determines the inclination angle of the rider with respect to the ground based on the inclination angle of the vehicle (300) with respect to the ground and the inclination angle of the rider with respect to the vehicle (300). determines the posture of the rider as the normal posture or the abnormal posture based on the inclination angle of the rider with respect to the ground. The system (302) according to claim 1, configured to perform the above.

8. A method (500, 600, 700) for assisting a rider of a vehicle (100, 200, 300, 400), detecting the speed of the vehicle (100, 200, 300, 400) by one or more speed sensors (104, 204, 304, 404) attached on the vehicle (100, 200, 300, 400) (501, 601, 701); capturing in real time at least one of one or more images and one or more videos of the rider by one or more image capturing units (106, 206, 306, 406) attached on the vehicle (100, 200, 300, 400) (502, 602, 702); when the control unit (108, 208, 308, 408) detects that the speed is greater than a predetermined speed, determining the posture of the rider as a normal posture or an abnormal posture based on at least one of the captured one or more images and videos (503, 603, 703). When the control unit (108, 208, 308, 408) determines that the posture of the rider is the abnormal posture over a time period longer than a predetermined time period, executing one or more predetermined operations (503, 603, 703); A method including this. **Claim 9** The step of executing includes: Instructing, by the control unit (108, 208, 308, 408), an acoustic warning device (210, 310, 410) to generate an acoustic warning; Instructing, by the control unit (108, 208, 308, 408), a visual warning device (212, 312, 412) to generate a visual warning; Instructing, by the control device (108, 208, 308, 408), a tactile warning device (214, 314, 414) to generate a tactile warning; Instructing, by the control device (108, 208, 308, 408), a speed control device (216, 316, 416) to control the speed of the vehicle (100, 200, 300, 400); The method (500, 600, 700) according to claim 8, including at least one of the above. **Claim 10** The step of determining the posture of the rider includes: Identifying, by a key feature identification unit (408a), an initial set of key features of the rider in at least one of the captured one or more images and videos (603a); Assigning, by the key feature identification unit (408a), a confidence score to each of the identified key features in the initial set of key features (603b); Determining, by the key feature identification unit (408a), a final set of key features based on the confidence score (603c); Determining, by a pose determination unit (408b), angles and distances between the key features in the final set of key features to estimate the posture of the rider (603d); Comparing, by the pose determination unit (408b), the estimated posture of the rider with one or more predetermined normal postures and abnormal postures (603e); Determining, by the pose determination unit (408b), the estimated posture of the rider as a normal posture or an abnormal posture based on the comparison (603f). The method (600) according to claim 8, including the above.

11. The step of determining the rider's posture comprises: - determining, by the control unit (308), a tilt angle of the rider with respect to the vehicle (300) based on at least one of the one or more captured images and videos (703a); - receiving, by the control unit (308) communicating with one or more tilt angle sensors (318) mounted on the vehicle (300), a tilt angle of the vehicle (300) with respect to the ground (703b); - determining, by the control unit (308), a tilt angle of the rider with respect to the ground based on the tilt angle of the rider with respect to the vehicle (300) and the tilt angle of the vehicle (300) with respect to the ground (703c); - determining, by the control unit (308), the posture of the rider as the normal posture or the abnormal posture based on the tilt angle of the rider with respect to the ground (703d). The method (700) according to claim 8, comprising the above steps.

Images