Method and system for monitoring vehicle occupants

Abstract

The present disclosure relates to a method and system for monitoring a vehicle occupant, in particular a method for monitoring an occupant located inside a vehicle. At least one image of the occupant is detected via a sensing device. Via a processing unit, a body point of the occupant is identified based on the at least one image, and a posture of the occupant is classified based on the detected body point. Via an output unit, a suggestion to adjust the posture of the occupant is provided based on the classified posture.

Description

Technical Field

[0001] This disclosure relates to methods and apparatus for monitoring occupants inside a vehicle. Background Technology

[0002] Many vehicles now incorporate devices and systems for monitoring the driver and other occupants. These devices and systems can be termed cockpit sensing solutions. Cameras or other imaging sensors capture image data from inside the vehicle and process these images to extract information about the driver's state, including basic characteristics such as eye position, direction of gaze, or the location of other body parts. This image data can be used for advanced interpretation to detect, for example, driver drowsiness or distraction. The basic features extracted from the image data are typically used in driver-supporting applications.

[0003] However, only general advice, such as that provided by the National Automobile Association, is typically available regarding proper seating posture for vehicle occupants. This advice, for example, involves avoiding back pain in the vehicle by correctly adjusting the seat height and backrest angle relative to the occupant's height and build. Additionally, advice is available on how to adjust the steering wheel's height and distance relative to the driver for proper grip.

[0004] However, such general recommendations may not be easily tailored to individual occupants. Furthermore, an occupant's posture is often uncontrolled relative to the seat and steering wheel for a considerable period of time. Therefore, an occupant may not notice for a considerable time that their posture is quite unfavorable and may cause, for example, back pain. Conversely, when the occupant is about to exit the vehicle, they may notice the effects of an unfavorable posture after a long period of sitting.

[0005] Therefore, there is a need for methods and systems that can monitor occupants in vehicles and provide suggestions for improving occupant posture. Summary of the Invention

[0006] This disclosure provides a computer-implemented method, system, computer system, and non-transitory computer-readable medium.

[0007] In one aspect, this disclosure relates to a computer-implemented method for monitoring occupants located inside a vehicle. According to the method, at least one image of the occupant is detected via a sensing device. A processing unit identifies body points of the occupant based on the at least one image and classifies the occupant's posture based on the detected body points. An output unit provides suggestions for adjusting the occupant's posture based on the classified posture.

[0008] The sensing device can be any device capable of providing some form of image, such as a camera, like an NIR camera, RGB camera, RGB-IR camera, time-of-flight camera, stereo camera, thermal imager, radar or lidar sensor, or some other imaging device configured to generate at least one channel of image information (i.e., a two-dimensional image). The sensing device can also support the acquisition of time-series image data.

[0009] The sensing device is located inside the vehicle and can be configured to capture images of at least one occupant in the seat and cover at least half of the occupant's body within its instrument field of view. At least one image detected by the sensing device can cover the occupant's entire upper body, i.e., from the head area down to the hips. Alternatively, for example, with a top-down camera, only the portion from the shoulders down to the legs may be available. In this case, some features described below (e.g., facial expression classification) may not be available.

[0010] The processing unit can use neural networks, along with preprocessing and post-processing algorithms, to process two-dimensional images or time-series images to detect occupant body points, which can be located at the occupant's head, shoulders, elbows, hips, knees, etc. That is, body points can be, for example, "key" points at joints in the occupant's arms or legs, and / or feature points on the occupant's face. However, any point that can be identified and tracked by the sensing device can be suitable as a body point, such as a point on the occupant's abdomen or back, or even a texture on the occupant's clothing. Therefore, the list of examples of body points provided above is not exhaustive.

[0011] Furthermore, neural networks can provide occupant posture classification to analyze whether an occupant is seated in a healthy manner. In other words, for example, classifying and analyzing an occupant's posture via a neural network to determine whether the current seating configuration is considered healthy and comfortable can represent one of the categories of occupant postures. In contrast, seating positions that may lead to tension or discomfort or pain that may result from prolonged sitting can be identified. For example, the method can detect whether an occupant frequently tilts to one side or forward with one shoulder lower, or whether the occupant has an unfavorable head orientation that increases discomfort over time. Therefore, the method according to this disclosure can detect potentially problematic postures or seating positions before an occupant actually feels discomfort.

[0012] As a result, the method provides suggestions to occupants, such as changing posture or seating position. For example, the method can indicate how to sit in different ways to improve comfort. During long journeys, if no change in posture is detected for a certain period, the method can also encourage occupants to change position after a period of time.

[0013] The output unit can provide suggestions for adjusting occupant posture via visual responses, such as icons or dialog boxes displayed on the screen of a head-up display or infotainment system. For example, when driver drowsiness may be detected, an icon similar to a coffee pot icon that might appear in other applications could be displayed. Additionally or alternatively, short animations or video clips can be displayed to encourage occupants to change their seating position, do some stretching exercises, or take a rest.

[0014] Therefore, the method according to this disclosure can interactively improve occupant posture during driving. Since individual body points, which are characteristics of the respective occupants, are detected, the classification of occupant postures and the provision of suggestions are tailored to specific occupants with individual positions of body points located, such as the head, shoulders, elbows, etc. Thus, the method can flexibly provide individual suggestions for specific occupants.

[0015] The method may include one or more of the following features:

[0016] Image sequences can be detected via sensing devices. Through a processing unit: a time series of body points can be determined based on the image sequence; displacement vectors can be estimated for each body point using the time series; and occupant movements can be classified based on the displacement vectors. Further suggestions for adjusting the occupant's posture can be provided based on the classified movements.

[0017] The body points may include facial points, which can be identified and tracked based on the image sequence. Facial expressions can be obtained and classified based on the detected facial points via the processing unit. Further suggestions for adjusting the occupant's posture can be provided based on the classified facial expressions. Additionally, a seating comfort score can be determined via the processing unit based on the occupant's classified posture, classified movement, and / or classified facial expressions, and suggestions for adjusting the occupant's posture can be based on the seating comfort score. Suggestions for adjusting the occupant's posture can be provided only if the seating comfort score is less than a predetermined threshold.

[0018] By identifying body points at the beginning and end of a predefined time period, the occupant's position can be monitored within that time period, and suggestions for adjusting the occupant's posture can be provided if the occupant's position does not change during the predefined time period.

[0019] An occupant may be located in the driver's seat, and the body points may include shoulder points, arm points, and / or hand points. The relative positions of the shoulder points, arm points, and / or hand points can be determined relative to the vehicle's steering wheel. Based on these relative positions, suggestions for adjusting the steering wheel can be provided via an output unit.

[0020] The occupant's response to the suggestion can be obtained through the input unit, and the response may include accepting or rejecting the current suggestion. The number and / or type of future suggestions can be determined based on the occupant's response. If the occupant accepts the current suggestion, the output unit can propose a guidance activity for the occupant, which may depend on the vehicle's driving conditions and environment. The occupant's posture can be further categorized based on the occupant's profile stored in the processing unit.

[0021] According to the implementation method, an image sequence can be detected via a sensing device. A processing unit can determine a time series of body points based on the image sequence, estimate displacement vectors for each body point using the time series, and classify occupant movements based on the displacement vectors. An output unit can further provide suggestions for adjusting occupant posture based on the classified occupant movements.

[0022] In this implementation, the suggestion to adjust occupant posture can be based not only on the static seated position obtained from body points in an image at a certain point in time. Additionally, occupant movements can be considered by classifying occupant movements based on displacement vectors. For example, occupant movements such as frequently reaching their hands to their neck or back, or performing massage-like actions, might indicate that the occupant is beginning to feel discomfort. Similarly, this onset of discomfort can be indicated by stretching the head and shoulder areas in a characteristic manner to relax muscles (e.g., through circular movements of the shoulders or head). The classification of occupant movements can also be performed using neural networks. Because occupant movements are additionally considered, the reliability of the method for identifying the onset of discomfort can be improved.

[0023] According to another embodiment, body points may include facial points that can be identified and tracked based on image sequences, and facial expressions can be obtained and classified based on the identified facial points via a processing unit. Suggestions for adjusting occupant posture can be further provided based on the classified facial expressions via an output unit.

[0024] An occupant's facial expressions can also indicate overall negative emotions or discomfort, which may be related to the occupant not being seated in a comfortable and healthy manner. On the other hand, facial expressions may be associated with comfort and overall positive emotions, indicating that the occupant is likely seated correctly. Therefore, additionally considering facial expressions can support this approach in the early identification of occupant discomfort caused by incorrect seating position.

[0025] According to another embodiment, a seating comfort score can be determined by a processing unit based on occupant's categorized postures, categorized movements, and / or categorized facial expressions, and suggestions for adjusting occupant posture can be based on this seating comfort score. Further suggestions for adjusting occupant posture can be provided only if the seating comfort score is less than a predetermined threshold. Therefore, the seating comfort score can combine information derived from the analysis of static postures, movements, and occupant facial expressions. In other words, the seating comfort score can be a combined indicator of comfort and enjoyment.

[0026] Because of the consideration of three different aspects, the resulting recommendations for occupants can be more reliable and better tailored to specific occupants. Providing recommendations only for low seating comfort scores (i.e., below a predetermined threshold) can reduce unnecessary messages to occupants on the display or screen.

[0027] By identifying body points at the beginning and end of a predefined time period, the occupant's position can be monitored within that period. If the occupant's position remains unchanged during the predetermined time period, suggestions for adjusting the occupant's posture can be provided. This aspect can be highly relevant during extended periods of travel. Suggestions can be made to encourage the occupant to change his or her position to prevent discomfort before it becomes apparent. Therefore, monitoring the occupant's position can serve as an additional source of information upon which suggestions can be made.

[0028] According to another embodiment, the occupant may be located in the driver's seat. Body points may include shoulder points, arm points, and / or hand points. In this case, the relative positions of the shoulder points, arm points, and / or hand points relative to the vehicle steering wheel can be determined. Based on this relative position, a steering wheel adjustment suggestion can be provided via an output unit. Incorrect occupant positioning relative to the steering wheel can be another source of discomfort during driving. Therefore, additional suggestions can be provided to adjust the steering wheel based on the relative position of the steering wheel to selected body points of the occupant.

[0029] The occupant's response to a suggestion can be acquired via an input unit, where the response can include accepting or rejecting the current suggestion. The number and / or type of future suggestions can be determined based on the occupant's response. Therefore, the occupant can interactively control the number and / or type of suggestions. This can facilitate the adaptation or customization of suggestions and the overall approach for specific occupants. Accepting or rejecting a current suggestion can be done using gesture commands, voice input, a button to be pressed, or a touchscreen command. By taking the occupant's response into account, unwanted suggestions can be suppressed during further driving.

[0030] If the occupant accepts the current suggestion, the output unit can propose guided activities for the occupant, which can be tailored to the vehicle's driving conditions and environment. That is, the proposed activities can be adapted to the "context" of the occupant's journey at a specific point in time. For example, if the vehicle is stuck in traffic or stopped at a traffic light (which can be noticed by determining the vehicle's speed), the guided activities could include some light movement or cabin activities. Additionally, if the vehicle is equipped with, for example, positioning and mapping devices, the guided activities can be based on predictions of the best parking locations along the vehicle's intended route. For example, the current suggestion and guided activities could include a suggestion to park, for example, at a park near the intended route, and leave the vehicle for a period of time to perform some suggested exercises. Guided activities can also be provided during autonomous driving. In short, guided activities can alleviate occupant discomfort early on. Furthermore, sensing devices and processing units can be used to monitor the guided activities if they are desired by the occupant.

[0031] According to another embodiment, occupant postures can be further categorized based on occupant profiles stored in the processing unit. The profile can be input into the processing unit manually or based on the identification of body and facial points obtained from image sequences. The occupant profile can further include responses to previous suggestions provided by the output unit. Furthermore, the occupant profile can also include recorded or documented seating comfort scores determined and stored in the processing unit during previous rides. In this way, the occupant profile helps to tailor the method for specific occupants, as it influences the classification of occupant postures. Therefore, occupant feedback can be included in the occupant profile, thus enabling a feedback loop in the method to correctly classify occupant postures. Moreover, when providing suggestions based on classified postures, individual preferences and comfort levels can be considered via the occupant profile.

[0032] In another aspect, this disclosure relates to a system for monitoring occupants located inside a vehicle. The system includes: a sensing device configured to detect at least one image of an occupant; a processing unit configured to identify body points of the occupant based on the at least one image and classify the occupant's posture based on the detected body points; and an output unit configured to provide suggestions for adjusting the occupant's posture based on the classified posture.

[0033] As used herein, the terms “processing apparatus,” “processing unit,” and “module” may refer to, be part of, or include: application-specific integrated circuits (ASICs); electronic circuits; combinational logic circuits; field-programmable gate arrays (FPGAs); processors (shared, dedicated, or grouped) that execute code; other suitable components that provide the aforementioned functionality; or combinations of some or all of the foregoing, such as in a system-on-a-chip. The term “module” may include memory (shared, dedicated, or grouped) storing code executed by the processor.

[0034] In summary, the system according to this disclosure includes a sensing device, a processing unit, and an output unit configured to perform the steps of the corresponding methods described above. Therefore, the benefits, advantages, and disclosures described above for the methods are also applicable to the apparatus according to this disclosure.

[0035] According to one embodiment, the system may further include an input unit configured to acquire the occupant's response to the suggestion, wherein the response may include accepting or rejecting the current suggestion. The number and / or type of future suggestions may depend on the occupant's response.

[0036] The processing unit may include machine learning algorithms or rule-based logic for classifying the occupant's posture. Furthermore, the sensing device may be configured to detect facial points, and the processing unit may be configured to classify facial expressions based on these facial points. The output unit may be configured to further provide suggestions for adjusting the occupant's posture based on facial expressions.

[0037] In another aspect, this disclosure relates to a computer system configured to perform some or all of the steps of the computer-implemented methods described herein.

[0038] The computer system may include a processing unit, at least one memory unit, and at least one non-transitory data storage unit. The non-transitory data storage unit and / or memory unit may include computer programs for instructing the computer to perform some or all of the steps or aspects of the computer-implemented methods described herein.

[0039] On the other hand, this disclosure relates to a non-transitory computer-readable medium comprising instructions for performing several or all of the steps or aspects of the computer-implemented methods described herein. The computer-readable medium can be configured as: optical media, such as a CD or DVD; magnetic media, such as a HDD; a SSD; a ROM; flash memory; and so on. Furthermore, the computer-readable medium can be configured as a data storage unit accessible via a data connection such as an Internet connection. For example, the computer-readable medium can be an online data repository or cloud storage.

[0040] This disclosure also relates to a computer program for instructing a computer to perform some or all of the steps or aspects of the computer-implemented methods described herein. Attached Figure Description

[0041] Exemplary embodiments and functions of this disclosure are described herein in conjunction with the following schematically illustrated figures:

[0042] Figure 1 The system depicts the occupants in a vehicle monitored by the system according to this disclosure.

[0043] Figure 2 An overview of the system according to this disclosure is described, and

[0044] Figure 3 Depicting by Figure 2 The system shown executes the following steps. Detailed Implementation

[0045] Figure 1 An occupant 10 inside a vehicle is schematically depicted as monitored by a system 11 according to this disclosure. The occupant 10 is seated in the driver's seat 12, with his or her hands on the vehicle's steering wheel 14.

[0046] System 11 includes a camera as a sensing device 13, which is connected to processing unit 15 (see also...). Figure 2 The processing unit 15 is configured to analyze the image sequence provided by the camera 13. The system 11 also includes an output unit 17, which is configured to provide suggestions for adjusting the posture of the occupant 10 based on the information provided by the processing unit 15, as will be explained in detail below. Additionally, the system includes an input unit 19, which allows the occupant 10 to provide input information to the system in response to the suggestions provided by the output unit 17. Therefore, the input unit 19 communicates with the processing unit 15.

[0047] also, Figure 1 Body points 21 are shown. These body points 21 include face points 23, shoulder points 25, arm points 27, and hand points 29, as well as additional body points 21 located, for example, at the head, back, hips, and / or knees of occupant 10. However, these examples of body points 21 are not exhaustive. Body points 21 and face points 23 are detected as part of image sequence 31 captured by camera 13 (see also...). Figure 3 Part of the data, and body point 21 and face point 23 are identified by the processing unit 15 using a neural network and preprocessing and postprocessing algorithms.

[0048] like Figure 1As shown in the upper part, the occupant's posture in seat 12 is clearly disadvantageous, especially in the area 26 of the occupant's back and the area 28 near the occupant's shoulders. During driving, the occupant 10, as... Figure 1 The posture shown above can cause discomfort to occupant 10, and may even cause back, neck, or shoulder pain. Therefore, the system 11 and method according to this disclosure aim to provide suggestions for improving the posture of occupant 10. Based on these suggestions, occupant 10 will be able to adjust his or her posture to achieve, for example... Figure 1 The lower part shows the appropriate posture. This corrected posture will increase the comfort of occupant 10 and prevent back, shoulder and neck pain in occupant 10.

[0049] In this embodiment of system 11, the sensing device or camera 13 is an RGB camera. Alternatively, the camera may also be an NIR camera, an RGB-IR camera, a time-of-flight camera, a stereo camera, a thermal imager, or some other imaging device capable of generating at least one image information channel (i.e., a two-dimensional image of the occupants 10 inside the vehicle).

[0050] Figure 3 Describing the use of, for example Figure 2 The system 11 shown is a schematic diagram of the steps performed according to the method of this disclosure. Camera 13 is capable of acquiring a time-series image 31. Camera 13 is also configured to capture images of a large portion of the occupant's body, such as the area from the head to the knees.

[0051] The processing unit 15 receives the image sequence 31 as input and processes the image sequence 31 using a series of algorithms (i.e., neural networks, as well as preprocessing and postprocessing algorithms). That is, the processing unit 15 performs body point detection 33 and face point detection 35 in order to identify body points 21 and face points 23 based on the image sequence 31.

[0052] Body point 21 and face point 23 are identified using a neural network, which is part of processing unit 15 and trained to identify points 21 and 23 within rectangular regions of interest associated with body and face candidates. That is, points 21 and 23 are extracted within the identified body and face regions. Additionally, processing unit 15 provides the absolute positions of points 21 and 23 and their relative positions to distinguish between different sitting postures of occupant 10, i.e., body postures in seat 12 (see...). Figure 1 ).

[0053] Based on body point detection 33, processing unit 15 performs static sitting posture classification 37 (see...). Figure 3 In other words, the absolute and relative positions of points 21 and 23 are used for the occupant 10 in seat 12 (see...). Figure 1The static poses of the occupants 10 are classified. Classification is performed by another stage of a neural network implemented in processing unit 15. The parameters or weights of the neural network are determined in a training step using training samples of the poses of multiple occupants 10 as basic facts. Alternatively, processing unit 15 can perform the classification of body points 21 using decision trees or any other machine learning algorithm or some rule-based logic that considers the position of body parts, the relative distances between predefined body points, and / or the distances to known reference points inside the vehicle.

[0054] The neural network of processing unit 15 is trained to distinguish between an upright occupant with an appropriate posture and an occupant who is leaning forward or to one side, or has a bent posture. Furthermore, if occupant 10 is the vehicle driver, processing unit 15 also uses shoulder point 25, arm point 27, and hand point 29 to evaluate these "key" points 25, 27, and 29 relative to the vehicle's steering wheel 14 (see [link to relevant documentation]). Figure 1 The relative position of the steering wheel 14 to the occupant 10 can be determined. For example, the elbow angle of the occupant 10 can be estimated or classified. In this way, the processing unit 15 determines whether the steering wheel 14 is too close, too far, or at an appropriate distance from the occupant 10.

[0055] In addition to static seated posture classification 37, processing unit 15 also performs motion classification 39 based on body point detection 33. Motion classification 39 is used to identify specific movements and body actions of occupant 10. Such movements may indicate, for example, lower back pain in occupant 10. The occupant may also stretch his or her upper body to avoid pain and relax the muscles in that body area. Another example of a specific movement is a rotational movement of the shoulder.

[0056] For action classification 39, body points 21 are tracked on consecutive image frames within the time image sequence 31. Between two image frames, the corresponding displacement vector is estimated for each body point 21. Additionally, other body points less relevant to static sitting posture classification 37 are considered, such as body points 21 located in the chest or abdominal region of the occupant 10. To determine suitable body points 21 to be tracked for action classification 39, the texture of the occupant's clothing can be used to find good features or body points 21 for tracking. Furthermore, pixel-wise segmentation of the occupant's upper body is used to track body points 21 and identify occupant movement. Pixel-wise segmentation is implemented using a deep neural network.

[0057] Alternatively, a foreground / background segmentation method based on a background model can be applied. Furthermore, some body points 21 (e.g., shoulder and hip points) can be connected using polygonal contours instead of pixel-level segmentation, and these polygonal contours can be traced based on the image sequence 31.

[0058] Additionally, a tracking process can be implemented, configured to identify and track continuous regions based on image sequence 31. In other words, the optical flow of continuous regions can be considered. Such tracking regions have well-defined characteristics, such as identical or similar textures on an occupant's clothing. By tracking this region as a whole, a displacement vector can be defined for the entire region as described above. Therefore, if it is not possible to track individual reference points or body points for a specific occupant, region-based tracking can be used.

[0059] A mapping function is used to map the magnitude or absolute value of the displacement vector of each body point 21 to an activity value or exponent. To identify occupant movement, the average absolute value of the displacement vectors selected for a body point 21 belonging to a certain body region can be estimated. Additionally, this average value of all displacement vectors across the entire body of occupant 10 is estimated to determine the general extent of occupant 10 movement. To classify or evaluate the actions or movements of occupant 10, the displacement lengths of body points 21 are normalized for different body regions, for example, by subtracting the average displacement calculated for body points within the same body region. Furthermore, the relative displacement vectors between different body regions are calculated to identify and classify the movement of occupant 10.

[0060] Based on facial point detection 35, the facial expression 41 of occupant 10 is obtained. That is, for example, facial points 23 are classified by another stage of the neural network within processing unit 15 to obtain facial expression 41. Facial expression can be associated with discomfort indicating that occupant 10 is not seated in a comfortable and healthy manner. Conversely, facial expression can also indicate overall positive emotions indicating the comfort level of occupant 10.

[0061] Based on static posture classification 37, action classification 39, and facial expression 41, processing unit 15 performs posture analysis 43 on occupant 10. Posture analysis 43 generates signals that serve as input to suggestion engine 45, which is the output unit 17 (see...). Figure 2 As part of the recommendation engine 45, the signals provided include the time the occupant 10 is in the current posture, i.e., the time during which there is no physical activity, where arm and hand movements are ignored. The signals also include the seating comfort score and activity index as described above.

[0062] As described above, the displacement of body point 21, as well as its absolute and relative positions, are identified and classified via a neural network. This classification yields a likelihood score for the seating comfort of occupant 10, also known as the posture comfort score. Due to the displacement of body point 21, characteristic movements indicating discomfort are also considered when estimating the posture comfort score.

[0063] Furthermore, facial expressions are also taken into account. That is, if facial expression 41 suggests discomfort or negative emotions, the sitting comfort score is reduced. Conversely, for neutral or positive emotions derived from facial expression 41, the sitting comfort score is increased.

[0064] If a time-of-flight camera is used as sensing device 13, three-dimensional information is provided, and the angle between the upper body and legs of occupant 10 can be accurately measured. If no three-dimensional information is available, such as for a typical RGB camera, similar information can be approximated from the two-dimensional images of image sequence 31 (see...). Figure 3 Additionally, the sitting posture analysis 43 includes determining the symmetry of the occupant 10's body posture.

[0065] Recommended engine 45 (see Figure 3 The system receives the results of posture analysis 43, namely, the signals regarding the time the occupant 10 has been in the current posture, the posture comfort score, and the activity index. These signals are processed by the suggestion engine 45 and can trigger events that provide the occupant 10 with corresponding user suggestions 47.

[0066] Such events are triggered if the seating comfort score falls below a configurable threshold and / or if the time spent in the current posture exceeds a similar configurable threshold for occupant 10. In these cases, the suggestion engine 45 triggers a "suggested posture change" command, which is presented to occupant 10 via a human-machine interface, such as on a display or screen. Furthermore, low activity indices and a lack of body posture symmetry can trigger similar events and commands. Whether user suggestions 47 are actually presented to occupant 10 may further depend on additional inputs, such as the duration of the ride, road conditions, navigation information, etc. When the driving task does not require full attention (e.g., in traffic jams or during autonomous driving), further suggestions might be "change your position in the seat," "engage in some specific activity / exercise," or simply "take a break and get out of the car for a while," for example, if a park or restaurant is identified nearby.

[0067] Additionally, if posture analysis 43 identifies some discomfort that may cause back pain (obtainable from activity classification 39 and facial expression 41), engine suggestion 45 may recommend activating the massage seat function. Furthermore, if steering wheel 14 (see...) Figure 1 If the relative positions of the engine 45 with shoulder point 25, arm point 27 and hand point 29 are determined to be unfavorable, it is recommended that the steering wheel position or seat configuration be adjusted.

[0068] The results of the sitting posture analysis 43, namely the time spent in the same posture, the sitting comfort score, and the activity index, are recorded and reported in the trip summary 49. This can be combined with the user profile 53 and further accessed via smartphone applications, websites, etc.

[0069] The input unit 19 of system 11 is capable of receiving user feedback 51 in response to user suggestion 47. Specifically, occupant 10 can be registered and identified by system 11, thus system 11 can store some personal preferences in a user profile 53, which is also part of the input unit 19. Therefore, user feedback 51 is used as an update 55 to the user profile 53. Direct user feedback 51 in response to user suggestion 47 includes accepting or rejecting the suggestion. If the suggestion is accepted, similar suggestions will be provided in similar situations in the future. Conversely, if the current suggestion is rejected, this type of suggestion will be suppressed in the future. Additionally, the user or occupant 10 can typically suppress certain suggestions via system settings through the user profile 53. Furthermore, statistical data 57 provided by posture analysis 43 can also be used to update the user profile 53. That is, system 11 automatically partially customizes for a specific occupant 10 via posture analysis 43 and statistical data 57, without requiring any action from the occupant 10.

[0070] As described above, user suggestion 47 is displayed as a visual response to an event triggered by suggestion engine 45. This visual response is displayed on the vehicle's infotainment system screen, for example, as an icon or dialog box. Alternatively, the visual response of system 11 can be displayed on a head-up display. Additionally, short animations or video clips can be displayed to encourage occupant 10 to change seating position, perform stretching exercises, or take a proactive rest. Furthermore, audio guidance, such as that provided by a virtual assistant, can be used. System 11 can additionally use input image sequence 31 to monitor the suggested activities of occupant 10.

[0071] List of reference numerals

[0072] 10 crew members

[0073] 11 System

[0074] 12 seats

[0075] 13. Sensing devices, cameras

[0076] 14. Steering wheel

[0077] 15 processing units

[0078] 17 Output Unit

[0079] 19 Input Units

[0080] 21 Body Points

[0081] 23 Facial Points

[0082] 25 shoulder points

[0083] 26 Back area

[0084] 27 arm points

[0085] 28. Shoulder area

[0086] 29 Hand Points

[0087] 31 Input image sequence

[0088] 33 Body Point Detection

[0089] 35 facial point detection

[0090] 37. Classification of Static Sitting Postures

[0091] 39 Action Classification

[0092] 41 Facial Expressions

[0093] 43. Sitting Posture Analysis

[0094] 45 Recommended Engines

[0095] 47 User Suggestions

[0096] 49. Trip Summary

[0097] 51 User Feedback

[0098] 53 User Profiles

[0099] Update 55

[0100] 57 Statistical Data

Claims

1. A computer-implemented method for monitoring occupants (10) located inside a vehicle, the method comprising the steps of: At least one image of the occupant (10) is detected via sensing device (13). The body points (21) of the occupant (10) are identified based on the at least one image via the processing unit (15). The occupant's (10) posture is classified based on detected body points (21) via the processing unit (15), wherein the processing unit (15) includes a preprocessing algorithm, a neural network, and a post-processing algorithm, and The output unit (17) provides suggestions (47) for adjusting the occupant's posture based on the classified posture, wherein the output unit (17) provides suggestions (47) as visual responses, the visual responses including icons or dialog boxes displayed on the head-up display or the screen of the vehicle infotainment system. The system obtains the occupant's (10's) response (51) to the suggestion (47) through the input unit (19). The response (51) includes accepting or rejecting the current suggestion (47), and determines the number and / or type of future suggestions (47) based on the occupant's response (51). If the occupant (10) accepts the current suggestion (47), the output unit (17) proposes a guidance activity for the occupant (10), the guidance activity depending on the driving conditions and environment of the vehicle.

2. The method according to claim 1, wherein, The image sequence (31) is detected via the sensing device (13). Via the processing unit (15): The time series of the body point (21) is determined based on the image sequence. By using the time series, displacement vectors are estimated for each body point in the body point (21). The movement of the occupant (10) is classified based on the displacement vector, and The output unit (17) further provides suggestions (47) for adjusting the occupant's posture based on the occupant's classified movements.

3. The method according to claim 2, wherein, The body point (21) includes a face point (23), which is identified and tracked based on the image sequence (31), and The processing unit (15) obtains facial expressions based on the identified facial points (23) and classifies the facial expressions. The output unit (17) further provides suggestions (47) for adjusting the occupant's posture based on the classified facial expressions.

4. The method according to claim 3, wherein, The processing unit (15) determines a seating comfort score based on the occupant's categorized posture, categorized movement, and / or categorized facial expressions. The suggestion (47) to adjust the occupant's posture is based on the seating comfort score.

5. The method according to claim 4, wherein, The suggestion to adjust the occupant’s posture is provided only if the seating comfort score is less than a predetermined threshold (47).

6. The method according to any one of claims 1 to 5, wherein, The position of the occupant (10) is monitored during the predefined time period by identifying the body points (21) at the beginning and end of the predefined time period. If the occupant’s position does not change during the predefined time period, the suggestion to adjust the occupant’s posture is provided (47).

7. The method according to claim 1, wherein, The occupant (10) is located in the driver's seat (12), The body points (21) include shoulder points (25), arm points (27) and / or hand points (29). The relative positions of the shoulder point (25), the arm point (27), and / or the hand point (29) are determined relative to the steering wheel (14) of the vehicle, and Based on the relative position, further suggestions (47) for adjusting the steering wheel (14) are provided via the output unit (17).

8. The method according to claim 1, wherein, The occupant's (10) posture is further classified based on the occupant's (10) profile (53) stored in the processing unit (15).

9. A system (11) for monitoring occupants (10) located inside a vehicle, the system (11) comprising: A sensing device (13) configured to detect at least one image of the occupant (10), A processing unit (15) is configured to identify body points (21) of the occupant (10) based on the at least one image, and to classify the posture of the occupant (10) based on the detected body points (21), wherein the processing unit (15) includes a preprocessing algorithm, a neural network, and a postprocessing algorithm, and An output unit (17) is configured to provide suggestions (47) for adjusting the occupant's posture based on a classified posture, wherein the output unit (17) provides suggestions (47) as visual responses, the visual responses including icons or dialog boxes displayed on the screen of a head-up display or vehicle infotainment system. The system further includes an input unit (19) configured to acquire the occupant's (10's) response (51) to the suggestion (47), the response (51) including accepting or rejecting the current suggestion (47), and the number and / or type of future suggestions (47) depending on the occupant's response (51). If the occupant (10) accepts the current suggestion (47), the output unit (17) proposes a guidance activity for the occupant (10), the guidance activity depending on the driving conditions and environment of the vehicle.

10. The system according to claim 9, wherein, The processing unit (15) includes a machine learning algorithm or rule-based logic for classifying the occupant's posture.

11. A computer system configured to perform a computer-implemented method according to any one of claims 1 to 8.

12. A non-transitory computer-readable medium comprising instructions for performing a computer-implemented method according to any one of claims 1 to 8.

Images