Device and method for determining a physical limitation

Abstract

By means of an apparatus and a method for analyzing a physical limitation of a vehicle occupant, environmental information is captured in the vehicle interior and corresponding environmental data is generated. A physical limitation of a body part is then determined on the basis of the environmental data.

Description

[0001] Device and method for determining a physical limitation

[0002] The invention relates to a device and a method for determining a physical limitation of a vehicle occupant. Environmental data from the vehicle interior are recorded and processed.

[0003] The physical condition of a vehicle occupant encompasses a variety of characteristics, such as body temperature, weight, and height, which can be measured using various methods. Part of the physical condition of a vehicle occupant also relates to posture, or the position of individual body parts.

[0004] A measurement method suitable for capturing the position of individual body parts of a vehicle occupant is the human body pose estimation (HBE) method. The HBE method is also known simply as human pose estimation. In established HBE methods, 3D key points are extracted from a sequence of images in a video stream recorded by a camera. Using HBE, the actual movement of the vehicle occupant is analyzed, and individual limbs can be determined. In a typical whole-body motion analysis, established HBE methods usually use 17 or 18 key points to determine the position and movement of a body part (e.g., arm, hand, head, eye) of a vehicle occupant.Based on the position of at least one key point of the vehicle occupant, the corresponding body part or another body part can be tracked in further images captured by the camera.

[0005] Human body pose estimation and similar image-based methods are used in vehicles, among other things, to capture vehicle occupants and their behavior. This information can be made available to driver assistance systems and other assistance systems to support vehicle occupants.

[0006] Document WO 2023 / 000119 A1 discloses a method and a device for gesture recognition. Based on captured image data from successively recorded images, gesture action information from the fingertips is assigned to a specific function or interaction object. Document JP 2021-025393 A discloses a device for controlling vehicle doors that can automatically open and close a vehicle door without requiring any special operation by the user. A control unit determines whether predetermined conditions for a user's intention to close the door are present and determines whether the door should be closed or not. If the conditions for detecting the intention to close the door are met, the control unit controls the door drive unit to close the door.

[0007] Document JP 2016-149105 A discloses a system for detecting the condition of a person in a vehicle. This allows, for example, the determination of whether the person is sober. Such an in-vehicle system comprises a detection unit for acquiring information relating to a person in the vehicle; a vehicle information input unit with at least one vehicle speed sensor; an information processing unit for calculating information acquired by the detection unit and the vehicle information input unit and for storing the information; and a control processing unit for outputting a control signal to each control information unit based on the calculation result of the information processing unit.

[0008] Existing systems for detecting vehicle occupants and their behavior are not designed to detect a physical disability of a vehicle occupant. Furthermore, these systems are not designed to accurately detect behavior resulting from such a physical disability. It is therefore desirable to correctly detect the physical condition of a person with a physical disability and to provide them with appropriate support while traveling in a vehicle.

[0009] The object of the invention is to provide a device and a method for determining a physical limitation, by which a physical limitation of the vehicle occupant can be easily determined.

[0010] This problem is solved by a device having the features of claim 1 and by a method having the features of the independent method claim.

[0011] Beneficial further training opportunities are listed in the dependent requirements.

[0012] A vehicle occupant may be in a physically restricted condition that makes it difficult or impossible for them to fully use a function of the vehicle. For example, a vehicle occupant may have a plaster cast on one arm. Due to the immobilization, the occupant's ability to move this arm is severely limited. The device according to claim 1 enables the detection of a physical restriction of the arm based on environmental data from the vehicle interior. Restrictions may also exist for other body parts, such as the legs, hands, neck, torso, and / or head of the vehicle occupant, in which case the restriction of the respective body part is determined. The device with the features of claim 1 enables the detection of the physical restriction of the vehicle occupant.Based on this finding, the vehicle occupant can be specifically supported during interaction with the vehicle.

[0013] A vehicle occupant can be a driver, a front passenger, or a passenger in one of the back seats.

[0014] A physical limitation can be caused by a variety of impairments. These include, for example, broken bones, muscle tension, obesity, pregnancies, advanced age, missing body parts, muscle or nerve diseases, or external impairments such as medical corsets, restrictive clothing, plaster casts, or objects in the vehicle interior.

[0015] The physical limitations of a body part can be determined using various environmental data. For example, artificial intelligence can be used to process image data, identify and interpret image content, and thus determine the physical limitations. Furthermore, image data can be processed to determine likely unachievable body positions based on previous body movements. The physical limitations can also be determined based on these unachievable body positions. Alternatively or additionally, weight data relating to a vehicle occupant and image data relating to their height can be used to determine whether the vehicle occupant is obese, thus also enabling the identification of such a physical limitation.

[0016] A body movement can be a movement of the body as a whole, or of a body segment, such as the upper or lower body, or of individual body parts located between two joints or at the ends of a joint, or of individual body parts, such as an arm, hand, neck, or head. It is advantageous if the processing unit is configured to use environmental data to determine information about at least one body movement of the vehicle occupant. Furthermore, the processing unit is configured to determine the degree of freedom of movement of at least one body part of the vehicle occupant that moves during the body movement. Finally, the processing unit is configured to compare the determined degree of freedom of movement of the body part of the vehicle occupant with a preset, stored limit value for the degree of freedom of movement for that body part.If the threshold value is not met, the processing unit can detect a physical restriction of the body part. This allows for a particularly reliable detection of physical restrictions, as a limited degree of freedom of movement directly impacts the vehicle occupant as a physical limitation.

[0017] A degree of freedom describes the range of possible positions of a body part in relation to another body part to which it is connected. The degree of freedom can be determined, for example, by assigning a reference point to a connection point between the body part and the other body part (e.g., a joint) and a movement point to another point on the body part (which could be at its end or at the nearest joint). The spatial arrangement of the reference point and the movement point relative to each other is then measured, from which the degree of freedom can be calculated. The reference and movement points can also be assigned differently if needed.

[0018] Degrees of freedom can also be determined using other methods, such as the human body pose estimation method or with the aid of a neural network or artificial intelligence.

[0019] The limit for the degrees of freedom of movement of a body part can, for example, be based on the population-average degrees of freedom of movement of the same body part. Furthermore, the limit can include an average value for the speed of movement of a body part and / or an average value for the acceleration during movement of a body part. The limit can also take other factors into account, such as sex or age, particularly as an offset.

[0020] It is advantageous if the processing unit is equipped to determine the degrees of freedom of movement of the body part being moved during body movement using a body pose estimation method. This allows body movements to be determined with particularly high precision, enabling more accurate determination of the degrees of freedom for the moving body part. It is also advantageous if the degrees of freedom of movement of the body part being moved during body movement are determined using an artificial intelligence-supported method. This makes the types of body movements that can be detected particularly flexible, allowing for very precise identification of a body part with a physical limitation.

[0021] Furthermore, it is advantageous if the processing unit is equipped to use environmental data to determine information about the facial expression of a vehicle occupant. The processing unit can classify a facial expression contorted in pain as a pain-associated expression. This classification can be achieved, for example, by matching the facial contours of the vehicle occupant with a database containing various contour patterns of a pain-contorted face. Alternatively, the classification can be performed using an artificial intelligence-supported method trained to recognize a pain-contorted face. The processing unit is also equipped to correlate the information about the pain-associated facial expression with the corresponding body movement.Based on the body movement associated with a pain-related facial expression, the processing unit can detect a pain-related limitation in the vehicle occupant. A pain-related limitation in the vehicle occupant means that the occupant's behavior is restricted by pain. By detecting a pain-related limitation in the vehicle occupant, the occupant's behavior can be accurately recorded, allowing for targeted support during interaction with the vehicle.

[0022] It is further advantageous if the processing unit is designed to determine, based on information about the vehicle occupant's body movement or a further body movement, in particular of another body part, an intention of the vehicle occupant to operate a vehicle control element. An intention of the vehicle occupant corresponds to a purpose or plan of the vehicle occupant, such as opening a vehicle door, adjusting the ventilation, or opening the glove compartment. A vehicle control element is an element of the vehicle that can be operated, e.g., a switch, a slider, a rotary switch, a lever, a button, a push button, or a touchscreen. The processing unit is further designed to determine, based on the detected physical limitation and the position of the control element, whether the vehicle occupant can operate the control element.It is also advantageous if the processing unit is designed to determine, based on the identified pain-related limitation and the position of the control element, whether the vehicle occupant can operate the control element without pain. Furthermore, it is advantageous if the device is designed to activate the operating function associated with the control element only if the vehicle occupant cannot operate the control element or cannot operate it without pain. This allows the vehicle occupant to use the vehicle's functions without restriction. Additionally, this creates a greater sense of control for the vehicle occupant, as they are not prevented from activating a function if they can do so without pain.

[0023] It is advantageous if the processing unit is designed to determine, based on environmental data and preset stored restrictions, whether activating the operating function assigned to the control is permissible. For example, the processing unit can determine, based on environmental data, whether there are obstacles in the vehicle that might make activating an operating function impractical. The processing unit can, for instance, determine that moving the front passenger seat towards the rear seat is not permissible if a person is sitting in the rear seat behind the front passenger seat whose legs would be trapped. Preset stored restrictions are defined rules under which certain behavior is not permitted. This includes, for example, a seatbelt requirement where unfastening the seatbelt while driving is not allowed or even dangerous.It is advantageous if the processing unit is configured to only activate the operating function assigned to the control element if activation is permitted. This increases safety during vehicle operation.

[0024] Furthermore, it is advantageous if the device includes an output unit and / or an input unit. The output unit can be, for example, a loudspeaker or a display, and the input unit can be, for example, a microphone or a touchscreen. It is advantageous if the processing unit is configured to offer the vehicle occupant the option of activating the detected operating function via the output unit before activating the operating function associated with the control element. It is also advantageous if the processing unit is configured to detect confirmation of the activation of the operating function via the input unit and only activate the operating function associated with the control element after such confirmation has been received. This ensures that the detected intention corresponds to the actual intention of the vehicle occupant.

[0025] It is advantageous if the processing unit is designed to determine the vehicle occupant's gaze direction based on environmental data. It is further advantageous if the processing unit is designed to determine the vehicle occupant's intention to operate the vehicle's controls based on information about body movement and gaze direction. This allows the vehicle occupant's intention to be determined correctly with a particularly high probability.

[0026] Furthermore, it is advantageous if the processing unit is designed to shift a display unit and / or adjust display elements shown on a display unit based on the detected physical limitation and / or the detected pain-related limitation. This reduces any restrictions on the vehicle occupant's interaction with the vehicle. A display unit is, for example, a unit for the visual display of information in the vehicle. A display element is, for example, an information element on a display surface, such as a video stream of the vehicle's surroundings.

[0027] It is advantageous if the processing unit is configured to store the detected physical limitation and / or the detected pain-related limitation as vehicle occupant-related information. It is further advantageous if the vehicle occupant-related information is stored in a central database and / or a mobile storage unit. A central database could be, for example, a database of the vehicle manufacturer or another central database. A mobile storage unit could be, for example, a smartphone, a smartwatch, a vehicle key with memory function, or another mobile device with storage. It is also advantageous if the processing unit is configured to read the vehicle occupant-related information, preferably from a central database and / or a mobile storage unit.Furthermore, it is advantageous if the processing unit is designed to determine the detected physical limitation and / or pain-related limitation of the vehicle occupant based on the stored and / or read-out vehicle occupant information. This allows a physical limitation and / or pain-related limitation to be detected immediately as soon as the vehicle occupant is in the vehicle. This enables faster, more targeted support for the vehicle occupant when interacting with the vehicle. Additionally, the vehicle occupant can also be assisted more quickly when interacting with another person.

[0028] The vehicle is specifically supported by a similar device.

[0029] It is advantageous if the environmental sensing unit includes at least one interior camera. The interior camera is designed to capture at least one image of at least a portion of the vehicle interior as environmental information and to generate corresponding image data as environmental data. The camera can be integrated into the vehicle's interior rearview mirror or its housing. The camera can also be positioned above the driver and / or front passenger in the area of ​​the vehicle's headliner. Furthermore, the camera can also be positioned on the B-pillar and / or A-pillar on the driver's side and / or front passenger's side and / or on a central display unit of the vehicle. By arranging cameras in multiple positions within the vehicle, the detectable area in the vehicle interior can be extended compared to capturing the vehicle interior with only one camera. It is also advantageous if the interior camera, for example,The camera used is an infrared camera and / or an RGB camera and / or a time-of-flight (TOF) camera. This allows for high-quality environmental data capture, making it easy to determine information about body movement and the degree of freedom. Combining multiple camera types further improves the quality of the environmental data. It is also advantageous if the camera's field of view has a detection angle of 100° to 150°, preferably being a monocular camera and / or an RGB-IR camera. This allows for image processing by the processing unit based on the captured IR images. The use of IR images is advantageous because these images are independent of ambient light. Alternatively, the camera can also be a pure IR camera and / or a stereo camera (3D camera).The camera's image data can be processed in a computer vision system provided by the processing unit. Using an RGB-IR camera combines the ability to operate independently of ambient light with the ability to provide RGB information when visible light is available.

[0030] It is further advantageous if the processing unit is configured to issue a notification to the output unit based on the detected physical limitation and / or the detected pain-related limitation. The notification can relate to a setting of the control element and / or another control element that can be changed. Preferably, the control element is a steering wheel or a vehicle seat. The notification can, for example, be issued specifically for the detected physical limitation and / or the detected pain-related limitation. Thus, the vehicle occupant can receive a notification that they can adjust the position of their seat if they have a physical limitation of their legs, or they can receive a notification that they can adjust the position of the steering wheel if they have a pain-related limitation of their arms.This improves the well-being of the vehicle occupant and helps them to fully utilize all the vehicle's functions.

[0031] The method with the features of the dependent method claim has the same advantages as the claimed device. In particular, the method can be further developed with the features of the dependent claims directed to the device and the aforementioned embodiments.

[0032] Examples of implementation are explained in more detail below with reference to the figures. These show:

[0033] Figure 1 is a perspective schematic representation of a vehicle cockpit;

[0034] Figure 2 shows a first image taken by the camera of a first view of the vehicle interior with a vehicle occupant with a physical limitation in the vehicle;

[0035] Figure 3 shows a second image taken by the camera of a second vehicle interior view with a vehicle occupant experiencing pain-related limitations in the vehicle;

[0036] Figure 4 shows a third image, taken by another camera, of a third vehicle interior view with a vehicle occupant with a physical disability inside the vehicle; and

[0037] Figure Seinen's procedure for determining the physical limitations of a vehicle occupant.

[0038] The embodiments of all figures can be combined with each other in any way and are not mutually exclusive.

[0039] Figure 1 shows a perspective schematic representation of the cockpit 100 of a vehicle 102. The cockpit 100 comprises a central information display 108 (CID), a head-up display 110, and a graphic instrument cluster 112, which are arranged in a dashboard 114 of the vehicle 102. The aforementioned display elements each form a functional unit of an output unit of the vehicle 102, which is configured to output information. At least one loudspeaker 116 and one microphone 117 of the vehicle 102 are also arranged in the cockpit 100. The loudspeaker 116 forms a functional unit of the output unit, and the microphone 117 forms a functional unit of an input unit. The functional units of the output unit and the input unit also serve as playback units for audio and / or video playback.

[0040] The cockpit 100 also includes a steering wheel 118 with controls 120, a gear selector 122, pedals 124, and an input unit 126 with a rotary dial and push-button function and / or a touch input panel. This input unit 126 is also referred to as the Ergo Commander.

[0041] In the upper area of ​​a windscreen 128 of the vehicle 102, an interior rearview mirror 132 and a camera 134 mounted in the area of ​​this interior rearview mirror 132 are arranged.

[0042] The camera 134 can, for example, be integrated into the interior rearview mirror 132 of the vehicle 102 or into the housing of the interior rearview mirror 132. Additionally or alternatively, the camera 134 can also be located above the driver's seat in the vehicle's headliner and / or on the B-pillar and / or A-pillar on the driver's side and / or the passenger's side and / or on a central display unit 108 of the vehicle. Furthermore, the field of view of the camera 134 can have a detection angle in the range of 100° to 150°. The camera 134 is designed to capture images depicting at least part of the vehicle interior as environmental information. The camera 134 is further designed to capture several successive images, particularly in the form of a video stream, to generate corresponding image data as environmental data, and to transmit this data to a control unit 138 serving as a processing unit.The camera 134 can be a monocular or stereo camera, preferably an infrared camera and / or an RGB camera and / or a time-of-flight (TOF) camera. A first image 200 of a first vehicle interior view captured by the camera 134 is shown in Figure 2. A second image 300 of a second vehicle interior view captured by the camera 134 is shown in Figure 3. The control unit 138 has data outputs 140 and data inputs 142, which serve to connect to other units of the vehicle 102, for example, to other cameras, sensors, input and output units, and control units of assistance systems.

[0043] Figure 2 shows the first image 200 of the first vehicle interior view taken by camera 134, showing a vehicle occupant 208 with a physical disability. Elements with the same structure or function have the same reference symbols.

[0044] A driver 204 is seated in a driver's seat 106 of a vehicle 102, and a passenger 208 is seated in a passenger seat 206. Rear seats 210 are located in the rear of the vehicle 102. A driver's door 212a and a passenger door 212b of the vehicle 102 are shown in the front of the vehicle 102, and vehicle doors 212c and 212d are shown to the side of the rear seats 210 in the rear of the vehicle 102. A door opener 214a is located on the driver's door 212a, and a door opener 214b is located on the passenger door 214b. A lighting unit 218 with a light switch 220 is located on the vehicle headliner 216. The center console 222 is located between the driver's seat 106 and the passenger seat 206.

[0045] The passenger 208 has a plaster cast 228 on her right arm 224 and part of her right hand 226. Her right arm 224 is also in a sling 230, which is around her neck 232 and stabilizes her torso 234. Her left arm 236 and left hand 238 rest on the center console 222. Her torso 234 and head 240 are turned towards the passenger door 212b, and her eyes 242 are directed towards the door handle 214b of the passenger door 212b. The right hand 244 and the left hand 246 of the driver 204 are positioned on the steering wheel 118, so that the right arm 248 and the left arm 250 of the driver 204 are located between the steering wheel 118 and the upper body 252 of the driver 204.The eyes 254 are facing in the direction of travel and the neck 256 is not turned, so that the head 258 of the driver 204 is facing forward in the direction of travel. The face 260 of the passenger 208 and the face 262 of the driver 204 are relaxed.

[0046] The control unit 138 is designed to process the image data from the camera 134 and, based on the image data, to determine a physical limitation of a body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258. The control unit 138 can, for example, use artificial intelligence to process and interpret the image data and, based on the plaster cast 228 and / or the sling 230, determine a physical limitation of the right arm 228.

[0047] Alternatively or additionally, the control unit 138 can use the image data to determine information about at least one body movement of a vehicle occupant 204, 208. The control unit 138 is further configured to determine, based on the image data, a degree of freedom of at least one body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258 of the vehicle occupant 204, 208 that moves during the body movement. The degree of freedom describes the range of possible positions for body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258 in relation to another body part to which it is connected. The degree of freedom can be determined, for example, by assigning a reference point to a connection point between body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258 and the other body part, e.g., a joint.The control unit 138 assigns a motion point to another point of body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258, which may be, for example, terminal or located at another nearest joint. The control unit 138 then determines the possible spatial arrangement of the reference point and the motion point relative to each other, from which it can calculate the degree of freedom of movement of body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258. The reference and motion points can also be assigned differently as needed. Degrees of freedom can also be determined using other methods, such as human body pose estimation, or with the aid of a neural network or artificial intelligence. For example, the control unit 138 can use a body pose estimation method to determine the degrees of freedom.

[0048] Furthermore, the control unit 138 is designed to compare the determined degree of freedom with a preset, stored limit value and to determine whether the determined degree of freedom reaches or falls below the limit value. If the determined degree of freedom reaches or falls below the preset, stored limit value, the control unit 138 can detect a physical restriction of body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258 of the vehicle occupant 204, 208.

[0049] The control unit 138 processes the image data of the first image 200 captured by the camera 134 and uses this data to determine information about the body movements of the passenger 208, for example, information about the movement of the passenger 208's right arm 224. Based on the image data, the control unit 138 also determines the degree of freedom of movement of the passenger 208's right arm 224. The control unit 138 compares the determined degree of freedom of movement of the right arm 224 with a preset, stored limit value. The determined degree of freedom of movement of the right arm 224 falls below the limit value, so the control unit 138 detects a physical restriction for the passenger 208's right arm 224.

[0050] Furthermore, the control unit 138 determines, based on the information about the movement of the right arm 224 of the passenger 208, an intention of the passenger 208 to activate a door opener 214b on the passenger door 212b of the vehicle 102.

[0051] In a further embodiment, the control unit 138 can determine information about a movement of the upper body 234 of the passenger 208 from the image data of the camera 134. Alternatively or additionally, based on information about the movement of the upper body 234 of the passenger 208 towards the passenger door 212b, the control unit 138 can determine the intention of the passenger 208 to activate the door opener 214b of the passenger door 212b of the vehicle 102. Furthermore, alternatively or additionally, based on the image data of the camera 134, the control unit 138 can determine information about the direction in which the passenger 208 is looking at the door opener 214b of the passenger door 212b.Based on the information about the movement of the right arm 224 and / or the information about the movement of the upper body 234 and / or the information about the direction of the passenger's gaze 208, the control unit 138 can then easily and reliably determine the intention of the passenger 208 to activate the door opener 214b on the passenger door 212b of the vehicle 102.

[0052] Subsequently, based on the detected physical limitation of the right arm 224 and the position of the door opener 214b on the passenger door 212b, the control unit 138 determines whether the passenger 208 can operate the door opener 214b. If the control unit 138 determines that the passenger 208 cannot operate the door opener 214b, the control unit 138 activates the door opening function for the passenger door 212b.

[0053] The control unit 138 can further determine, based on the image data from camera 134 and on preset stored restrictions, whether activating the door opening function for the passenger door 212b is permissible. The control unit 138 then only activates the door opening function for the passenger door 212b if its activation has been determined to be permissible. For example, based on the image data from camera 134, the control unit 138 can determine that activating the door opening function for the passenger door 212b is not permissible if an object in the vehicle interior is leaning against the passenger door 212b, as this object could fall out when the passenger door 212b is opened.In another embodiment, the control unit 138 can determine, based on the preset stored restriction, that the activation of the door opening function for the passenger door 212b is not permitted when the vehicle 102 is in motion, as this action violates traffic rules.

[0054] Furthermore, before activating the door opening function for the passenger door 212b, the control unit 138 can offer the passenger 208 the option to activate the function via the speaker 116 and / or a display 108, 110, 112. Subsequently, the control unit 138 can detect confirmation of the activation of the operating function via the input unit, e.g., the microphone 117 or a touch display, and only then activate the door opening function for the passenger door 212b.

[0055] The control unit 138 can further store the detected physical limitation of the right arm 224, for example, on a central database and / or a mobile storage device, such as a smartphone, smartwatch, or vehicle key with memory function, as information relating to the passenger 208. If the passenger 208 is in the vehicle 102 again at a later time, the control unit 138 can read out information relating to the passenger 208, such as the detected physical limitation of the right arm 224. The passenger 208 may at that later time be the driver 204, a passenger 208, or a passenger in the rear seats 210 of the vehicle 102. The information relating to the passenger 208 can be read out, for example, from the central database and / or the mobile storage device, such as the smartphone, smartwatch, or vehicle key with memory function.The control unit 138 can then determine the detected physical limitation of the right arm 224 based on the stored and / or read information relating to the passenger 208.

[0056] Figure 3 shows the second image 300 taken by camera 134 of the second vehicle interior view with a vehicle occupant 204 with a pain-related limitation.

[0057] Figure 3 differs from Figure 2 in that there is no passenger 208 in the vehicle. Furthermore, the driver 204 has now raised his right arm 248 towards the vehicle roof 216, with his right hand 244 pointing towards the light switch 220 of the lighting unit 218. The driver 204's eyes 254 are directed towards the light switch 220, and his face 262 is contorted in pain.

[0058] Based on the image data from camera 134, the control unit 138 determines information about the body movements of the driver 204, for example, information about a movement of the driver's right arm 248. Additionally, based on the image data from camera 134, the control unit 138 determines information about the driver's facial expression during the movement of the right arm 248. Here, the control unit 138 classifies the driver's pain-distorted face 262 as a pain-associated facial expression. The control unit 138 then assigns the movement of the right arm 248 to the pain-associated facial expression. Based on the movement of the right arm 248 assigned to the pain-associated facial expression, the control unit 138 can determine a pain-related restriction of the driver's right arm 248.

[0059] Furthermore, the control unit 138 determines, based on the information about the movement of the right arm 248 of the driver 204, an intention of the driver 204 to activate the light switch 220 of the lighting unit 218 of the vehicle 102.

[0060] In another embodiment, the control unit 138, based on the image data from the camera 134, alternatively or additionally determines information about a movement of the driver's right hand 244. Based on information about the movement of the driver's right hand 244 towards the light switch 220, the control unit 138 can also determine the driver's intention to activate the light switch 220 of the vehicle's lighting unit 218 102. Furthermore, based on the image data from the camera 134, the control unit 138 can alternatively or additionally determine information about the driver's 204's viewing direction towards the light switch 220.The control unit 138 can then determine the intention of the driver 204 to operate the light switch 220 of the lighting unit 218 of the vehicle 102, based on the information about the movement of the right arm 248 and / or the information about the movement of the right hand 244 and / or based on the information about the direction of the driver's gaze 204.

[0061] Subsequently, the control unit 138 determines, based on the detected pain-related restriction of the driver 204 and the position of the light switch 220 on the vehicle's headliner 216, whether the driver 204 can operate the light switch 220 without pain. If the control unit 138 determines that the driver 204 cannot operate the light switch 220 without pain, the control unit 138 activates a switching function for the lighting unit 218.

[0062] The control unit 138 can further store the detected pain-related limitation of the driver 204, for example, on the central database and / or the mobile storage device, such as a smartphone, smartwatch, or vehicle key with memory function, as information related to the driver 204. If the driver 204 is in the vehicle 102 again at a later time, the control unit 138 can read out information related to the driver 204, such as the detected pain-related limitation of the driver 204. The driver 204 may at that later time be in the vehicle 102 as the driver 204, a passenger 208, or a passenger in the rear seats 210. The information related to the driver 204 can be read out, for example, from the central database and / or the mobile storage device, such as a smartphone, smartwatch, or vehicle key with memory function.The control unit 138 can then easily and reliably determine the identified pain-related limitation of the driver 204 based on the stored and / or read information relating to the driver 204.

[0063] Figure 4 shows a third image 400 taken by another camera of a third vehicle interior view with a vehicle occupant 204 with a physical restriction in the vehicle 102.

[0064] Figure 4 differs from Figure 1 in that it depicts a driver 204 in the cockpit 100. The driver's right hand 244 and left hand 246 are positioned on the steering wheel 118, such that the driver's right arm 248 and left arm 250 are located between the steering wheel 118 and the driver's upper body 252. The driver's eyes 254, neck 256, and head 258 are facing forward. Due to a condition affecting his muscles, also known as neck stiffness, the driver 204 can barely turn his neck 256. The driver's face 262 is contorted in pain.

[0065] In this third restriction situation, the control unit 138 detects a physical restriction for the neck 256 of the driver 204 according to a scheme described in relation to Figure 2, e.g. after a determined degree of freedom of movement of the neck falls below a limit value.

[0066] Alternatively or additionally, the control unit 138 detects a pain-related restriction of the driver 204 according to a scheme described in relation to Figure 3, after a movement of the neck 256 has been assigned to a pain-associated facial expression. For this purpose, the control unit can additionally process image data from the camera 134.

[0067] Based on the determined physical limitation of the driver's neck 256 and / or the determined pain-related limitation of the driver's neck 204, the control unit 138 adjusts the display elements shown on a display unit 108, 110, 112. For example, content from the central information display 108 is shown on the head-up display 110, so that the driver 204 does not have to turn their neck 256 to view the content on the central information display 108. This content includes, in particular, a blind spot assistant and a representation of the rear view from the driver's perspective, which is otherwise displayed in the vehicle's side mirrors 102. Alternatively, the content of the central information display 108 can also be shown on the graphic instrument cluster 112, or on both the graphic instrument cluster 112 and the head-up display 110.In another embodiment, the central information display 108 shifts so that the driver 204 does not have to turn his neck 256 to perceive the contents on the central information display 108.

[0068] Figure 5 shows a procedure for determining a physical limitation of a vehicle occupant 204, 208.

[0069] The process starts in step S100. Subsequently, in step S102, environmental data is determined within the vehicle interior of vehicle 102. This data can be captured and generated, for example, by an interior camera 134, which can be an infrared camera, an RGB camera, and / or a TOF camera. Then, in step S104, the control unit 138 uses this environmental data to detect a physical restriction of a body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258. The process then ends in step S106. Reference numerals:

[0070] 100 Cockpit

[0071] 102 vehicles

[0072] 106 Driver's seat

[0073] 108 Central Information Display

[0074] 110 Head Up Display

[0075] 112 Graphic instrument cluster

[0076] 114 Dashboard

[0077] 116 speakers

[0078] 117 Microphone

[0079] 118 Steering wheel

[0080] 120 Control element

[0081] 122 Gear selector

[0082] 124 Pedals

[0083] 126 Input unit

[0084] 128 Windscreen

[0085] 132 Interior rearview mirror

[0086] 134 Camera

[0087] 138 Control unit

[0088] 140 data outputs

[0089] 142 data inputs

[0090] 200 images

[0091] 204 drivers

[0092] 206 Passenger seat

[0093] 208 female passenger

[0094] 210 rear seats

[0095] 212 Vehicle door

[0096] 214 door openers

[0097] 216 Vehicle headliner

[0098] 218 Lighting unit

[0099] 220 light switches

[0100] 222 Center console 224 Right arm

[0101] 226 Right hand

[0102] 228 Plaster cast

[0103] 230 Carrying sling

[0104] 232 Neck

[0105] 234 Upper body

[0106] 236 Left arm

[0107] 238 Left hand

[0108] 240 heads

[0109] 242 eyes

[0110] 244 Right hand

[0111] 246 Left hand

[0112] 248 Right arm

[0113] 250 Left arm

[0114] 252 Upper body

[0115] 254 eyes

[0116] 256 neck

[0117] 258 head

[0118] 260, 262 Face

[0119] 300, 400 image

[0120] S100 to S106 Procedure steps

Claims

Claims 1. Device for determining a physical restriction of a vehicle occupant (204, 208), comprising an environment detection unit configured to capture environmental information in the vehicle interior and to generate corresponding environmental data, and a processing unit (138) configured to process the environmental data and, based on the environmental data, to determine a physical restriction of a body part (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258).

2. Device according to claim 1, wherein the processing unit (138) is further configured to use environmental data to determine information on at least one body movement of the vehicle occupant (204, 208) and a degree of freedom of at least one body part (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) of the vehicle occupant (204, 208) that moves during the body movement, and wherein the processing unit (138) is configured to process the determined degree of freedom of the body part (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) of the vehicle occupant (204, 208) to compare with a preset stored limit value for the degree of freedom of movement for this body part (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258), and to determine a physical restriction of the body part (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) when the limit value is undershot.

3. Device according to claim 2 wherein the processing unit (138) is configured to determine the degree of freedom of movement of the body part (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) moved during body movement using a body pose estimation method.

4. Device according to claim 2 or 3, wherein the processing unit (138) is further configured to determine information about a facial expression of the vehicle occupant (204, 208) using the environmental data and to classify this as a pain-associated facial expression in the case of a pain-distorted face (260, 262), and wherein the processing unit (138) is configured to assign the information about the pain-associated facial expression to the body movement, and wherein the processing unit (138) is configured to determine a pain-related restriction of the vehicle occupant (204, 208) based on the body movement assigned to the pain-associated facial expression.

5. Device according to one of claims 2 to 4, wherein the processing unit (138) is configured to determine, based on information about the body movement or a further body movement of the vehicle occupant (204, 208), an intention of the vehicle occupant (204, 208) to actuate a control element (120, 126, 214, 220) of the vehicle (102), and wherein the processing unit (138) is configured, based on the detected physical limitation and a position of the control element (120, 126, 214, 220), to determine whether the vehicle occupant can actuate the control element (120, 126, 214, 220), and / or wherein the processing unit (138) is configured, based on the detected pain-related limitation and the position of the control element (120, 126, 214, 220) to determine whether the vehicle occupant (204, 208) can operate the control element (120, 126, 214, 220) without pain, and wherein the device is designed,only if the vehicle occupant (204, 208) cannot operate the control element (120, 126, 214, 220) or the vehicle occupant (204, 208) cannot operate the control element (120, 126, 214, 220) without pain, to activate the operating function assigned to the control element (120, 126, 214, 220).

6. Device according to claim 5, wherein the processing unit (138) is configured based on the environmental data and based on preset to determine, based on stored restrictions, whether the activation of the operating function assigned to the control element (120, 126, 214, 220) is permissible, and only if the activation of the operating function assigned to the control element (120, 126, 214, 220) is permissible, to activate the operating function assigned to the control element (120, 126, 214, 220).

7. Device according to claim 5 or 6, wherein the device comprises an output unit (108, 110, 112, 116) and an input unit (112, 126), wherein the processing unit (138) is configured to offer the vehicle occupant (204, 208) the activation of the identified operating function via the output unit (108, 110, 112, 116) prior to the activation of the operating function associated with the control element (120, 126, 214, 220), and wherein the processing unit (138) is configured to detect confirmation of the activation of the operating function via the input unit (112, 126), and only then to activate the operating function associated with the control element (120, 126, 214, 220).

8. Device according to one of claims 5 to 7, wherein the processing unit (138) is configured to determine, based on the environmental data, information on the direction of view of the vehicle occupant (204, 208), and, based on the information on the body movement and the information on the direction of view, to determine the intention of the vehicle occupant (204, 208) to actuate the control element (120, 126, 214, 220) of the vehicle (102).

9. Device according to one of the preceding claims, wherein the processing unit (138) is configured to shift a display unit (108, 110, 112) and / or adjust display elements shown on a display unit (108, 110, 112) based on the detected physical limitation and / or based on the detected pain-related limitation.

10. Device according to one of the preceding claims, wherein the processing unit (138) is configured to store the detected physical limitation and / or the detected pain-related limitation as vehicle occupant-related information, and wherein the processing unit (138) is configured to store the vehicle occupant-related information preferably in a central database and / or a mobile storage unit, and wherein the processing unit (138) is configured to read the vehicle occupant-related information preferably from a central database and / or a mobile storage unit, and to determine the detected physical limitation and / or the detected pain-related limitation of the vehicle occupant (204, 208) based on the stored and / or read-out vehicle occupant-related information.

11. Device according to one of the preceding claims, wherein the environment detection unit comprises at least one interior camera (134), which is in particular an infrared camera and / or an RGB camera and / or a TOF camera, wherein the interior camera (134) captures at least one image of at least a part of the vehicle interior as environment information and generates image data corresponding to the image as environment data.

12. Device according to one of claims 5 to 11, wherein the processing unit (138) is configured to output a message to the output unit (108, 110, 112, 116) indicating, based on the detected physical limitation and / or the detected pain-related limitation, that a setting of the control element (120, 126, 214, 220) and / or a further control element (120, 126, 214, 220), in particular a steering wheel (118) or a vehicle seat (106, 206), can be changed.

13. Method for determining a physical limitation of a vehicle occupant (204, 208) in which environmental information is recorded in the vehicle interior and environmental data is generated, where, based on the environmental data, a physical limitation of a body part (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) is detected.

Images