Devices and methods for determining physical limitations
By collecting environmental data from the vehicle's interior space and using artificial intelligence and body posture estimation methods, the system determines the occupants' physical limitations and pain status, and adjusts vehicle functions to adapt to the occupants' abilities and needs. This solves the problem of existing technologies being unable to identify occupants' physical limitations, and achieves personalized occupant support and safety assurance.
Patent Information
- Application Number
- CN202580028458.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-07-29
- Filing Date
- 2025-05-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing vehicle occupant behavior collection systems are unable to effectively identify and address occupants' physical limitations, resulting in a lack of targeted support and safety guarantees.
By collecting environmental data from the vehicle's interior space and using artificial intelligence and body posture estimation methods, the system can determine the occupants' physical limitations and pain levels, and adjust vehicle functions to suit the occupants' abilities and needs.
Accurately identify occupants' physical limitations and pain conditions, provide personalized vehicle operation support, and improve occupant comfort and safety.
Smart Images

Figure CN123125627A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an apparatus and method for determining physical limitations of vehicle occupants. It involves collecting and processing environmental data from the interior space of a vehicle. Background Technology
[0002] The physical condition of vehicle occupants includes various characteristics such as body temperature, weight, or height, which can be collected using different measurement methods. Some characteristics of a vehicle occupant's physical condition also involve body posture, or the position of various body parts.
[0003] One measurement method suitable for capturing the positions of various body parts of a vehicle occupant is the human-body-pose-estimation method. This method is also known as human pose estimation. In known human pose estimation methods, 3D key points are extracted from an image sequence of a video image stream captured by a camera. These key points can then be used to analyze the actual movement of the vehicle occupant and identify individual limbs. In conventional whole-body motion analysis, known human pose estimation methods typically use 17 or 18 key points to determine the position and movement of body parts (such as arms, hands, head, and eyes) of the vehicle occupant. Based on the position of at least one key point of the vehicle occupant, the corresponding body part or other body parts can be tracked in additional images captured by the camera.
[0004] In vehicles, body pose estimation methods and similar image-based methods are particularly used to collect data on vehicle occupants and their behavior. This information can then be provided to driver assistance systems and other support systems to assist vehicle occupants.
[0005] Document WO2023 / 000119A1 discloses a method and apparatus for gesture recognition. It assigns fingertip gesture information to specific functions or interactive objects based on image data acquired from sequentially captured images.
[0006] Document JP2021-025393A discloses a device for controlling vehicle doors, which can automatically open and close the doors without requiring user intervention. In this device, a control unit determines whether predetermined conditions for recognizing a user's intention to close the door are met and determines whether the door should be closed. When the conditions for recognizing a door-closing intention are met, the control unit controls the door drive unit to close the door.
[0007] Document JP2016-149105A discloses a system for collecting the status of occupants in a vehicle. This allows, for example, the determination of whether an occupant is conscious. This in-vehicle system includes: a data acquisition unit for determining information relating to occupants in the vehicle; a vehicle information input unit having at least one vehicle speed sensor; an information processing unit for calculating and storing the information determined by the data acquisition unit and the vehicle information input unit; and a control processing unit for outputting control signals to each control information unit based on the calculation results of the information processing unit.
[0008] Known systems for collecting data on vehicle occupants and their behavior do not incorporate methods for collecting data on the physical limitations of vehicle occupants. Furthermore, these known systems do not incorporate methods for correspondingly collecting behaviors arising from physical limitations. Therefore, it is desirable to accurately collect the physical states of individuals with physical limitations and to provide support to these individuals while they are traveling in a vehicle. Summary of the Invention
[0009] The object of the present invention is to provide an apparatus and method for determining physical limitations of vehicle occupants, by which the physical limitations of vehicle occupants can be determined simply.
[0010] This task is solved by an apparatus having the features of claim 1 and a method having the features of the independent method claim. Advantageous extensions are given in the dependent claims.
[0011] Vehicle occupants may be physically restricted, making it difficult or impossible for them to use vehicle functions without restriction. For example, an occupant may have a plaster cast on their arm. Due to the immobilization provided by the cast, the occupant can only move their arm with very limited freedom of movement. With the device according to claim 1, the physical restriction on the arm can be determined based on environmental data of the vehicle's interior space. Restrictions may also exist on other body parts (such as the occupant's legs, hands, neck, torso, and / or head), in which case the restriction on that body part can be determined. This is achieved by the device having the features of claim 1: physical restrictions on the vehicle occupant can be determined. Based on this determination, targeted support can be provided to the vehicle occupant during interactions with the vehicle.
[0012] Vehicle occupants can be passengers in the driver's seat, front passenger seat, or rear seat.
[0013] Physical limitations can be caused by a variety of injuries. These include, for example, fractures, muscle tension, obesity, pregnancy, advanced age, loss of body parts, muscle or nerve disorders, or external injuries such as medical orthotic garments, restrictive clothing, plaster casts, or objects inside a vehicle.
[0014] Body limitations can be determined using various environmental data. For example, artificial intelligence can be used to process image data to identify and interpret its content, thereby determining body limitations. Furthermore, image data can be processed to determine body postures that are not expected to be achieved based on previous body movements. Body limitations can also be determined based on these unattainable postures. Alternatively or additionally, such body limitations can also be determined by analyzing weight data related to the vehicle occupant's weight and image data related to the vehicle occupant's height to determine whether the occupant is obese.
[0015] Body movement can be movement of the body as a whole, or movement of body segments (such as the upper or lower body), or movement of individual limbs located between two joints or at the ends of joints, or movement of individual body parts (such as arms, hands, neck, or head).
[0016] Advantageously, the processing unit is configured to determine information about at least one body movement of a vehicle occupant using environmental data. Furthermore, the processing unit is configured to determine the degrees of freedom of movement of at least one body part of the vehicle occupant that moves during the body movement. Additionally, the processing unit is configured to compare the determined degrees of freedom of movement of the vehicle occupant's body part with a preset, stored limit value for that body part's degrees of freedom. When the degree of freedom is below this limit value, the processing unit can determine a body restriction of the body part. This allows for particularly reliable determination of body restrictions because the restricted degrees of freedom specifically affect the vehicle occupant as body restrictions.
[0017] Degrees of freedom (DOF) describe the area of position a body part can occupy relative to another body part connected to it. DOF can be determined, for example, by assigning a reference point to the connection point (e.g., a joint) between the body part and the other body part, and a motion point to another point on the body part (which could be, for example, an end point or located at another nearest joint). The spatial arrangement between the reference point and the motion point is then measured to calculate the DOF. The reference point and motion point can also be assigned in other ways as needed. DOF can also be determined using other methods, such as body pose estimation methods, or with the aid of neural networks or artificial intelligence.
[0018] Limits for the degrees of freedom of movement of a body part can be based, for example, on the average degrees of freedom of movement of the same body part among a population. Furthermore, limits can include the average velocity of movement of the body part and / or the average acceleration of the body part during movement. Limits can also take into account other factors, such as gender or age, especially as offsets.
[0019] Advantageously, the processing unit is configured to determine the degrees of freedom of motion of body parts moving during body movement using body posture estimation methods. This allows for the determination of body movements with exceptionally high accuracy, thereby enabling more accurate identification of the degrees of freedom for moving body parts. Furthermore, it is advantageous to use artificial intelligence-assisted methods to determine the degrees of freedom of motion of body parts moving during body movement. This allows for a particularly flexible range of body movements that can be collected, enabling very precise identification of body parts with body constraints.
[0020] Furthermore, the processing unit is advantageously configured to determine information about the facial expressions of a vehicle occupant using environmental data. The processing unit can classify facial expressions as pain-related when the face is contorted in pain. Classification as pain-related can be performed, for example, by matching the vehicle occupant's facial contours with a database containing various contour patterns of faces contorted in pain. Classification can also be performed using an AI-assisted method trained to recognize faces contorted in pain. Additionally, the processing unit is configured to assign information about pain-related facial expressions to body movements. Therefore, the processing unit can determine pain-related limitations of the vehicle occupant based on the body movements assigned to pain-related facial expressions. Pain-related limitations of a vehicle occupant indicate that their behavior is restricted due to pain. By determining pain-related limitations, the vehicle occupant's behavior can be accurately captured, enabling targeted support for the vehicle occupant during interactions with the vehicle.
[0021] Furthermore, it is advantageous that the processing unit is configured to determine the vehicle occupant's intention to operate the vehicle's control elements based on information about the occupant's body movements or other body movements (especially other body parts). The occupant's intention corresponds to their plan or intention, such as opening a door, activating ventilation, or opening the glove box. The vehicle's control elements are manipulatory components of the vehicle, such as switches, sliders, rotary switches, joysticks, buttons, keys, or touch displays. The processing unit is also configured to determine whether the occupant can operate the control elements based on determined physical limitations and the location of the control elements. Similarly, it is advantageous that the processing unit is configured to determine whether the occupant can operate the control elements painlessly based on determined pain-related limitations and the location of the control elements. Furthermore, it is advantageous that the device is configured to activate the operating function assigned to the control element only when the occupant cannot operate the control element or cannot operate it painlessly. Thus, the occupant can use the vehicle's functions without restriction. Moreover, it creates a sense of autonomous use for the occupant because the control function is not pre-activated when the occupant can activate it painlessly.
[0022] Advantageously, the processing unit is configured to determine whether to allow activation of an operational function assigned to an operating element based on environmental data and preset stored restrictions. For example, the processing unit can determine, based on environmental data, the presence of an obstacle in the vehicle, under which activating a certain operational function might be unreasonable. The processing unit can determine, for example, that movement of the front passenger seat towards the rear passenger seat is not permitted when a person is sitting in the rear seat behind the front passenger seat, whose legs would be trapped. The preset stored restrictions are defined rules under which certain behaviors are not permitted. This includes, for example, the obligation to wear a seatbelt; that is, unfastening the seatbelt while driving is not permitted, or is also dangerous. Advantageously, the processing unit is also configured to activate an operational function assigned to an operating element only when activation of the assigned operational function is permitted. This improves safety during vehicle operation.
[0023] Furthermore, it is advantageous that the device includes an output unit and / or an input unit. The output unit may be, for example, a speaker or a display, and the input unit may be, for example, a microphone or a touch display. Advantageously, the processing unit is configured to propose activation of a determined operating function to the vehicle occupant via the output unit before activating the operating function assigned to the operating element. Furthermore, it is advantageous that the processing unit is configured to acquire confirmation of activation of the operating function via the input unit and activate the operating function assigned to the operating element only upon confirmation. This ensures that the determined intention matches the actual intention of the vehicle occupant. Figure 1 To.
[0024] Advantageously, the processing unit is configured to determine information about the vehicle occupant's line of sight direction based on environmental data. Furthermore, it is advantageous that the processing unit is configured to determine the vehicle occupant's intention to operate the vehicle's control elements based on information about body movement and information about the line of sight direction. This allows for the accurate determination of the vehicle occupant's intention with a particularly high probability.
[0025] Furthermore, it is advantageous that the processing unit is configured to shift and / or adjust the display elements shown on the display unit based on determined physical limitations and / or determined limitations caused by pain. This reduces the limitations on vehicle occupants when interacting with the vehicle. The display unit is, for example, a unit for visually reproducing information in the vehicle. Display elements are, for example, information elements on a display interface, such as a video stream of the vehicle environment.
[0026] Advantageously, the processing unit is configured to store the determined physical limitations and / or determined pain-related limitations as vehicle occupant-related information. Furthermore, it is advantageous to store the vehicle occupant-related information in a central database and / or a mobile storage unit. The central database is, for example, the vehicle manufacturer's database or other central database. The mobile storage unit is, for example, a smartphone, smartwatch, vehicle key with storage capabilities, or other mobile devices with memory. Also advantageously, the processing unit is configured to preferably read the vehicle occupant-related information from the central database and / or the mobile storage unit. Furthermore, it is advantageously that the processing unit is configured to determine the determined physical limitations and / or determined pain-related limitations of the vehicle occupant based on the stored and / or read vehicle occupant-related information. Thus, physical limitations and / or pain-related limitations can be determined immediately once the vehicle occupant is in the vehicle. Therefore, targeted support can be provided to the vehicle occupant more quickly when interacting with the vehicle. Furthermore, the vehicle occupant can also receive targeted support more quickly when interacting with other vehicles including similar devices.
[0027] Advantageously, the environmental acquisition unit includes at least one interior space camera. This interior space camera is configured to acquire at least one image of at least a portion of the vehicle's interior space as environmental information and generate corresponding image data as environmental data. The camera can be integrated into the vehicle's rearview mirror or into the mirror's housing. The camera can also be positioned in the area of the headliner above the driver and / or passenger seats. Furthermore, the camera can be positioned on the B-pillar and / or A-pillar on the driver's and / or passenger's sides and / or on the vehicle's central display unit. By arranging cameras in multiple locations within the vehicle, the area that can be acquired within the vehicle's interior space can be expanded compared to using only a single camera. It is also advantageous that the interior space camera is, for example, an infrared camera and / or an RGB camera and / or a time-of-flight camera (TOF camera). This allows for high-quality acquisition of environmental data, enabling the easy determination of information about body movement and degrees of freedom. The combination of multiple camera shooting types improves the quality of the environmental data. Another advantage is that the camera's field of view has an acquisition angle ranging from 100° to 150°, wherein the camera is preferably a monocular camera and / or an RGB-IR camera. This allows the processing unit to perform image processing based on the captured IR images. Using IR images is advantageous because these images are independent of ambient light. Alternatively, the camera can also be a pure IR camera and / or a stereo camera (3D camera). The processing of the camera's image data can be performed within a computer vision system provided by the processing unit. The use of an RGB-IR camera combines the possibility of being unaffected by ambient light with the possibility of providing RGB information simultaneously in the presence of visible light.
[0028] Furthermore, it is advantageous that the processing unit is configured to output a prompt via an output unit based on detected physical limitations and / or detected limitations caused by pain. This prompt may relate to the setting of an operating element and / or other operating elements, which can be changed. Preferably, the operating element is a steering wheel or vehicle seat. The prompt may, for example, be specifically output for detected physical limitations and / or detected limitations caused by pain. Thus, a vehicle occupant can receive a prompt regarding their adjustable seat position when they have leg limitations, or a prompt regarding their adjustable steering wheel position when they have arm limitations caused by pain. This improves the comfort of the vehicle occupant and supports their full use of all vehicle functions.
[0029] The method having the features of the parallel method claims has the same advantages as the claimed device. In particular, the method can be extended by means of the features of the dependent claims relating to the device and the aforementioned extensions. Attached Figure Description
[0030] The embodiments will now be described in detail with reference to the accompanying drawings. The drawings are as follows: Figure 1 A perspective view of the vehicle's cockpit is shown. Figure 2 A first image taken by a camera shows a view of the interior space of a first vehicle, in which there are vehicle occupants with physical limitations in the vehicle; Figure 3 A second image taken by a camera shows a view of the interior space of a second vehicle, in which there are vehicle occupants with limitations due to pain. Figure 4 A third image, taken by another camera, shows a view of the interior space of a third vehicle, in which a vehicle occupant with physical limitations is present; and Figure 5 A flowchart is shown for determining the physical limitations of vehicle occupants. Detailed Implementation
[0031] The embodiments in all the accompanying drawings can be combined with each other arbitrarily without mutual exclusion.
[0032] Figure 1 A perspective view of the cockpit 100 of vehicle 102 is shown.
[0033] The cockpit 100 includes a central information display 108 (CID), a head-up display 110, and a graphic instrument cluster 112, which are arranged in the instrument panel 114 of the vehicle 102. These display elements each constitute a functional unit of the output unit of the vehicle 102, which is configured to output information. At least one speaker 116 and microphone 117 of the vehicle 102 are also arranged in the cockpit 100. The speaker 116 constitutes a functional unit of the output unit, and the microphone 117 constitutes a functional unit of the input unit. The functional units of the output and input units also serve as playback units for audio and / or video playback.
[0034] The cockpit 100 also includes a steering wheel 118 with operating elements 120, a gear selector switch 122, a pedal assembly 124, and an input unit 126 with rotary wheel and button functions and / or touch input area. This input unit 126 is also referred to as the Ergo Commander.
[0035] An interior rearview mirror 132 and a camera 134 mounted in the area of the windshield 128 of vehicle 102 are arranged in the upper region of the windshield 128 of vehicle 102. The camera 134 may be integrated into the interior rearview mirror 132 of vehicle 102 or into the housing of the interior rearview mirror 132. Alternatively or additionally, the camera 134 may also be arranged in the headliner above the driver's seat and / or on the B-pillar and / or A-pillar on the driver's side and / or passenger side and / or on the central display unit 108 of the vehicle. Furthermore, the field of view of the camera 134 may have an acquisition angle in the range of 100° to 150°. The camera 134 is configured to acquire images as environmental information, the images having an image of an area including at least a portion of the vehicle's interior space. The camera 134 is also configured to acquire multiple temporally successive images, particularly in the form of a video stream, generate image data corresponding to these images as environmental data, and transmit these images to a control unit 138 serving as a processing unit. Camera 134 can be a monocular camera or a stereo camera, preferably an infrared camera and / or an RGB camera and / or a time-of-flight (TOF) camera. A first image 200 of the first interior space view of the vehicle, captured by camera 134, is shown below. Figure 2 The second image 300, taken by camera 134, shows a view of the interior space of the second vehicle. Figure 3 middle.
[0036] The control unit 138 has a data output terminal 140 and a data input terminal 142, which are used to connect to other units of the vehicle 102, such as other cameras, sensors, input and output units, and control units of auxiliary systems.
[0037] Figure 2 A first image 200, taken by camera 134, shows a view of the interior space of a first vehicle, in which a vehicle occupant 208 with physical limitations is present. Elements with the same structure or function have the same reference numerals.
[0038] The driver 204 sits in the driver's seat 106 of vehicle 102, and the passenger 208 sits in the passenger seat 206. Rear seats 210 are located in the rear of vehicle 102. The driver's side door 212a and the passenger side door 212b of vehicle 102 are shown in the front of vehicle 102, while doors 212c and 212d are shown beside the rear seats 210 in the rear of vehicle 102. Door openers 214a and 214b are located on the driver's side door 212a and the passenger side door 212b, respectively. A lighting unit 218 with a light switch 220 is provided on the headliner 216. A center console 222 is located between the driver's seat 106 and the passenger seat 206.
[0039] The passenger 208 has a plaster cast 228 on part of its right arm 224 and right hand 226. Furthermore, the right arm 224 is positioned within a suspension band 230, which is guided around the passenger 208's neck 232 and stabilizes the right arm 224 against the upper body 234 of the passenger 208. The passenger 208's left arm 236 and left hand 238 rest on the center console 222. The upper body 234 and head 240 of the passenger 208 are turned towards the passenger-side door 212b, and the passenger 208's eyes 242 are directed towards the door opener 214b of the passenger-side door 212b. The driver 204's right hand 244 and left hand 246 are positioned on the steering wheel 118, such that the driver 204's right arm 248 and left arm 250 are positioned between the steering wheel 118 and the driver 204's upper body 252. The eyes 254 are directed toward the direction of travel, and the neck 256 is not turned, thus the driver 204's head 258 is forward and facing the direction of travel. The face 260 of the passenger 208 and the face 262 of the driver 204 are relaxed.
[0040] Control unit 138 is configured to process image data from camera 134 and, based on the image data, determine body limitations of body parts 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, and 258. Control unit 138 may, for example, use artificial intelligence to process and interpret the image data and determine body limitations of the right arm 228 based on the plaster cast 228 and / or sling 230.
[0041] Alternatively or additionally, the control unit 138 may use image data to determine information regarding at least one body movement of vehicle occupants 204, 208. The control unit 138 is also configured to determine, based on the image data, the degrees of freedom of movement of at least one body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258 of the vehicle occupants 204, 208 in body movement. The degrees of freedom describe the positional area that body part 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258 can occupy relative to another body part connected to that body part. For example, degrees of freedom of movement can be determined by assigning reference points to the connection points (e.g., joints) between body parts 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258 and the other body part. Control unit 138 assigns a motion point to another point of body parts 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258 (which may be, for example, distal or located at another nearest joint). Subsequently, the control unit 138 determines the spatial arrangement that the reference point and the motion point can occupy relative to each other. Based on this, the control unit 138 can determine the degrees of freedom of motion for body parts 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, and 258. The reference point and the motion point can also be allocated in other ways as needed. The degrees of freedom of motion can also be determined using other methods, such as body posture estimation methods, or with the aid of neural networks or artificial intelligence. For example, the control unit 138 can use body posture estimation methods to determine the degrees of freedom of motion.
[0042] Furthermore, the control unit 138 is configured to compare the determined degree of freedom of motion with a preset stored limit value and determine whether the determined degree of freedom of motion reaches or falls below the limit value. If the determined degree of freedom of motion reaches or falls below the preset stored limit value, the control unit 138 can determine the body restrictions of the body parts 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, and 258 of the vehicle occupants 204 and 208.
[0043] Control unit 138 processes image data from the first image 200 captured by camera 134 and determines information about the body movements of the passenger 208 based on this image data, such as information about the movement of the passenger 208's right arm 224. Control unit 138 also determines the degrees of freedom of movement of the passenger 208's right arm 224 based on the image data. Control unit 138 compares the determined degrees of freedom of movement of the right arm 224 with preset stored limit values. If the determined degrees of freedom of movement of the right arm 224 are lower than the limit value, control unit 138 determines a body restriction for the passenger 208's right arm 224.
[0044] Furthermore, the control unit 138 determines the intention of the passenger 208 to operate the door opener 214b on the passenger side door 212b of the vehicle 102 based on information about the movement of the right arm 224 of the passenger 208.
[0045] In another embodiment, control unit 138 can determine information about the movement of the upper body 234 of the passenger 208 based on image data from camera 134. Alternatively or additionally, control unit 138 can now determine the passenger 208's intention to operate the door opener 214b of the passenger side door 212b of vehicle 102 based on information about the movement of the upper body 234 of the passenger 208 toward the passenger side door 212b. Alternatively or additionally, control unit 138 can also determine information about the passenger 208's line of sight toward the door opener 214b of the passenger side door 212b based on image data from camera 134. Then, control unit 138 can simply and reliably determine the passenger 208's intention to operate the door opener 214b of the passenger side door 212b of vehicle 102 based on information about the movement of the right arm 224 and / or the movement of the upper body 234 and / or the line of sight of the passenger 208.
[0046] Subsequently, the control unit 138 determines whether the passenger 208 can operate the door opener 214b based on the determined physical limitations of the right arm 224 and the position of the door opener 214b on the passenger-side door 212b. If the control unit 138 determines that the passenger 208 cannot operate the door opener 214b, the control unit 138 activates the door opening function for the passenger-side door 212b.
[0047] The control unit 138 can also determine whether to allow activation of the door opening function for the passenger-side door 212b based on image data from the camera 134 and preset stored restrictions. The control unit 138 activates the door opening function for the passenger-side door 212b only when it determines that activation is permitted. For example, the control unit 138 can determine, based on image data from the camera 134, that activation of the door opening function for the passenger-side door 212b is not permitted when an object is resting against the passenger-side door 212b in the vehicle interior, because the object might fall when the passenger-side door 212b is opened. In another embodiment, the control unit 138 can determine, based on preset stored restrictions, that activation of the door opening function for the passenger-side door 212b is not permitted when the vehicle 102 is in motion, because such an operation violates traffic rules.
[0048] Furthermore, the control unit 138 can propose to the passenger 208 to activate the door opening function for the passenger-side door 212b via the speaker 116 and / or displays 108, 110, 112 before activating the function. Subsequently, the control unit 138 can acquire confirmation of activation of the function via an input unit (e.g., microphone 117 or touch display) and only then activate the door opening function for the passenger-side door 212b.
[0049] The control unit 138 can also store the determined physical limitation of the right arm 224 as relevant information about the passenger 208, for example, in a central database and / or on a mobile storage unit, such as a smartphone, smartwatch, or vehicle key with storage capabilities. When the passenger 208 is later back in the vehicle 102, the control unit 138 can read the relevant information about the passenger 208 (such as the determined physical limitation of the right arm 224). Later, the passenger 208 may be in the vehicle 102 as a driver 204, a passenger 208, or a passenger in the rear seat 210. The relevant information about the passenger 208 can be read, for example, from a central database and / or mobile storage unit (such as a smartphone, smartwatch, or vehicle key with storage capabilities). The control unit 138 can then determine the determined physical limitation of the right arm 224 based on the stored and / or read relevant information about the passenger 208.
[0050] Figure 3 A second image 300, taken by camera 134, shows a view of the interior space of a second vehicle, in which a vehicle occupant 204 with a limitation due to pain is present.
[0051] Figure 3 and Figure 2The difference is that there is no passenger seat 208 in the vehicle. Furthermore, the driver 204 now raises his right arm 248 to the roof liner 216, with his right hand 244 pointing towards the light switch 220 of the lighting unit 218. The driver 204's eyes 254 are directed towards the light switch 220, and the driver 204's face 262 is contorted in pain.
[0052] The control unit 138 determines information about the driver 204's body movements, such as the movement of the driver 204's right arm 248, based on image data from the camera 134. Furthermore, the control unit 138 determines information about the driver 204's facial expression during the movement of the right arm 248, based on the image data from the camera 134. Here, the control unit 138 classifies the driver 204's pain-induced facial contortions 262 as a pain-related facial expression. Subsequently, the control unit 138 assigns the movement of the right arm 248 to this pain-related facial expression. Based on the movement of the right arm 248 assigned to the pain-related facial expression, the control unit 138 can determine whether the driver 204's right arm 248 is restricted due to pain.
[0053] Furthermore, the control unit 138 determines the driver 204's intention to operate the light switch 220 of the vehicle 102's lighting unit 218 based on information about the movement of the driver 204's right arm 248.
[0054] In another embodiment, the control unit 138 determines, alternatively or additionally, information regarding the movement of the driver 204's right hand 244 based on image data from the camera 134. The control unit 138 may also determine the driver 204's intention to operate the light switch 220 of the vehicle 102's lighting unit 218 based on information regarding the movement of the driver 204's right hand 244 toward the light switch 220. The control unit 138 may also, alternatively or additionally, determine information regarding the driver 204's gaze direction toward the light switch 220 based on image data from the camera 134. The control unit 138 can then determine the driver 204's intention to operate the light switch 220 of the vehicle 102's lighting unit 218 based on information regarding the movement of the right arm 248 and / or the movement of the right hand 244 and / or the direction of the driver 204's gaze.
[0055] Subsequently, the control unit 138 determines, based on the driver 204's assessment of pain-induced limitation and the position of the light switch 220 on the headliner 216, whether the driver 204 can operate the light switch 220 painlessly. If the control unit 138 determines that the driver 204 cannot operate the light switch 220 painlessly, the control unit 138 activates the switching function for the lighting unit 218.
[0056] The control unit 138 can also store the driver 204's determined pain-related limitations as driver 204-related information, for example, in a central database and / or a mobile storage unit (such as a smartphone, smartwatch, or vehicle key with storage capabilities). When the driver 204 is later back in the vehicle 102, the control unit 138 can read the driver 204-related information, such as the driver 204's determined pain-related limitations. Later, the driver 204 may be in the vehicle 102 as the driver 204, a front passenger 208, or a passenger in the rear seat 210. The driver 204-related information can be read, for example, from the central database and / or a mobile storage unit (such as a smartphone, smartwatch, or vehicle key with storage capabilities). The control unit 138 can then easily and reliably determine the driver 204's determined pain-related limitations based on the stored and / or read driver 204-related information.
[0057] Figure 4 A third image 400, taken by another camera, shows a view of the interior space of a third vehicle, in which a vehicle occupant 204 with physical limitations is present in vehicle 102.
[0058] Figure 4 and Figure 1 The difference lies in the fact that the driver 204 is shown in the cockpit 100. The driver 204's right hand 244 and left hand 246 are positioned on the steering wheel 118, such that the driver 204's right arm 248 and left arm 250 are positioned between the steering wheel 118 and the driver 204's upper body 252. The driver 204's eyes 254, neck 256, and head 258 are facing the direction of travel. The driver 204 can barely turn his neck 256 due to his muscle state (also known as neck stiffness). The driver 204's face 262 is contorted in pain.
[0059] In this third limiting condition, the control unit 138 refers to... Figure 2 The described scheme determines physical restrictions on the driver's neck 256, for example, after the determined degrees of freedom of neck movement are below the limit.
[0060] Alternatively or additionally, control unit 138, according to reference Figure 3 The described scheme determines the driver's (204's) pain-related limitation after assigning movement of the neck (256) to pain-related facial expressions. For this purpose, the control unit can additionally process image data from the camera (134).
[0061] Based on determined physical limitations of the driver 204's neck 256 and / or determined limitations due to pain, the control unit 138 adjusts the display elements shown on the display units 108, 110, 112. Therefore, the content of the central information display 108 is displayed on the head-up display 110, allowing the driver 204 to access the content of the central information display 108 without turning their neck 256. This content includes, in particular, blind spot assist systems and a display of the rearward view, which from the driver's perspective would normally be shown in the side mirrors of the vehicle 102. Alternatively, the content of the central information display 108 may also be displayed on the graphic instrument cluster 112 or on both the graphic instrument cluster 112 and the head-up display 110. In another embodiment, the central information display 108 is shifted so that the driver 204 can access the content of the central information display 108 without turning their neck 256.
[0062] Figure 5 A flowchart is shown for determining the physical limitations of vehicle occupants 204 and 208.
[0063] In step S100, the process begins. Then, in step S102, environmental data within the interior space of vehicle 102 is determined. This environmental data may be collected and generated, for example, by an interior space camera 134, which may be an infrared camera and / or an RGB camera and / or a TOF camera. Subsequently, in step S104, the control unit 138 determines body limitations for body parts 224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, and 258 based on the environmental data. Then, in step S106, the process ends.
[0064] List of reference numerals
[0065] 100 Cockpit
[0066] 102 vehicles
[0067] 106 Driver's Seat
[0068] 108 Central Information Display
[0069] 110 Head-up Display
[0070] 112 Graphic Combination Instrument
[0071] 114 Dashboard
[0072] 116 speakers
[0073] 117 microphones
[0074] 118 Steering Wheel
[0075] 120 Operating element
[0076] 122 Gear Selection Switch
[0077] 124 pedal set
[0078] 126 Input Units
[0079] 128 Front windshield
[0080] 132 Interior rearview mirror
[0081] 134 cameras
[0082] 138 Control Unit
[0083] 140 Data Output Terminal
[0084] 142 Data Input Terminal
[0085] 200 images
[0086] 204 Drivers
[0087] 206 Passenger Seat
[0088] 208 Passenger Seat
[0089] 210 Rear Seats
[0090] 212 Car Door
[0091] 214 Door Opener
[0092] 216 Roof Liner
[0093] 218 lighting units
[0094] 220 Light Switch
[0095] 222 Central Control Console
[0096] 224 Right arm
[0097] 226 Right hand
[0098] 228 Plaster bandage
[0099] 230 suspension belt
[0100] 232 Neck
[0101] 234 Upper body
[0102] 236 Left arm
[0103] 238 Left hand
[0104] 240 heads
[0105] 242 eyes
[0106] 244 Right hand
[0107] 246 Left hand
[0108] 248 Right arm
[0109] 250 Left arm
[0110] 252 Upper body
[0111] 254 eyes
[0112] 256 Neck
[0113] 258 heads
[0114] 260, 262 Facial
[0115] 300, 400 images
[0116] S100 to S106 Method Steps
Claims
1. A device for determining the physical limitations of vehicle occupants (204, 208), - It has an environmental acquisition unit, which is configured to collect environmental information in the vehicle's interior space and generate corresponding environmental data. - It has a processing unit (138) configured to process the environmental data and determine body limitations of body parts (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) based on the environmental data.
2. The apparatus according to claim 1, wherein, The processing unit (138) is further configured to determine, with the aid of the environmental data, information regarding at least one body movement of the vehicle occupants (204, 208) and the degrees of freedom of movement of at least one body part (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) of the vehicle occupants (204, 208) moving in the said body movement, and The processing unit (138) is configured to compare the determined degrees of freedom of motion of the body parts (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) of the vehicle occupants (204, 208) with preset stored limit values of the degrees of freedom of motion of the body parts (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258), and determine the body restriction of the body parts (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) when the value is lower than the limit value.
3. The apparatus according to claim 2, wherein, The processing unit (138) is configured to determine the degrees of freedom of motion of body parts (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258) moving in the body movement by means of a body posture estimation method.
4. The apparatus according to claim 2 or 3, wherein, The processing unit (138) also constructs information for determining facial expressions of vehicle occupants (204, 208) using the environmental data and classifying facial expressions (260, 262) as pain-related facial expressions when the faces are contorted in pain. The processing unit (138) is configured to assign information about the pain-related facial expressions to the body movements, and The processing unit (138) is configured to determine the pain-related limitations of the vehicle occupants (204, 208) based on the body movements assigned to the pain-related facial expressions.
5. The apparatus according to any one of claims 2 to 4, wherein, The processing unit (138) is configured to determine the intention of the vehicle occupants (204, 208) to operate the operating elements (120, 126, 214, 220) of the vehicle (102) based on information regarding the body movements or other body movements of the vehicle occupants (204, 208), and The processing unit (138) is configured to determine, based on the determined physical limitations and the position of the operating elements (120, 126, 214, 220), whether the vehicle occupant is capable of manipulating the operating elements (120, 126, 214, 220), and / or The processing unit (138) is configured to determine, based on the determined pain-induced limitation and the position of the operating elements (120, 126, 214, 220), whether the vehicle occupants (204, 208) can operate the operating elements (120, 126, 214, 220) painlessly; and The device is configured to activate the operating functions assigned to the operating elements (120, 126, 214, 220) only when the vehicle occupants (204, 208) cannot operate the operating elements (120, 126, 214, 220) or when the vehicle occupants (204, 208) cannot operate the operating elements (120, 126, 214, 220) painlessly.
6. The apparatus according to claim 5, wherein, The processing unit (138) is configured to determine whether to allow activation of the operation functions assigned to the operation elements (120, 126, 214, 220) based on the environmental data and based on preset storage restrictions, and to activate the operation functions assigned to the operation elements (120, 126, 214, 220) only when activation of the operation functions assigned to the operation elements (120, 126, 214, 220) is allowed.
7. The apparatus according to claim 5 or 6, wherein, The device includes output units (108, 110, 112, 116) and input units (112, 126). The processing unit (138) is configured to propose to the vehicle occupants (204, 208) via the output units (108, 110, 112, 116) to activate a determined operating function before activating the operating function assigned to the operating elements (120, 126, 214, 220). The processing unit (138) is configured to acquire confirmation of activation of the operation function through the input unit (112, 126) and only activate the operation function assigned to the operation element (120, 126, 214, 220) at this time.
8. The apparatus according to any one of claims 5 to 7, wherein, The processing unit (138) constructs information about the direction of the vehicle occupants' (204, 208) gaze based on the environmental data and determines the intention of the vehicle occupants (204, 208) to manipulate the operating elements (120, 126, 214, 220) of the vehicle (102) based on the information about the body movement and the direction of the gaze.
9. The apparatus according to any one of the preceding claims, wherein, The processing unit (138) is configured to shift and / or adjust the display elements shown on the display units (108, 110, 112) based on determined physical limitations and / or determined limitations caused by pain.
10. The apparatus according to any one of the preceding claims, wherein, The processing unit (138) is configured to store the determined physical limitations and / or the determined limitations caused by pain as vehicle occupant-related information, and The processing unit (138) is configured to preferably store the vehicle occupant-related information in a central database and / or a mobile storage unit, and The processing unit (138) is configured to preferably read the vehicle occupant-related information from a central database and / or a mobile storage unit, and Based on stored and / or retrieved vehicle occupant-related information, determine the determined physical limitations and / or determined pain-related limitations of the vehicle occupants (204, 208).
11. The apparatus according to any one of the preceding claims, wherein, The environmental acquisition unit includes at least one internal space camera (134), which is particularly an infrared camera and / or an RGB camera and / or a TOF camera. The interior space camera (134) acquires at least one image of at least a portion of the vehicle's interior space as environmental information and generates image data corresponding to the image as environmental data.
12. The apparatus according to any one of claims 5 to 11, wherein, The processing unit (138) is configured to output a prompt via the output unit (108, 110, 112, 116) based on detected physical limitations and / or detected limitations caused by pain, that is, the settings of the operating elements (120, 126, 214, 220) and / or other operating elements (120, 126, 214, 220), especially the steering wheel (118) or vehicle seat (106, 206), can be changed.
13. A method for determining physical limitations of vehicle occupants (204, 208), wherein, Collect environmental information from inside the vehicle and generate environmental data. Based on the environmental data, body limitations are determined for body parts (224, 226, 232, 234, 236, 238, 240, 244, 246, 248, 250, 252, 256, 258).
Citation Information
Patent Citations
In-vehicle human-sensitive system
JP2016149105A
Vehicle door control apparatus
JP2021025393A
Gesture recognition method and apparatus, system, and vehicle
WO2023000119A1
