Transparent operator obstacle detection for motor vehicle

Abstract

An obstacle detection method includes detecting an operator's proximity to a machine. The proximity is monitored to determine a set of gait parameters of the operator based on outputs of a set of gait sensors. The set of gait parameters is provided to a long short term memory (LSTM) recurrent neural network that determines a gait score by regressing the set of gait parameters. The gait score is compared to an obstacle threshold and the operator is engaged in response to the gait score exceeding the obstacle threshold.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to impairment detection for a vehicle operator, and more particularly to transparently identifying potential impairments of a vehicle operator. BACKGROUND

[0002] Operation of a motor vehicle can result in improper and undesirable operation of the vehicle in the event of an impairment due to a chemical substance (e.g., alcohol) or a medical event. To prevent improper operation, some vehicles include active systems that aim to identify impaired vehicle operators. One such system in common use is an ignition interlock system that requires breathalyzer analysis to activate the vehicle ignition.

[0003] In some cases, it is desirable for such systems to be as transparent as possible to the vehicle operator. Transparency of a system refers to the degree of knowledge the vehicle operator has of the system and how much direct interaction is required between the vehicle operator and the system. Greater transparency in the case of an impairment detection system can reduce the ability of the vehicle operator to attempt to bypass the system and can provide more comprehensive detection, allowing the system to identify instances of impairment (e.g., medical events) that the vehicle operator can not be aware of.

[0004] Accordingly, it is desirable to provide a vehicle system that identifies potential impairments of a vehicle operator in a transparent manner and automatically while the vehicle operator is outside of the vehicle. SUMMARY

[0005] In one example embodiment, an impairment detection method includes detecting an operator approaching a machine. The approach is monitored to determine a set of gait parameters of the operator based on output of a set of gait sensors. The set of gait parameters is provided to a long short-term memory (LSTM) recurrent neural network that determines a gait score by regressing on the set of gait parameters. The gait score is compared to an impairment threshold, and the operator is engaged in response to the gait score exceeding the impairment threshold.

[0006] The impairment detection method is transparent to the operator in addition to one or more features described herein when the gait score is less than the impairment threshold.

[0007] Detecting the approach of the operator includes one of detecting a location of at least one token object carried by the operator relative to the machine, detecting remote activation of at least one function of the machine, and confirming a human operator has approached by a human interacting with the machine in addition to one or more features described herein.

[0008] In addition to one or more of the features described herein, detecting the proximity of an operator includes confirming the person who has approached as an operator through human-machine interaction, wherein different gait scores are determined for all approaching people, and wherein the gait score compared to the impairment threshold is the gait score corresponding to the confirmed operator.

[0009] In addition to one or more of the features described herein, determining the set of gait parameters of the operator based on the output of the set of gait sensors includes isolating at least one gait parameter from the output of the set of gait sensors using at least one of a convolutional neural network (CNN), a gait energy image (GEI) classification module, a convolutional LSTM, a visual transformer, a graph neural network, a Bayesian network, a deep Gaussian process module, a multi-modal LLM, a visual language model, and a rule-based physiological image analysis.

[0010] In addition to one or more of the features described herein, the set of gait parameters includes velocity consistency, stride length, body sway, upper body bend, lower body bend, and path of travel.

[0011] In addition to one or more of the features described herein, the set of gait sensors includes at least one camera and at least one range-finding sensor.

[0012] In addition to one or more of the features described herein, the at least one range-finding sensor is a light detection and ranging (LIDAR) sensor.

[0013] In addition to one or more of the features described herein, engaging the operator in response to the gait score exceeding the impairment threshold includes outputting one of a textual notification and an audio notification to the operator in response to the gait exceeding the impairment threshold by any amount.

[0014] In addition to one or more of the features described herein, engaging the operator in response to the gait score exceeding the impairment threshold includes using at least one secondary impairment detection system to engage the operator.

[0015] In addition to one or more of the features described herein, the method further includes responding to the secondary impairment detection system providing an impairment detection below a secondary detection threshold by detecting a fatigue state of the operator and placing the machine in an alert state, the alert state including at least one of: louder notifications, larger text on at least one display screen, higher contrast on at least one display screen, and increased brightness on at least one display screen.

[0016] In addition to one or more of the features described herein, the method further includes responding to the secondary impairment detection system providing an impairment detection above a secondary detection threshold by disabling at least one machine system.

[0017] In addition to one or more of the features described herein, engaging the operator in response to the gait score exceeding the impairment threshold includes disabling at least one machine system in response to the gait score exceeding the impairment threshold by a maximum amount of impairment.

[0018] In addition to one or more of the features described herein, the machine is a motor vehicle, and wherein the operator is a driver of the motor vehicle.

[0019] In another example embodiment, a motor vehicle includes a controller having a gait detection module and at least one impairment operation prevention module. A set of sensors is in communication with the controller and is configured to sense an approaching vehicle operator. The gait detection module is configured to detect the approaching vehicle operator, determine a set of gait parameters of the approaching vehicle operator based on outputs of the set of gait sensors, and regress the set of gait parameters over time using a long short-term memory (LSTM) recurrent neural network to determine a gait score. The gait score is compared to an impairment threshold, and in response to the gait score exceeding the impairment threshold, the vehicle operator is engaged using the at least one impairment operation prevention module.

[0020] In addition to one or more of the features described herein, determining the set of gait parameters of the approaching vehicle operator based on outputs of the set of gait sensors includes isolating at least one gait parameter from the outputs of the set of gait sensors using at least one of a convolutional neural network (CNN), a gait energy image (GEI) classification module, a convolutional LSTM, a visual transformer, a graph neural network, a Bayesian network, a deep Gaussian process module, a multi-modal LLM, a visual language model, and a rule-based physiological image analysis.

[0021] In addition to one or more of the features described herein, the set of gait parameters includes parameters including velocity consistency, stride length, body sway, upper body bend, lower body bend, and path of travel.

[0022] In addition to one or more of the features described herein, the set of gait sensors includes at least one camera and at least one range-finding sensor.

[0023] The above features and advantages of the present disclosure, and other features and advantages, are readily apparent from the following detailed description, when taken in connection with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] Other features, aspects, and details are described in only an example in the following detailed description, with reference made to the drawings, in which:

[0025] Figure 1 is a schematic view of a motor vehicle;

[0026] Figure 2 depicts a combined view of a gait monitoring system of a vehicle operator approaching and a motor vehicle;

[0027] Figure 3 is a flowchart of the operation of an impairment detection system of a motor vehicle;

[0028] Figure 4 depicts sequential operation of a long short-term memory (LSTM) recurrent neural network that detects a motor vehicle operator that is in an impaired condition;

[0029] Figure 5 shows a set of motor vehicle operator states used to detect a gait using physiological analysis of a motor vehicle operator; and

[0030] Figure 6 demonstrates integration of motor vehicle gait detection with an impairment detection and response system of a motor vehicle. DETAILED DESCRIPTION

[0031] The following description is merely exemplary in nature and is not intended to limit the present disclosure, its application or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features.

[0032] According to exemplary embodiments, methods, apparatuses, and systems are provided for implementing a transparent impairment detection system of a motor vehicle. The impairment detection system uses sequential images and other sensor readings of a motor vehicle operator proximate to the vehicle to analyze the gait of the motor vehicle operator. The gait of the motor vehicle operator refers to the manner in which the vehicle operator walks and includes speed consistency, stride length, body sway, upper body bend, lower body bend, and path of travel.

[0033] The gait is assigned a gait score that represents how likely the motor vehicle operator is currently impaired. When the gait score exceeds a predefined set of criteria (e.g., a gait threshold), the vehicle responds by using one or more consolidated systems to notify the vehicle operator that they can be impaired. In some embodiments, the vehicle further activates one or more vehicle systems to prevent the impaired motor vehicle operator from operating the vehicle.

[0034] The embodiments described herein present a number of advantages and technical effects. Included among the advantages and technical effects is the ability to transparently identify potential impairment of a vehicle operator before the vehicle operator directly engages with the vehicle.

[0035] Embodiments are not limited for use with any particular vehicle and can be applied in a variety of additional contexts. For example, the transparent impairment detection can be incorporated into other heavy machinery including construction equipment, stationary machinery, or any similar system.

[0036] Figure 1 An exemplary vehicle 10 is shown that includes a vehicle body 12. Figure 2An isometric view of vehicle 10 and approaching vehicle operator 40 is shown. Vehicle controller 20 is disposed in vehicle 10 and includes gait monitoring module 22 and one or more obstacle operation prevention systems 24. While shown in vehicle 10 as a single, dedicated controller 20, it should be understood that in alternative examples, controller 20 can be a general purpose vehicle controller, multiple control systems distributed about vehicle 10 and operating in conjunction with one another, a remote controller in communication with vehicle 10, and / or any similar range of control configurations. Figure 1

[0037] One or more gait detection sensors 30 are arranged about vehicle 10 and are configured to monitor for the approach of vehicle operator 40. Detection from gait detection sensors 30 is provided to gait monitoring module 22 within controller 20. The gait monitoring module detects an impaired gait using statistical analysis of physiological parameters detected using sensor outputs and image processing, machine learning methods trained on sequential images generated of an impaired person, or a combination of both. When gait monitoring module 22 detects an impairment, a notification is provided to one or more obstacle operation prevention systems 24, allowing systems 24 to respond accordingly. In the illustrated example, gait sensors 30 are concentrated on driver side door area 32 in order to conserve processing resources of controller 20. In other examples, gait sensors 30 can monitor a larger area, up to the entire perimeter of vehicle 10.

[0038] In one example, gait detection sensors 30 include imaging devices such as cameras and ranging systems such as light and LiDAR systems.

[0039] In another example, gait detection sensors 30 can include one or more sensors from other vehicle systems such as mirror replacement systems, object detection and identification systems, and any similar vehicle systems including related sensor types. In such examples, the output of the shared sensors is provided to all relevant vehicle systems via a data sharing system such as a controller area network (CAN) system.

[0040] In one example, controller 20 detects the approach of vehicle operator 40 by detecting the relative position of a key fob, the operator’s mobile phone, or any similar token object carried by vehicle operator 40. In another example, controller 20 detects the approach of vehicle operator 40 in response to the remote initiation of one or more vehicle systems such as remote door unlocking and / or remote ignition systems.

[0041] ​In another example, the controller 20 analyzes the approach of any person, and when the approaching person has an impaired gait and is identified as a vehicle operator 40 by interacting with the vehicle 10 in some specified manner, the controller 20 engages the system 24. For example, the vehicle operator 40 can be identified when the approaching person opens a drive side door, attempts to start the vehicle 10, enters the vehicle 10, sits in the driver’s seat, or performs any similar action corresponding to being a vehicle operator 40.

[0042] With continued reference to Figures 1-2 , Figure 3 An advanced method 300 for operating the gait detection module 22 of the controller 20 using a machine learning based approach is shown. Initially, the method 300 begins operation in response to detecting an approaching vehicle operator 40 at step 310.

[0043] After the method 300 is initiated, the gait detection sensors 30 monitor the gait of the vehicle operator 40 approaching the vehicle 10 in a monitor gait step 320.

[0044] During the monitoring, output values such as images and distances provided by the gait detection sensors 30 are provided to the gait detection module 22. The gait detection module 22 then provides the sensor outputs to a machine learning based system, such as a long short-term memory (LSTM) recurrent neural network, which analyzes the sequential parameter values to determine a gait score corresponding to the gait of the vehicle operator 40. The LSTM recurrent neural network is trained using video feeds of the gaits of multiple people at multiple different impairment levels.

[0045] As the vehicle operator 40 approaches the vehicle 10 and is iteratively analyzed, the LSTM converges to a single gait score, and the converged gait score is compared to an impairment threshold in a gait impairment check 330. When the gait score does not exceed the impairment threshold (NO), the method 300 allows the vehicle operator 40 to continue vehicle operation as normal without notifying the vehicle operator 40 to perform an impairment check in an allow vehicle operation step 340. When no impairment is detected, the method 300 is completely transparent to the vehicle operator 40.

[0046] When the gait score exceeds the impairment threshold in the gait impairment check 330 (YES), the method 300 continues to engage the vehicle operator 40 in an engage vehicle operator step 350. The engagement can take the form of providing a warning to the vehicle operator 40 that the vehicle operator 40 can have an impairment, activating one or more additional vehicle systems 24 such as an ignition interlock, and / or actively disabling one or more functions of the vehicle 10.

[0047] With continued reference to Figure 3 , Figure 4An example gait impairment detection is shown using LSTM to process sequential images 412a, 412b, 412m, 412n of the vehicle operator 40. Each of the images 412a, 412b, 412m, 412n is analyzed using a convolutional neural network (CNN 420) to isolate the vehicle operator 40 and identify one or more features from the images 412a, 412b, 412m, 412n. The determined features of each image 412a, 412b, 412m, 412n and / or each pair of sequential images 412a, 412b, 412m, 412n are provided to the LSTM 410, which uses the features to identify learned parameters corresponding to an impaired gait. As described above, the LSTM is trained with a plurality of different individuals at a plurality of different impairment levels. The LSTM 410 provides the learned parameters to the neural network analysis portion 430, which regresses the learned parameters of the sequential images over time in order to determine how likely the vehicle operator 40 is impaired.

[0048] Figure 5 A plurality of gait parameters of the vehicle operator 40 are shown, which can be physiologically measured and are indicative of potential impairment. The physiological measurements of Figure 5 can be processed using statistical analysis including established impairment rules to determine a gait impairment score based on the physiological measurements. The gait parameters include stride 510, body sway 520, upper bend 530 at the waist of the vehicle operator 40, and lower bend 540 at the knees of the vehicle operator 40. In alternative examples, additional impairment gate parameters can be detected depending on the available sensors and positioning of the gait detection sensors 30 available.

[0049] In some alternative examples, the physiological measurements can be further input into the LSTM 410 along with corresponding images 412a, 412b, 412m, 412n, thereby providing a combined physiological and machine learning based gait score.

[0050] In further examples, the controller 20 can include additional machine learning methods configured to profile the non-impairment gait of the vehicle operator as well as one or more instances of the impaired gait, thereby further allowing the controller 20 to identify the impaired gait and the non-impairment gait of a particular vehicle operator. The controller 20 can further apply this identification to distinguish the vehicle operator from other pedestrians or people that can be approaching the vehicle 10.

[0051] In further examples, one or more gait parameters can be detected using alternative systems including gait energy image (GEI) classification, convolutional LSTMs, vision transformers, graph neural networks, Bayesian networks, deep Gaussian methods, multi-modal LLMs, and vision language models, and / or rule-based image analysis. In some examples, multiple gait parameter detection systems can be used in combination to detect different parameters and provide the detected parameters to the LSTM 410.

[0052] In some examples, the gait detection module 22 is configured to detect impairment in a cause-agnostic manner. In this way, the gait detection module 22 is able to detect impairment from alcohol or other chemicals as well as impairment that can occur due to a medical condition (e.g., stroke) that the vehicle operator 40 can not be aware of or that the vehicle operator 40 can not be aware of the severity of which results in impairment.

[0053] In some examples, the method of Figure 3 may be modified to include additional checks for levels of impairment such that the module 22 can provide increasingly strict responses as the gait impairment score increases. As an example, the engaging the vehicle operator step 350 can stop providing warnings to the vehicle operator 40 when the gait impairment score is slightly above a threshold, activate active impairment checks (e.g., ignition interlock) when the gait impairment score is above the threshold but below a maximum threshold, and completely prevent operation of the vehicle 10 by the vehicle operator 40 when the threshold is above the maximum threshold.

[0054] With continuing reference to Figures 1-5 , Figure 6 A method 600 is shown that integrates the method 300 of Figure 3 with a particular vehicle system that includes an ignition interlock. When the method 600 proceeds to the engaging the vehicle operator step 340, 610, the vehicle operator 40 is instructed to provide a breath sample to the ignition interlock to confirm an allowable blood alcohol content (BAC) level. During this step 340, 610, the controller 20 maintains the vehicle 10 in a no-gear, no-drive state that completely prevents the vehicle 10 from being operated.

[0055] The ignition interlock compares the detected BAC to a BAC limit in a BAC check 620. When the detected BAC is below the limit but a gait impairment is still detected, the method 600 continues in a notifying the vehicle operator step 630 that notifies the vehicle operator 40 that the detected BAC is below the limit. At the same time, the controller 20 notifies the vehicle operator 40 of potential impairment and enables operation of the vehicle 10. In some cases, one or more alert or awareness systems are activated when the vehicle 10 is allowed to be operated. In some examples, the awareness systems can include louder notifications, larger text, higher contrast, and brighter information screens.

[0056] Once the vehicle operator 40 begins operating the vehicle 10, the vehicle operator 40 is continuously monitored using existing driver monitoring systems. As operation continues, one or more alertness or awareness modules can be turned off or have their intensity reduced as the vehicle operator 40 establishes an alert driving profile.

[0057] When the BAC check 620 indicates that the vehicle operator 40 has a BAC above a threshold, the vehicle operator 40 is warned that their BAC exceeds the threshold and operation of the vehicle 10 is disabled in a disable vehicle step 640. In some examples, the vehicle 10 can include an emergency override by which the vehicle operator 40 or a proxy operator can override the lockout and allow operation of the vehicle 10.

[0058] The terms “a” and “an” do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item. The term “or” means “and / or” unless clearly indicated otherwise. References throughout this specification to “aspects” mean that a particular element described can be included in the at least one aspect described and can or can not be present in other aspects. Furthermore, it is to be understood that the described elements can be combined in any suitable manner in the various aspects.

[0059] When an element such as a layer, film, region, or substrate is referred to as being “on” another element, it can be directly on the other element or intervening elements can also be present. In contrast, when an element is referred to as being “directly on” another element, there are no intervening elements present.

[0060] Unless specified to the contrary herein, all test standards are the most recent standard in effect as of the filing date of this application, or, if priority is claimed, the filing date of the earliest priority application in which the test standard appears.

[0061] Unless defined otherwise, technical and scientific terms used herein have the same meaning as is commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0062] While the foregoing disclosure has been described in reference to exemplary embodiments, it will be understood by those skilled in the art that various changes can be made and equivalents can be substituted for elements thereof without departing from the scope of the disclosure. In addition, many modifications can be made to adapt a particular situation or material to the teachings of the disclosure without departing from the essential scope thereof. Therefore, it is intended that the disclosure not be limited to the particular embodiments disclosed, but will include all embodiments falling within the scope of the disclosure.

Claims

1. A method of impairment detection, comprising: detecting operator proximity to a machine; determining a set of gait parameters of the operator based on a set of gait sensor outputs; receiving the set of gait parameters at a long short-term memory (LSTM) recurrent neural network and determining a gait score by regressing the set of gait parameters; comparing the gait score to an impairment threshold; and engaging the operator in response to the gait score exceeding the impairment threshold. The method of impairment detection is transparent to the operator when the gait score is less than the impairment threshold.

2. The method of claim 1, wherein, Detecting operator proximity includes one of detecting a location of at least one token object carried by the operator relative to the machine, detecting remote activation of at least one function of the machine, and confirming a person who has approached as an operator by the person interacting with the machine.

3. The method of claim 1, wherein, Detecting operator proximity includes confirming a person who has approached as an operator by the person interacting with the machine, wherein a different gait score is determined for all persons who are approaching, and wherein the gait score compared to the impairment threshold is the gait score corresponding to the confirmed operator.

4. The method of claim 3, wherein, Determining a set of gait parameters of the operator based on a set of gait sensor outputs includes isolating at least one gait parameter from the set of gait sensor outputs using at least one of a convolutional neural network (CNN), a gait energy image (GEI) classification module, a convolutional LSTM, a visual transformer, a graph neural network, a Bayesian network, a deep Gaussian process module, a multi-modal LLM, a visual language model, and a rule-based physiological image analysis, and wherein the set of gait parameters includes pace consistency, stride length, body sway, upper body bend, lower body bend, and path of travel, and wherein the set of gait sensors includes at least one camera and at least one ranging sensor.

5. The method of claim 1, wherein, Engaging the operator in response to the gait score exceeding the impairment threshold includes outputting one of a textual notification and an audio notification to the operator in response to the gait exceeding the impairment threshold by any amount, and wherein engaging the operator in response to the gait score exceeding the impairment threshold includes using at least one secondary impairment detection system to engage the operator.

6. The method of claim 1, wherein, The at least one secondary impairment detection system includes a breathalyzer test system.

7. The method of claim 6, wherein, 8. The method of claim 6, further comprising responding to the secondary impairment detection system providing an impairment detection below a secondary detection threshold by detecting a fatigue state of the operator and placing the machine in an alert state, the alert state including at least one of louder notifications, larger text on at least one display screen, higher contrast on the at least one display screen, and increased brightness on the at least one display screen.

9. The method of claim 6, further comprising responding to the secondary impairment detection system providing an impairment detection above a secondary detection threshold by disabling at least one machine system. ​ 10. The method of claim 6, wherein, Engaging the operator in response to the gait score exceeding the impairment threshold includes disabling at least one machine system in response to the gait score exceeding the impairment threshold by a maximum amount of impairment.

Images