Prediction of operator input to steering element
The operator image is captured by the camera and combined with the torque sensor data, and a node model is constructed to judge the contact status of the operator's hands and steering wheel, solving the problem of insufficient detection accuracy in the prior art and improving the accuracy and safety of the assisted driving system.
Patent Information
- Application Number
- CN202410542566.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-04-30
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to accurately detect the operator's contact with the steering wheel, especially when the operator's hand is invisible within the camera's field of view, resulting in the assisted driving system being unable to accurately determine whether the torque sensor data should be responded to.
By capturing the operator's image by a camera mounted on the dashboard, building a node model, and combining torque sensor data, using a computer to perform programming to estimate the position and confidence level of the obstructed feature, to determine whether the operator's hand is in contact with the steering wheel, and then Determine the correlation to actuate the vehicle components.
Accurate detection of the operator's hands and steering wheel contact status is achieved, the accuracy of the assisted driving system is improved, and the safety of vehicle operation and the effectiveness of assisted driving functions are ensured.
Smart Images

Figure CN118907116A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure provides techniques for enhancing detection of operator contact with a steering wheel. Background Art
[0002] Torque sensors may be used in vehicle steering systems to determine the amount of torque applied to the steering wheel, for example, by an operator's hands and / or as a result of some other source. Summary of the invention
[0003] In an example, a camera and / or other sensor mounted on a dashboard of a vehicle may obtain an image that may include an operator positioned in a driver's seat of the vehicle, wherein certain parts of the operator (e.g., one or more of the operator's hands, forearms, etc.) are not visible within the camera's field of view. The computer may then execute programming using the image to construct a node model of the operator. The computer may additionally estimate the positions of features of the operator that are blocked from the camera's field of view. The computer may then execute programming to determine whether the confidence level about the operator's node model (including the estimated positions of features blocked from the camera's field of view) is consistent with the node model of the operator whose hands are in contact with the steering wheel. Based on a threshold confidence level (e.g., greater than 95%) that the operator's hands are in contact with the steering wheel, the computer may utilize torque measurements from a sensor positioned on a steering column coupled to the steering wheel. The torque measurements may then be input into an assisted driving application. Conversely, based on a threshold confidence level (e.g., less than 95%) that the operator's hands are in contact with the steering wheel, the computer may execute programming to actuate, for example, a notification that the operator has placed one or more hands in contact with the steering wheel.
[0004] In an example, a system may include a computer having a processor and a memory storing instructions executable by the processor to capture an image of an operator positioned in a steering position and determine a correlation between a confidence level of operator contact with a steering element and data output from a torque sensor based on parameters extracted from the captured image. The instructions may be further used to actuate a component in a vehicle based on the determined correlation.
[0005] In an example, output data from a torque sensor may represent applied torque as a function of time, and the steering element may include a steering wheel.
[0006] In an example, output data from the torque sensor may be obtained for a duration that an image of the operator is captured for a duration.
[0007] In an example, output data from the torque sensor may represent a torsional moment applied to the steering element.
[0008] In an example, the instructions may also include instructions for extracting parameters of the operator via a non-camera sensor.
[0009] In an example, output data from a torque sensor may be indicative of weight applied to a steering element.
[0010] In an example, the instructions may further include instructions for obtaining an input signal from a capacitive sensor of the steering element.
[0011] In an example, the extracted parameters may include the operator's head position.
[0012] In an example, the extracted parameters may include an operator's eye gaze direction.
[0013] In an example, the extracted parameters may include an operator's arm position.
[0014] In an example, the instructions may also include instructions for estimating a location of a feature of the operator that is occluded from a field of view of a camera capturing the image.
[0015] In an example, the instructions may also include instructions for implementing a learned model of an operator with one or more hands in contact with a steering element based on one or more of the operator's head position, the operator's eye gaze direction, or the operator's arm position.
[0016] In an example, the instructions may further include instructions for correlating the data output from the torque sensor prior to actuating the component.
[0017] In an example, a method may include: capturing an image of an operator positioned in a steering position; and determining a correlation between a confidence level of contact of the operator with a steering element and data output from a torque sensor based on parameters extracted from the captured image. The method may further include actuating a component in a vehicle based on the confidence level.
[0018] In an example, the output data from the torque sensor represents applied torque as a function of time, and wherein the steering element may include a steering wheel.
[0019] In an example, output data from a torque sensor may be obtained for a duration, and an image of the operator may be captured during the duration.
[0020] In an example, the images may include camera images and / or images derived from non-camera sensors.
[0021] In an example, the extracted parameters may include an operator's head position, an operator's eye gaze direction, and / or an operator's arm orientation.
[0022] In an example, the method may additionally include estimating a position of a feature of the operator that is occluded from the field of view of a camera capturing the image.
[0023] In an example, the method may further include implementing a learned model of the operator in contact with a steering element based on one or more of a head position of the operator, an eye gaze direction of the operator, and / or an arm orientation of the operator. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a block diagram of an example vehicle.
[0025] Figure 2A is an example camera view of an operator seated in a vehicle.
[0026] Figure 2B yes Figure 2A An example node model for an operator.
[0027] Figure 3A , Figure 3B is an example graph illustrating the confidence level that the operator's hand is in contact with the steering wheel.
[0028] Figure 4 is a process flow diagram illustrating an example process for detecting operator contact with a steering wheel. DETAILED DESCRIPTION
[0029] Figure 1 is a block diagram of an example vehicle. Figure 1 As shown in , the system 100 includes a vehicle 102, which includes a computer 104, which is communicatively coupled to various elements via a vehicle network 106, including sensors 108, subsystems or components 110, such as steering, propulsion, braking, human-machine interface (HMI) 112, and communication components 114. The computer 104, as well as the server 118 discussed below, includes a processor and memory. The memory of the computer 104 (such as those described herein) includes one or more forms of non-transitory media readable by the computer 104, and can store instructions executable by the computer 104 for performing various operations so that the vehicle computer is configured to perform various operations, including those disclosed herein.
[0030] For example, the computer 104 may include a general-purpose computer with a processor and memory as described above, and / or may include an electronic control unit (ECU) or controller for a specific function or set of functions, and / or a dedicated electronic circuit, the dedicated electronic circuit including an ASIC (application-specific integrated circuit) manufactured for a specific operation, (e.g., an ASIC for processing data from the sensor 108 and / or transmitting data from the sensor). In another example, the computer 104 may include an FPGA (field programmable gate array), which is an integrated circuit manufactured to be configurable by a user. In an example embodiment, a hardware description language such as VHDL (very high speed integrated circuit hardware description language) is used in electronic design automation to describe digital and mixed signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, and the logic components inside the FPGA can be configured based on VHDL programming (e.g., stored in a memory electrically connected or coupled to the FPGA circuit). In some examples, a combination of a processor, an ASIC, and / or an FPGA circuit may be included in the computer 104. Furthermore, the computer 104 may include multiple computers (eg, multiple ECUs, etc.) in a vehicle that operate together to perform the operations attributed herein to the computer 104 .
[0031] The memory of the computer 104 may include any type, such as a hard drive, a solid state drive, or any volatile or non-volatile media. The memory may store the collected data transmitted by the sensor 108. The memory may be a separate device from the computer 104, and the computer 104 may retrieve the information stored by the memory via a communication network in the vehicle (such as, the vehicle network 106) (e.g., through a controller area network (CAN) bus, a local interconnect network (LIN) bus, a wireless network, etc.). Alternatively or in addition, the memory may be part of the computer 104, for example, as a memory internal to the computer 104.
[0032] The computer 104 may include or access instructions to operate one or more components 110, such as vehicle braking, propulsion (e.g., one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior and / or exterior lights, an infotainment system, navigation, etc., and to determine whether and when the computer 104 (rather than a human operator) controls such operations. The computer 104 may include or be communicatively coupled to more than one processor, such as via a vehicle network 106, which may be included in components 110 (such as sensors 108, electronic control units (ECUs), etc.) included in the vehicle for monitoring and / or controlling various vehicle components (e.g., a powertrain controller, a brake controller, a steering controller, etc.).
[0033] The vehicle sensors 108 may also include a torque sensor 122 that operates to measure the torque (i.e., twisting moment) applied to a steering element (e.g., steering wheel 124) when an operator applies a rotational force to the steering wheel to control the heading of the vehicle 102. In an example, the torque sensor 122 is mounted to the steering column 126 to measure torque in the range of 0.02 Newton meters (Nm) to 0.2 Nm. The torque sensor 122 may include a calibrated strain gauge, for example, to provide a voltage signal proportional to the torque applied to the steering wheel 124. In an example, the torque sensor 122 is capable of always or nearly always measuring the torque of the operator steering the vehicle 102 with at least one hand resting on the steering wheel 124 as the vehicle 102 moves along the path 150. In response to the operator removing their hands from the steering wheel 124, the torque sensor 122 measures zero torque or other negligible torque amount transmitted to the steering column 126. It should be noted that the torque sensor 122 may measure torque corresponding to other objects placed in contact with the steering wheel 124, such as various foreign objects, e.g., a laptop computer, a water bottle, a coffee container, etc. Figure 1 A steering wheel is shown, but the techniques described herein may be applied to steering components other than a vehicle's steering wheel, such as a joystick, an aircraft control yoke, and the like.
[0034] The steering wheel 124 may include additional sensors 108, such as capacitive sensors, mounted in, on, or near the steering wheel 124, for example, to determine whether the operator's hand is in contact with the steering wheel 124. Such sensors may be operable to record a change in self-capacitance in response to the operator's hand being in proximity to a capacitive sensor mounted on the steering wheel 124. However, it should also be noted that capacitance (e.g., self-capacitance (such as the capacitance of a single capacitive sensor), mutual capacitance (such as the capacitance between two or more capacitive sensors), etc.) may be affected by the proximity of certain foreign objects (such as a water bottle, coffee container, etc.). In an example, other sensors may be positioned on the steering wheel 124, such as sensors for measuring force, torque, temperature, and other properties in a global or local coordinate reference position and / or orientation that may be directly or indirectly related to the operator's input to the steering wheel.
[0035] The computer 104 may generally be arranged to communicate over a vehicle network 106, which may include a communication bus in the vehicle, such as a controller area network (CAN), and / or other wired and / or wireless mechanisms. The vehicle network 106 corresponds to a communication network that may facilitate the exchange of messages between various on-vehicle devices (e.g., sensors 108, components 110, computer 104, and computers on the vehicle 102). The computer 104 may generally be programmed to send and / or receive messages to and from other devices in the vehicle (e.g., any or all of the ECUs, sensors 108, actuators, components 110, communication modules, human-machine interfaces (HMIs) 112) via the vehicle network 106. For example, various component 110 subsystems (e.g., components 110) may be controlled by corresponding ECUs.
[0036] Furthermore, in implementations where the computer 104 actually includes multiple devices, the vehicle network 106 may be used for communication between the devices represented in this disclosure as the computer 104. For example, the vehicle network 106 may provide communication capabilities via a wired bus (such as a CAN bus, a LIN bus), or may utilize any type of wireless communication capabilities. The vehicle network 106 may include a network in which any other wired communication technology and / or wireless communication technology (e.g., Ethernet, In some implementations, the vehicle network 106 may represent a combination of multiple networks that may be of different types that support communication between devices on the vehicle. For example, the vehicle network 106 may include: a CAN bus, in which some sensors and / or components in the vehicle communicate via the CAN bus; and a wired or wireless local area network, in which certain devices in the vehicle communicate according to Ethernet, and / or Bluetooth communication protocols.
[0037] The vehicle 102 typically includes a variety of sensors 108, in addition to torque sensors, capacitive sensors, and other sensors associated with determining whether the operator has placed his hands in contact with the steering wheel 124. The sensors 108 may include a set of devices that can obtain one or more measurements of one or more physical phenomena. Some of the sensors 108 detect variables that characterize the operating environment of the vehicle, such as vehicle speed settings, vehicle drag parameters, vehicle braking parameters, engine torque output, engine and transmission temperatures, battery temperatures, vehicle steering parameters, etc. Some of the sensors 108 detect variables that characterize the physical environment of the vehicle 102, such as ambient air temperature, humidity, weather conditions (e.g., rain, snow, etc.), parameters related to the inclination or slope of the road or other type of path on which the vehicle is traveling. In an example embodiment, the sensors 108 are operable to detect the position or orientation of the vehicle using, for example, signals from: a satellite positioning system (e.g., a global positioning system or GPS); an accelerometer, such as a piezoelectric or micro-electromechanical system MEMS; a gyroscope, such as a rate gyroscope, a ring laser gyroscope, or a fiber gyroscope; an inertial measurement unit IMU; and a magnetometer. In an example embodiment, the sensors 108 may include sensors for detecting aspects of the environment outside the vehicle 102, such as radar sensors, scanning laser rangefinders, cameras, etc. The sensors 108 may also include light detection and ranging (lidar) sensors that operate to detect the distance to an object by emitting laser pulses and measuring the flight time for the pulses to travel to the object and back. The sensors 108 may include a controller and / or a microprocessor that executes instructions to perform, for example, analog-to-digital conversion to convert sensed analog measurements and / or observations into input signals that may be provided to the computer 104, for example, via the vehicle network 106.
[0038] The computer 104 may be configured to utilize vehicle-to-vehicle (V2V) communications via the communication component 114 and / or may interface with devices external to the vehicle, for example, via a wide area network (WAN) 116 via V2V communications. The computer 104 may communicate external to the vehicle 102, such as via vehicle-to-infrastructure (V2I) communications, vehicle-to-external (V2X) communications, or V2X including cellular communications C-V2X and / or wireless communications cellular dedicated short range communications DSRC, etc. Communications external to the vehicle 102 may be facilitated via direct radio frequency communications and / or via a network server 118. The communication component 114 may include one or more mechanisms by which the computer 104 communicates with vehicles external to the vehicle 102, including any desired combination of wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any one or more desired network topologies when utilizing multiple communication mechanisms.
[0039] The vehicle 102 may include an HMI 112 (human machine interface), such as one or more of an infotainment display, a touch screen display, a microphone, a speaker, a haptic device, etc. A user (such as an operator of the vehicle 102) may provide input to a device such as a computer 104 via the HMI 112. The HMI 112 may communicate with the computer 104 via the vehicle network 106, for example, the HMI 112 may send messages to the computer 104 including user input provided via a touch screen, a microphone, a camera that captures gestures, etc., and / or may display output, such as via a display, a speaker, etc. In addition, the operation of the HMI 112 may be performed by a portable user device (not shown) (such as a smart phone, etc.) that communicates with the vehicle computer 104, such as via Bluetooth, etc.
[0040] WAN 116 may include one or more mechanisms by which computer 104 may communicate with server 118. Server 118 may include a device having one or more computing devices accessible via WAN 116, for example, with corresponding processors and memory and / or associated data repositories. In an example embodiment, vehicle 102 may include a wireless transceiver (i.e., a transmitter and / or a receiver) to send and receive messages outside vehicle 102. Thus, the network may include one or more of a variety of wired or wireless communication mechanisms, including any desired combination of wired (e.g., cable and fiber) and / or wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any one or more desired network topologies when utilizing multiple communication mechanisms. Exemplary communication networks include, for example, wireless communication networks using Bluetooth, Bluetooth Low Energy BLE, IEEE802.11, V2V or V2X (such as, cellular V2XCV2X, DSRC, etc.), local area networks, and / or wide area networks 116 including the Internet.
[0041] In an example implementation, the computer 104 may utilize a camera mounted on the dashboard of the vehicle to obtain an image of a portion of the interior of the vehicle 102, which portion may include an operator sitting in the driver's seat of the vehicle 102. In some instances, the image of the operator may include, for example, the operator's head, neck, shoulders, chest, and upper arms, but may exclude images of the operator's hands. Thus, suitable image processing programming executed by the computer 104 may extract parameters of the image to form a node model of the portion of the operator's body that is within the camera's field of view. The computer 104 may additionally execute programming to estimate the location of features that are excluded or obscured from the camera's field of view. The computer 104 may also execute programming to determine a confidence level regarding whether the node model of the operator is consistent with or at least consistent with the node model of the operator with one or more hands placed in contact with the steering wheel. In response to the confidence level meeting a predetermined threshold (e.g., greater than 95%) at the location of the operator's hands on the steering wheel, the computer 104 may obtain a measured steering column torque. The measurement of the steering column torque may then be input into an assisted driving application or an advanced driver assistance system (ADAS). Conversely, in response to the confidence level being below a predetermined threshold, the computer 104 may execute programming to actuate or generate a signal to indicate that one or more of the operator's hands are not in contact with the steering wheel 124. In such circumstances, the assisted driving application may ignore the measurement of steering column torque, which may represent the weight of a foreign object (e.g., a water bottle, a cup, a computing device, etc.) placed on the steering wheel. Thus, advantageously, the operator may be confident that the assisted driving function is being utilized in a manner consistent with the vehicle's operating guidelines, thereby impacting the experience of operating the vehicle 102 in a positive manner. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.
[0042] Exemplary System Operation
[0043] Figure 2A is an example camera view 200 of an operator 205 seated in the vehicle 102 . Figure 2AThe camera view of the vehicle 102 may be obtained via a dashboard-mounted camera 210 of the vehicle 102, which may be positioned near a central portion of the dashboard of the vehicle 102. The dashboard-mounted camera 210 may provide a field of view 215 that includes the operator 205 in the steering position of the vehicle, the passenger to the right of the operator 205, and the passenger behind the operator 205. The dashboard-mounted camera 210 may include any camera imaging device capable of detecting electromagnetic radiation within a certain wavelength range. For example, the dashboard-mounted camera 210 may detect visible light, infrared radiation, ultraviolet light, or a range of wavelengths including visible light, infrared light, and / or ultraviolet light. For example, the dashboard-mounted camera may include an image sensor such as a charge coupled device (CCD), an active pixel sensor such as a complementary metal oxide semiconductor (CMOS) sensor, and the like. In some examples, the image captured by the dashboard-mounted camera 210 may be supplemented or enhanced by an additional non-camera sensor of the sensor 108. In some examples, the image captured by the dashboard-mounted camera 210 may be complementary and / or fused with images captured by other cameras of the vehicle 102. Alternatively or additionally, images of operator 205 (which may be fused with or supplemented by images from other cameras of vehicle 102) may be fused with depth measurements and / or velocity measurements that indicate movement of operator 205 based on signals from a dashboard-mounted radar.
[0044] like Figure 2A As shown in , operator 205 appears to have directed his gaze away from the forward direction relative to vehicle 102 in order to view and / or interact with controls of the vehicle (e.g., infotainment display, climate control interface, etc.). Figure 2A As shown in , the operator 205 has tilted the right shoulder in a manner that is consistent with interacting with the dashboard-mounted controls of the vehicle 102. Figure 2A As further shown in FIG. 1 , the hand position of the operator 205 is obscured from the field of view of the dashboard-mounted camera 210. However, the detection of whether the operator 205 has placed one or more hands in contact with the steering wheel 124 of the vehicle 102 may be estimated through additional processing, such as through image processing and other programming of the computer 104 as further described below.
[0045] Figure 2B yes Figure 2A An example node model 250 of operator 205 is shown in FIG. 2 . In this context, a “node model” means a representation of operator 205 by a system of nodes connected by lines to represent body parts or features of operator 205. Thus, in FIG. Figure 2BIn the example of , the image of operator 205 may be represented by nodes representing the shoulders, elbows, hips, etc. of operator 205, with lines drawn to connect the nodes representing such features. Node model 250 may be generated using a skeletonization technique that operates to reduce a captured image to, for example, a one-pixel-wide representation. In this context, the "skeletonization" technique means representing the features of operator 205 in the image by the node model just described. Thus, in Figure 2B In the example, a skeletonization technique has been applied to the operator 205 so that the features of the operator 205 are represented by the node model 250. In the example, the image processing programming of the computer 104 can skeletonize the image of the operator 205 by continuously passing the images obtained by the camera 210 mounted on the dashboard. According to the skeletonization algorithm, during the continuous frame-by-frame image processing operation, the boundary pixels of the current image frame can be identified and removed from the current image frame based on whether the boundary pixels of the current image frame destroy the connectivity with the image frame represented by the remaining pixels of the image in the subsequent frame. Therefore, based on the image processing programming determining that all or substantially all pixels remaining in the current frame maintain their connectivity with the image from the previous frame, the node model 250 can be generated.
[0046] In response to generating the node model 250, the computer 104 may execute programming to compare whether the node model is consistent with or at least approximates a node model of an operator with one or more hands in contact with (e.g., resting on) the steering wheel 124. For example, the computer 104 may utilize a measure of the angle of a line drawn between nodes representing the shoulders of the operator 205. Alternatively or in addition, the computer 104 may utilize a measure of the angle between a line drawn between nodes representing the shoulders of the operator 205 and a line representing the upper arms of the operator. Alternatively or in addition, the computer 104 may utilize a measure of the angle between a line representing the upper arms of the operator 205 and a line representing the lower arms of the operator. Alternatively or in addition, the computer 104 may utilize a distance between points on the line representing the upper arms of the operator 205 and the line drawn between nodes representing the shoulders of the operator 205. The computer 104 may utilize additional parameters of the node model 250 to determine whether the node model is consistent with a node model of an operator with one or more hands in contact with the steering wheel 124.
[0047] In an example, the computer 104 may be programmed to implement a machine learning technique, such as a neural network, that is capable of predicting whether one or more hands of the operator 205 are in contact with (e.g., resting on) the steering wheel 124 based on the node model 250. For example, the neural network may be trained to receive an image of the operator 205 that is aggregated with additional data describing the operator's position and movement (such as data from other cameras / sensors inside the vehicle 102). The neural network may predict a skeletonized model, a confidence, the state of the hands on the steering wheel, and a predicted applied torque. The predicted skeletonized model confidence (e.g., the state of the hands on the steering wheel, etc.) may be compared to actual instances where the operator 205 has one or more hands in contact with the steering wheel 124. In an example, the time series of predicted torque and measured torque may be filtered to exclude certain one-time events, such as the vehicle 102 encountering a speed bump, a depression in the vehicle's path, etc. In an example, ground truth data collection may be used to train the neural network or provide training for use with another machine learning technique. Ground truth data collection may be based on technology utilized by Datagen, Inc., located at HaMelacha St. 3, Tel Aviv, Israel 6721503 (https: / / datagen.tech). In response to initial training of the machine learning implementation, cross-correlation and / or time series analysis may be utilized to determine whether the machine learning implementation is consistent and / or correlated within a time window (such as a time window in which the operator 205 intermittently places his hand in contact with the steering wheel 124). The machine learning model may be manually adjusted for certain variations, such as to account for the size and stature of the operator 205 and the effect of the operator's larger hands versus smaller hands on the torque applied to the steering wheel 124.
[0048] In some examples, the programming of the computer 104 performs a comparison of the node model 250 with the node models of operators who are known to place one or more hands in contact with the steering wheel 124. Such reference node models or parameters derived from the node models may be stored in a memory accessible to the computer 104. In such instances, the node model of the operator with one or more hands in contact with the steering wheel 124 may be compared with the node model of the operator without one or more hands away from the steering wheel. For example, the node model of the operator with the hands placed in contact with the steering wheel 124 may exhibit a particular angle or range of angles between the line representing the upper arm and the line drawn between the nodes representing the shoulders of the operator. Therefore, in response to the node model 250 exhibiting a similar angle between the line representing the upper arm of the operator 205 and the line drawn between the nodes representing the shoulders of the operator 205, the computer 104 may determine a confidence level that at least one of the operator's hands 205 is in contact with the steering wheel 124.
[0049] In an example, the computer 104 may execute programming to estimate one or more positions of one or more hands or other features of the operator 205 based on a comparison with a node model stored in a memory of the computer 104 (e.g., a node model derived from an image of an operator similar to the operator 205). In an example, the computer 104 may define a coordinate system for the interior of the vehicle 102, such as a three-dimensional (3D) Cartesian coordinate system having a specified origin and orthogonal X-axis, Y-axis, and Z-axis. The coordinate system may be used to describe properties of the camera 210, such as camera pose, depth of focus, etc. In addition, defining a coordinate system for the interior of the vehicle 102 may allow the computer 104 to specify the positions of nodes of the node model 250 relative to the defined coordinate system. For example, in response to determining that a line representing an upper arm portion of the operator 205 is tilted in a direction toward the base of the driver's seat, such as Figure 2B As shown in , the computer 104 may estimate the position of the right forearm and hand of the operator 205 based on a comparison with a node model stored in the memory of the computer 104. In another example, also as Figure 2A As shown in , the node representing the left elbow of the operator 205 is below the node representing the left shoulder of the operator. Therefore, the computer 104 can estimate the position of the left forearm and left hand of the operator 205 based on a comparison with the node model stored in the memory of the computer 104.
[0050] In some examples, the programming of computer 104 may implement a suitable learning model of operator 205 that refines over a period of time an estimate of whether operator 205's hands are in contact with steering vehicle 102 (e.g., placed in contact with steering wheel 124). For example, the learning model may include using a node model that includes a node representing the position of the operator's 205 head relative to other nodes, the operator's eye gaze direction, other positions of nodes representing the operator's 205 arms, etc. In addition, the learning model may be used, such as by accessing ( Figure 1 102) WAN 116 and server 118 to access the node model of the operator of the vehicle having capabilities similar to the capabilities of vehicle 102. Thus, computer 104 of vehicle 102 can collaborate with similar vehicles, which can further limit the ability to estimate the location of features of operator 205 that are excluded from the field of view of camera 210.
[0051] In an example, a node model of an operator known to place one or more hands in contact with the steering wheel 124 (which may be referred to as a reference model) may be stored in a memory accessible to the computer 104. Determining whether the node model 250 is consistent with the reference node model of an operator known to place one or more hands in contact with the steering wheel 124 may be the result of applying a suitable supervised machine learning process, such as via the server 118, which may utilize a corpus of node models representing operators having one or more hands in contact with the steering wheel 124 or (conversely) one or more hands separated from the steering wheel. In another example, a suitable unsupervised machine learning technique may be utilized, such as a generative adversarial network, which performs unsupervised development and / or improvement of a process for determining whether one or more hands of the operator 205 are in contact with the steering wheel 124. Thus, for example, during an unsupervised training process, the programming steps performed by the computer 104 and / or the server 118 may generate a node model of an operator having a hand placed in contact with the steering wheel 124 (e.g., "hands on") or conversely separated from the steering wheel. Such node models may be evaluated and / or reviewed by the programming of the computer 104 without the benefit of obtaining a high-level indication of the hands-on / hands-off state of the generated node model. In response to the computer 104 inaccurately identifying the hands-on / hands-off state of the operator 205, one or more parameters of the generated node model may be modified. In response to the modification of the parameters of the node model, the generative adversarial network may present the same or similar node model for review by the computer 104. Thus, in an example training environment that may be hosted by a server 118 in cooperation with the computer 104, the generative adversarial network may iteratively repeat the process of generating a node model, followed by attempted hands-on / hands-off detection by the programming of the computer 104, refinement of the parameters of the node model, generation of another node model, and the like.
[0052] The computer 104 may then execute programming to determine a confidence level that the operator's estimated feature locations (e.g., estimated feature locations 255, 260) are consistent with the operator's feature locations with a hand (or hands) placed in contact with the steering wheel 124. In this context, "confidence level" means the degree to which the node model 250 is consistent with a node model known to represent an operator with one or more hands placed in contact with the steering wheel 124. Thus, for example, perfect agreement between the node model 250 and a node model of an operator known to have one or more hands in contact with the steering wheel 124 stored in a memory accessible to the computer 104 may be assigned a confidence level of 100%. In another example, a significant disagreement between the node model 250 and a node model stored or accessible to the computer 104 may be assigned a lower confidence level, such as 80%, 75%, etc.
[0053] Also in this context, “estimated feature positions” means the predicted positions of limbs of operator 205 that are excluded or occluded from the field of view of camera 210 based on the observed positions of features of operator 205 represented by node model 250. For example, Figure 2B As shown in , the estimated feature location 255 corresponding to the right forearm and hand of the operator 205 is located near the estimated feature location 270 representing the leg of the operator 205. Therefore, the programming of the computer 104 can assign a relatively low confidence level that the right hand of the operator 205 is in contact with the steering wheel 124. In another example, also as shown in FIG. Figure 2B , the estimated feature position 260 corresponding to the left forearm and hand of the operator 205 is located, for example, in a coordinate system inside the vehicle 102 at a position close to the steering wheel 124. Therefore, the computer 104 can assign a relatively high confidence level that the left hand of the operator 205 is in contact with the steering wheel 124.
[0054] In an example of a machine learning model implementing a neural network, a Bayesian neural network or an ensemble approach may be used to improve the confidence level, where the variance of the confidence level may be reduced by generating additional data for training from a data set using a combination with repetition to produce multiple sets of original data sets. In addition, in an example, input signals from other components of the HMI 112 of the vehicle 102 may be used to improve the confidence level. For example, in response to the operator 205 interacting with the infotainment component of the vehicle 102, this may be verified using input signals from the infotainment component, and may also be verified to improve the prediction or confidence level of the node model 250. In another example, a first node model may be generated using features extracted from an image captured by the camera 210 and from a second camera having a field of view of the interior of the vehicle 102. The node model generated by the programming of the computer 104 that extracts the features captured by the camera 210 and the features captured from the second camera may enhance the confidence in the prediction of the position of the operator 205's hand. The confidence level may be further enhanced during operation of the vehicle at a time of day when the sun glare on the camera 210 is minimal. Additionally, in response to operator 205 applying a propulsion command, the operator may be more likely to place a hand in contact with steering element 124. During such times, the confidence level may be increased in response to operator 205 applying a propulsion command to vehicle 102, during which operator 205 may be likely to place one or more hands in contact with steering element 124.
[0055] In an example, the computer 104 may be programmed to estimate a predicted torque delivered to the steering column 126 in response to one or more hands of the operator 205 being in contact with the steering wheel 124. For example, in response to the computer 104 determining, for example, with a relatively high confidence level, that the hands and forearms of the operator 205 are in contact with the steering wheel 124, the computer 104 may predict an expected torque measured by the torque sensor 122. For example, in response to the computer 104 determining with a relatively high confidence level that the hands of the operator 205 are resting on the steering wheel 124, the computer 104 may predict, based on measurements from previous instances, that the torque sensor 122 will measure a steering column torque of 0.05 Nm corresponding to the weight of the hands of the operator 205. In another example, in response to the computer 104 determining with a relatively high confidence level that the hands of the operator 205 are resting on the steering wheel 124, the computer 104 may predict that the torque sensor will measure a steering column torque of 0.10 Nm corresponding to the weight of the hands of the operator 205.
[0056] FIG. 3A to FIG. 3B is an example graph showing the confidence level that the operator's hand is in contact with the steering element at the measured torque. Figure 2A , Figure 2BAs described, the dashboard-mounted camera 210 may capture an image of an operator 205 of the vehicle 102, wherein certain features of the operator are obscured from the camera's field of view. Image processing programming operating on the computer 104 may perform, for example, a skeletonization algorithm, wherein during a continuous frame-by-frame image processing operation, boundary pixels are identified and removed from the current image frame based on whether the boundary pixels of the current image frame break connectivity with the image frame represented by the remaining pixels of the image in the subsequent frame. Based on the image processing programming determining that all or substantially all pixels remaining in the current frame retain their connectivity with the image from the previous frame, a node model 250 may be generated. The computer 104 may then estimate the locations of the features obscured from the camera's field of view and determine a confidence level that such features indicate that the operator 205's hands are in contact with the steering wheel 124. Based on a correlation between the confidence level that the operator 205 has placed one or more hands in contact with the steering wheel 124 and the output data from the torque sensor 122, the computer 104 may determine whether the signal output from the torque sensor 122 can be considered to indicate that the operator 205 has placed one or more hands in contact with the steering wheel 124. Thus, in an example, in response to the computer 104 determining a high confidence level (e.g., greater than 95%) that the operator has placed one or more hands in contact with the steering wheel 124, the computer 104 may be programmed to determine that the signal output from the torque sensor 122 indicates that the operator has one or more hands in contact with the steering wheel 124. In response to the computer 104 determining a high confidence level (e.g., less than 60%) that the operator has placed one or more hands in contact with the steering wheel 124, the computer 104 may be programmed to not output a determination that the signal output from the torque sensor 122 indicates that the operator has one or more hands in contact with the steering wheel 124.
[0057] As shown in example 300, based on the node model 250 of the operator 205, the programming of the computer 104 may be executed to determine a confidence level that the operator 205 has at least one hand in contact with the steering wheel 124. The confidence level may then be associated with the output data from the torque sensor 122. Such a correlation (e.g., a correlation greater than 0.95) between the confidence level and the output data from the torque sensor 122 may be utilized to determine that the output data from the torque sensor 122 indicates that the operator 205 has one or more hands placed in contact with the steering wheel 124. Any suitable technique for calculating correlation may be utilized, such as calculating a Pearson correlation coefficient, a Matthews correlation coefficient, etc. Thus, as Figure 3A As shown in , over a certain duration, such as between time t0 and time t1, the programming of computer 104 may calculate a relatively high confidence level that may be associated with the output data from sensor 122 indicating the varying (positive and negative) levels of torque applied to steering wheel 124. However, as Figure 3AIt can also be seen in that between time t1 and time t2, the programming of the computer 104 can calculate a relatively low confidence level, which can be associated with the output data from the sensor 122 indicating the changing torque level. Therefore, the data from the torque sensor 122 may be below a specified confidence threshold. The specified threshold may be set by an assisted driving application (e.g., ADAS) according to the design parameters of such an application. For example, in the case where the confidence level of the torque output data is low, the operator 205 may have removed his hands from the steering wheel 124 and may temporarily position a foreign object on the steering wheel. Between time t2 and time t3, the computer 104 can calculate a relatively high confidence level, which may indicate that the operator 205 has placed one or more hands in contact with the steering wheel 124 again. Therefore, the data from the torque sensor 122 may be assigned a confidence level that exceeds the threshold, thereby indicating that the operator 205 has one or more hands in contact with the steering wheel 124.
[0058] exist Figure 3B In the example graph 350 shown in FIG. 1 , based on the node model 250 of the operator 205, the programming of the computer 104 can calculate the confidence level that the operator 205 has at least one hand in contact with the steering wheel 124. Figure 3B As shown in , the computer 104 has calculated a relatively low confidence level. As previously described, the computer 104 may calculate such a relatively low confidence level in response to estimating that the operator 205's hands may not be in contact with the steering wheel 124. However, as also shown in FIG. Figure 3B As shown in , the output data from the torque sensor 122 indicates that a relatively constant torque value has been applied to the steering wheel 124. Therefore, there is a low correlation between the confidence level calculated by the computer 104 and the torque activity. Therefore, the output data from the torque sensor 122 can be assigned a confidence level below the threshold, indicating that the operator 205 cannot be considered to have placed one or more hands in contact with the steering wheel 124. Such a low correlation (e.g., less than 0.8) between the confidence level calculated by the computer 104 and the torque activity can indicate that the operator 250 has placed a foreign object on the steering wheel 124, such as a laptop computer, an entertainment device, etc.
[0059] Figure 4 4 is a process flow diagram 400 illustrating an example process for detecting contact of the operator 205 with the steering wheel 124. For example, the process 400 may begin at block 405, such as in response to the vehicle 102 being placed in an on state or "drive" state to operate on a road. Block 405 may include the dashboard-mounted camera 210 capturing an image of the interior of the vehicle 102 that may include the operator 205, wherein the captured image may exclude certain features, such as the forearm and hand of the operator 205.
[0060] Process 400 may continue at block 410, which may include computer 104 performing image processing, such as a skeletonization technique, wherein in each of several processed frames, it is determined whether the boundary pixels of the image frame are connected or not connected to other pixels in the processed frame. In response to the pixels not being connected, such pixels are removed and a subsequent processed frame is initiated. The skeletonization technique is typically completed after there are no additional unconnected pixels in the processed frame. Block 410 may include extracting parameters representing additional features of operator 205, such as determining the operator's head orientation, the operator's eye gaze direction, etc.
[0061] Process 400 may continue at block 415, which may include forming a node model of operator 205 in response to computer 104 performing the skeletonization technique performed at block 410. Computer 104 may execute programming to estimate the positions of features excluded from the field of view of camera 210, such as the position of a forearm and hand of operator 205. Computer 104 may access a database of node models similar to the node model of operator 205, and / or may employ a learned model of operator 205, wherein estimates of feature positions are refined over a period of time to enhance the accuracy of the estimated feature positions.
[0062] In an example where the programming of computer 104 implements a neural network to determine the confidence level, block 415 may be omitted. In such an example, the neural network may be operable to analyze parameters extracted from images captured by camera 210 without applying a skeletonization technique, for example, to the parameters extracted from the captured images.
[0063] The process 400 may continue at block 420, which may include the computer 104 executing programming to determine a confidence level that one or more hands of the operator 205 are in contact with the steering wheel 124 based on the node model 250. For example, based on the node model 250 indicating that the estimated position of the operator's 205 hands is proximate to the steering wheel 124, the computer 104 may assign a relatively high confidence level that the operator 205 has one or more hands in contact with the steering element of the vehicle 102. In another example, based on the node model 250 indicating that the estimated position of the operator's 205 hands is distal to the steering wheel 124, the computer 104 may assign a relatively low confidence level that the operator 205 has one or more hands in contact with the steering element of the vehicle 102.
[0064] Process 400 may continue at block 425, which may include the computer 104 obtaining torque measurements from the torque sensor 122. The torque sensor 122 may include any suitable torque measurement device, such as a calibrated strain gauge, that may output a signal trace that varies over time. In an example, the torque sensor 122 may output a non-zero or non-negligible signal, for example, in response to one or more hands of the operator 205 contacting the steering wheel 124.
[0065] The process 400 may continue at block 430, which may include the computer 104 executing programming to correlate the confidence level obtained at block 420 based on the node model 250 of the obtained image with the measurement results from the torque sensor 122. The computer 104 may calculate the correlation coefficient via any suitable technique (such as via a Pearson correlation coefficient, a Matthews correlation coefficient, etc.). The correlation at block 430 may enhance the likelihood that the output data from the torque sensor 122 indicates that the operator 205 is occupied (e.g., the operator 205's hands are placed in contact with the steering wheel 124 while operating the vehicle 102). In an example, in response to the computer 104 assigning a relatively low confidence level that the operator 205's hands are in contact with the steering wheel 124, the programming of the computer 104 may determine not to provide an output that the operator 205 has one or more hands in contact with the steering wheel 124. In such an example, the signal from the torque sensor 122 may represent the weight of a foreign object (e.g., a water bottle, a piece of fruit, a coffee cup, a laptop computer, etc.) resting on the steering wheel 124 instead of one or more hands of the operator 205. In another example, in response to computer 104 assigning a relatively low confidence level that operator 205 's hands are in contact with steering wheel 124 , programming of computer 104 may determine that the output signal from torque sensor 122 represents a condition in which operator 205 has one or more hands in contact with steering wheel 124 .
[0066] Process 400 may continue at block 435, including computer 104 executing programming to determine whether the correlation between the confidence level that the operator 205's hands are in contact with the steering wheel 124 and the measurements from the torque sensor 122 is above a predetermined threshold. In response to determining that the correlation exceeds the threshold (e.g., greater than 0.95), process 400 may return to block 405, at which time the camera 210 captures a second image of the operator 205. Thus, it can be seen that process 400 represents a dynamic operator monitoring cycle, in which the camera 210 captures images of the operator 205 at predetermined intervals (block 405). The images are then processed to form a node model and the locations of features that are excluded from the camera's field of view are estimated (blocks 410, 415). A confidence level is then determined (block 420), the confidence level is associated with the torque measurements (block 425), and a determination that the operator currently has one or more hands in contact with the steering wheel 124 is output based on the correlation (block 430).
[0067] In response to determining at block 435 that there is less than a predetermined threshold level of correlation (e.g., less than 0.95) between the confidence level determined at block 420 and the output signal from the torque sensor 122, process 400 may continue at block 440, where the computer 104 may output a signal or command to actuate a component 110 of the vehicle 102. A correlation less than the predetermined threshold may indicate a situation where the operator 205 has placed a foreign object on the steering wheel 124. In an example, block 440 may include the computer 104 actuating or generating a message to the operator 205 via the HMI 112. Alternatively or in addition, the computer 104 may initiate a tactile output to the steering wheel 124, or may cause the HMI 112 to output an audible signal. Further alternatively or in addition, the computer 104 may output a command to control a subsystem or component 110, including steering, braking, and / or propulsion of the vehicle 102, for example, to steer and / or decelerate the vehicle 102 in response to determining that the operator's hands are not in contact with the steering wheel.
[0068] After block 440, process 400 ends.
[0069] The present disclosure has been described in an illustrative manner, and it should be understood that the terminology which has been used is intended to be in the nature of words of description rather than limitation. In light of the above teachings, many modifications and variations of the present disclosure are possible, and the present disclosure may be practiced in other ways than specifically described.
[0070] In the accompanying drawings, the same reference numerals indicate the same elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring in a particular order, unless otherwise specified or as can be seen from the context, such processes may be practiced in the case where the steps are performed in an order other than the order described herein. Similarly, it should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the process herein is provided for the purpose of illustrating certain embodiments and should never be interpreted as limiting the claimed invention.
[0071] The adjectives first and second are used throughout this document as identifiers and, unless expressly stated otherwise, are not intended to denote importance, order, or quantity.
[0072] The term exemplary is used herein in the sense of referring to an example, for example, reference to an exemplary widget should be construed to refer only to the example of the widget.
[0073] The use of "responsive to," "based on," and "upon determining" herein indicates a causal relationship, not merely a temporal relationship.
[0074] Computer executable instructions can be compiled or interpreted by a computer program created using various programming languages and / or technologies, including but not limited to Java, C, C++, Visual Basic, Java Script, Perl, Python, HTML, etc., alone or in combination. In general, a processor (e.g., a microprocessor) receives instructions, for example, from a memory, a computer-readable medium, etc., and executes these instructions, thereby performing one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using various computer-readable media. Files in a networked device are typically a collection of data stored on a computer-readable medium (such as a storage medium, a random access memory, etc.). Computer-readable media include any medium that participates in providing data (e.g., instructions) that can be read by a computer. Such media can take many forms, including but not limited to non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, including internal components that constitute a system bus connected to a processor of a computer. Common forms of computer readable media include, for example, RAM, PROM, EPROM, Flash EEPROM, any other memory chip or cassette or any other medium from which a computer can read.
[0075] According to the present invention, a system is provided, having: a computer, the computer including a processor and a memory, the memory storing instructions executable by the processor to: capture an image of an operator positioned in a steering position; determine a correlation between a confidence level of contact of the operator with a steering element and data output from a torque sensor based on parameters extracted from the captured image; and actuate a component in a vehicle based on the determined correlation.
[0076] According to an embodiment, the output data from the torque sensor represents applied torque as a function of time, and wherein the steering element comprises a steering wheel.
[0077] According to an embodiment, the output data from the torque sensor is obtained over a duration, and wherein the image of the operator is captured during the duration.
[0078] According to an embodiment, the output data from the torque sensor is representative of a torsional moment applied to the steering element.
[0079] According to an embodiment, the instructions further comprise instructions for extracting parameters of the operator via a non-camera sensor.
[0080] According to an embodiment, the output data from the torque sensor is indicative of a weight applied to the steering element.
[0081] According to an embodiment, the instructions further comprise instructions for obtaining an input signal from a capacitive sensor of the steering element.
[0082] According to an embodiment, the extracted parameters include the head position of the operator.
[0083] According to an embodiment, the extracted parameters include an eye gaze direction of the operator.
[0084] According to an embodiment, the extracted parameters include the operator's arm position.
[0085] According to an embodiment, the instructions further comprise instructions for estimating the position of a feature of the operator that is occluded from the field of view of a camera capturing the image.
[0086] According to an embodiment, the instructions also include instructions for implementing a learning model of the operator having one or more hands in contact with the steering element based on one or more of the operator's head position, the operator's eye gaze direction, or the operator's arm position.
[0087] According to an embodiment, the instructions further comprise instructions for correlating data output from the torque sensor prior to actuating the component.
[0088] According to the present invention, a method includes: capturing an image of an operator positioned in a steering position; determining a correlation between a confidence level of contact between the operator and a steering element and data output from a torque sensor based on parameters extracted from the captured image; and actuating a component in a vehicle based on the confidence level.
[0089] In one aspect of the invention, said output data from said torque sensor represents applied torque as a function of time, and wherein said steering element comprises a steering wheel.
[0090] In one aspect of the invention, said output data from said torque sensor is obtained over a duration, and wherein said image of said operator is captured during said duration.
[0091] In one aspect of the invention, the image comprises a camera image and / or an image derived from a non-camera sensor.
[0092] In one aspect of the invention, the extracted parameters include the operator's head position, the operator's eye gaze direction and / or the operator's arm orientation.
[0093] In one aspect of the invention, the method comprises estimating the position of features of the operator that are occluded from the field of view of a camera capturing the image.
[0094] In one aspect of the invention, the method includes implementing a learned model of an operator in contact with the steering element based on one or more of the operator's head position, the operator's eye gaze direction, and / or the operator's arm orientation.
Claims
1. A method, include: capturing an image of an operator positioned in a steering position; determining a correlation between a confidence level of operator contact with a steering element and data output from a torque sensor based on parameters extracted from the captured images; as well as Components in the vehicle are actuated based on the determined correlation.
2. The method of claim 1, wherein the output data from the torque sensor represents applied torque as a function of time, and wherein the steering element comprises a steering wheel.
3. The method of claim 1, wherein the output data from the torque sensor is obtained over a duration, and wherein the image of the operator is captured during the duration.
4. The method of claim 1, wherein the output data from the torque sensor represents a torsional moment applied to the steering element.
5. The method according to claim 1, further comprising: include: Parameters of the operator are extracted via non-camera sensors.
6. The method of claim 1, wherein the output data from the torque sensor is indicative of a weight applied to the steering element.
7. The method according to claim 1, further comprising: include: An input signal is obtained from a capacitive sensor of the steering element.
8. The method of claim 1, wherein the extracted parameters include a head position of the operator.
9. The method of claim 1, wherein the extracted parameters include an eye gaze direction of the operator.
10. The method of claim 1, wherein the extracted parameters include the operator's arm position.
11. The method of claim 1, further comprising: include: The positions of features of the operator that are occluded from the field of view of a camera that captured the image are estimated.
12. The method of claim 1, further comprising: include: A learned model of the operator having one or more hands in contact with the steering element is implemented based on one or more of the operator's head position, the operator's eye gaze direction, or the operator's arm position.
13. The method of claim 1, further comprising: include: The data output from the torque sensor is correlated prior to actuating the component.
14. A computer programmed to implement the method of any one of claims 1 to 13.
15. A vehicle comprising the computer of claim 14.
