SYSTEMS AND METHODS FOR DETECTING DRIVER BEHAVIOR

Abstract

A vehicle is disclosed that includes a steering wheel, a first detection unit, a second detection unit, and a processor. The first detection unit can be configured to detect a first parameter associated with the steering wheel, and the second detection unit can be configured to receive an input associated with a vehicle occupant. The processor can be configured to receive the input and the first parameter and to estimate a second parameter associated with a vehicle occupant's interaction with the steering wheel based on the input. The processor can further be configured to correlate the first and second parameters and, based on the correlation, to determine whether a predefined condition is potentially met.Furthermore, the processor can transmit a notification in response to a determination that the predefined condition may be met.

Description

FIELD OF TECHNOLOGY

[0001] The present disclosure relates to systems and methods for detecting driver behavior, and more particularly to systems and methods for detecting driver steering spoofing based on depth sensing. GENERAL STATE OF THE ART

[0002] Many modern vehicles include features that assist drivers in driving their vehicles comfortably. For example, vehicles include features that assist drivers during lane changes, reversing the vehicle, navigating traffic, and / or the like. Modern vehicles also include features for detecting driver behavior in the vehicle, including driver inattention. Upon detecting that the driver may be inattentive, the vehicle can warn the driver to pay attention. There are cases in which drivers spoof the vehicle feature that detects driver behavior in the vehicle. A system is required to detect such cases. BRIEF PRESENTATION

[0003] The present disclosure describes a vehicle configured to detect whether a vehicle driver may be spoofing the vehicle or a steering wheel. In particular, the vehicle may be configured to detect whether the driver's hands are actually on the steering wheel or whether the driver may be using an external object or spoofing device to spoof the steering wheel. The vehicle may include a driver spoofing detection system ("System") that may receive inputs from a first detection unit and a second detection unit of the vehicle and determine, based on the received inputs, that the driver may be spoofing the steering wheel.The first detection unit can be, for example, one or more steering wheel sensors and the second detection unit can be, for example, one or more time-of-flight (TOF) cameras in the vehicle interior, sensors for radio-based positioning and distance measurement (Radio Detection and Ranging - Radar), sensors for optical distance and speed measurement (Light Detection and Ranging - Lidar) and / or the like.In some aspects, the system may determine that the driver may be spoofing the steering wheel based on the inputs obtained from the first and second detection units when the driver's hands near the steering wheel may not be visible in the images captured by the vehicle's in-vehicle cameras, but an upper body portion of the driver may be visible in the depth images / map obtained from the second detection unit.

[0004] In some aspects, to determine a possible spoofing action by a driver, the system may first estimate a position of a driver's hands in a vehicle interior portion based on the inputs obtained from the second detection unit. In an example aspect, the system may estimate the position of the driver's hands in the vehicle interior portion by generating a digital driver skeleton model based on one or more artificial intelligence / machine learning-based image processing algorithms. In response to estimating the position of the driver's hands, the system may estimate a probability that the driver's hands are touching the steering wheel and a torque / rotation angle of the steering wheel based on the estimated driver's hand position. The system may then correlate the estimated probability / estimated steering wheel torque with the inputs obtained from the first detection unit.Based on the correlation, the system may determine that the driver may be spoofing the steering wheel. Specifically, based on the correlation, the system may determine that the driver's hands may not actually be on the steering wheel. For example, the system may determine that the driver's hands may not be on the steering wheel if the degree of correlation between the estimated probability / estimated steering wheel torque and the inputs obtained by the first detection unit is low (e.g., less than a predefined threshold).

[0005] In response to determining that the driver may be spoofing the steering wheel, the system may transmit a notification to a human-machine interface of the vehicle and / or a user device requesting the driver to hold the steering wheel with their hands and remove the spoofing device / external object.

[0006] The present disclosure discloses a vehicle that detects whether the driver may be spoofing the steering wheel. The system may require the driver to use their hands to hold the steering wheel if the vehicle detects that the driver may be spoofing. Furthermore, the vehicle uses existing vehicle hardware and components to perform the detection, thus requiring no external hardware components to operate according to the present disclosure.

[0007] These and other advantages of the present disclosure are provided in detail in this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The detailed description is presented with reference to the accompanying drawings. The use of the same reference numerals may indicate similar or identical elements. Different elements and / or components than those illustrated in the drawings may be used for different embodiments, and some elements and / or components may not be present in different embodiments. The elements and / or components in the figures are not necessarily drawn to scale. Throughout the disclosure, the singular and plural terms may be used interchangeably, depending on the context. Fig. 1 illustrates an exemplary vehicle in which techniques and structures for providing the systems and methods disclosed in this document may be implemented. Fig. 2 illustrates a block diagram of an exemplary driver behavior detection system according to the present disclosure. Fig. 3 illustrates an example driver image, a driver skeleton model, and an example steering wheel touch probability and an example steering wheel torque according to the present disclosure. Fig. 4 illustrates first example estimated parameters associated with a driver interaction with a steering wheel and inputs obtained from one or more vehicle sensors, according to the present disclosure. Fig. 5 illustrates second example estimated parameters associated with driver interaction with a steering wheel and inputs obtained from one or more vehicle sensors, according to the present disclosure. Fig. 6 illustrates a flowchart of an exemplary method for detecting driver behavior in a vehicle according to the present disclosure. DETAILED DESCRIPTION

[0009] The disclosure is described in more detail below in this specification with reference to the accompanying drawings, in which exemplary embodiments of the disclosure are illustrated and are not intended to be limiting.

[0010] Fig. 1 illustrates an exemplary vehicle 100 in which techniques and structures for providing the systems and methods disclosed herein may be implemented. The vehicle 100 may take the form of any passenger or commercial vehicle, such as a car, a work vehicle, a crossover vehicle, a truck, a van, a minivan, a taxi, a bus, etc. The vehicle 100 may be a manually driven vehicle and / or configured to operate in a partially autonomous mode, and may include any powertrain, such as a gasoline engine, one or more electrically actuated motors, a hybrid system, etc. A vehicle occupant / user 102 (or a driver 102) may drive the vehicle 100. In one exemplary aspect, the driver 102 may drive the vehicle 100 by using and maneuvering a vehicle steering wheel 104 (or steering wheel 104).

[0011] In some aspects, the vehicle 100 may include one or more steering wheel sensors (in Fig. 2 as the first detection unit 242) that may be configured to detect touch of or interaction with the steering wheel 104 by a driver's hands. For example, the steering wheel sensors may include capacitive touch sensors and / or steering wheel rotation or torque sensors that may be configured to detect touch of the driver's hands on the steering wheel 104 (e.g., using capacitive sensing) and the rotation angle / torque of the steering wheel (e.g., using rotation or torque sensors) when the driver 102 turns the steering wheel 104.

[0012] The steering wheel sensors may be configured to continuously (or at a predefined frequency) detect an interaction of the driver's hands, e.g., a touch or movement associated with the steering wheel 104, when the vehicle 100 may be in motion. The vehicle 100 may be configured to display a notification on a human-machine interface (HMI) / infotainment system (in Fig. 2 as infotainment system 240) and / or a user device (in Fig. 2 as user device 204) requesting the driver 102 to hold the steering wheel 104 with their hands when the steering wheel sensors detect that the driver's hands may not be touching the steering wheel 104 and the vehicle 100 may be moving. In response to hearing / viewing the notification, the driver 102 may hold the steering wheel 104 with their hands.

[0013] There may be instances where the driver 102 may be spoofing the vehicle 100 or the steering wheel sensors when the vehicle 100 may be moving, such that the steering wheel sensors may detect that the driver's hands are on the steering wheel 104; however, in reality, the driver's hands may not be on the steering wheel 104. For example, the driver 102 may place an object 106 (e.g., a ball, etc.) on the steering wheel 104 (as shown in view 108), such that the steering wheel sensors may "detect" that the driver's hands are on the steering wheel 104; however, in reality, the object 106 may be on the steering wheel 104. In this case, the driver's hands may be engaged in another activity (e.g., operating a user device or the hands may just be resting on the driver's lap).In some aspects, the driver 102 may also install third-party electromechanical, electromagnetic, or digital spoofing devices that can spoof the steering wheel sensors to detect that the driver's hands may be on the steering wheel 104.

[0014] To detect such cases where a driver spoofs the vehicle 100 or the steering wheel 104, the vehicle 100 may include a driver spoofing detection system (in Fig. 2 as a driver spoofing detection system 214) that can detect when the driver 102 attempts to spoof the vehicle 100. In some aspects, the driver spoofing detection system (“System”) can be configured to receive inputs from one or more depth sensing sensors in the vehicle interior (in Fig. 2 as second detection unit 244) and, based on the acquired inputs, determine whether the driver 102 may be spoofing the vehicle 100. The depth sensing sensors may include one or more vehicle interior cameras, including, but not limited to, a time-of-flight (TOF) camera, a gesture-triggered TOF camera, and / or the like. The depth sensing sensors may further include a radar sensor, a lidar sensor, and / or the like. In some aspects, according to the present disclosure, the depth sensing sensors may not require the steering wheel 104 and / or the driver's hand to be within the field of view (FOV) of the sensors.In other words, based on the inputs obtained from the depth sensing sensors, the system may determine whether the driver 102 may be spoofing the vehicle 100, even if the steering wheel 102 and / or the driver's hands are not within the FOV of the sensors.

[0015] In some aspects, the system may be an artificial intelligence (AI) / machine learning (ML)-based system that may be configured to estimate a driver's hand position within the vehicle 100 (particularly within a vehicle interior portion) based on the inputs obtained from the depth-sensing sensors (which may not include a view of the steering wheel 104 and / or the driver's hands, as described above). In response to estimating the driver's hand position within the vehicle interior portion, the system may estimate a probability that the driver's hands are contacting the steering wheel 104 and a rotation angle or torque of the steering wheel when the driver 102 moves the hands while contacting the steering wheel 104 based on the estimated driver's hand position.If the driver 102 has adjusted the position of the steering wheel 104, the system may determine the adjusted position of the steering wheel by obtaining inputs from a control unit of the vehicle (in . Fig. 2 as VCU 212) and / or by correlating the predicted driver hand position and inputs obtained from one or more steering wheel sensors.

[0016] In some aspects, in addition to or alternatively to estimating the probability as described above, the system may use Bayesian or non-Bayesian approaches (as part of AI / ML) to determine a "confidence value" or "confidence level" of the detection of the driver's hands touching the steering wheel 104, which may be used for thresholding (e.g., only confidence levels above a predefined threshold may be considered for further action). The system may also use an end-to-end neural network to determine the confidence level.

[0017] The system may correlate or compare the estimated probability and steering wheel torque with "actual" data / information associated with the capacitive sensing and steering wheel angle / torque obtained from the steering wheel sensors. The system may also use other sensors for correlation, and the examples described herein should not be construed as limiting. The system may determine that the driver 102 may be spoofing the vehicle 100 if the estimated probability and steering wheel torque do not correlate with the actual data / information. In other words, the system may determine that the driver 102 may be spoofing the vehicle 100 if a correlation level value between the estimated probability / steering wheel torque and the actual data / information is less than a predefined threshold.In some aspects, the predefined threshold may be tuned or adjusted based on one or more parameters, including, but not limited to, an operating mode of the vehicle, known noise factors (e.g., the TOF camera may detect multipath interference or a low signal-to-noise ratio due to sun glare), and / or the like. In response to such a determination, the system may issue a notification to the vehicle HMI and / or user device requesting the driver 102 to hold the steering wheel 104 with their hands (and remove the spoofing device / object 106).

[0018] In some aspects, the system may estimate the driver’s hand position in the vehicle interior portion by generating a digital driver skeleton model (in Fig. 3 as a digital driver skeleton model 306) based on the inputs obtained from the depth sensing sensors. In some aspects, the system may use a partial depth view of the driver 102 obtained from the depth sensing sensors (with views that may be occluded or outside the FOV of the sensors) to generate a skeleton model of the driver in view. One of ordinary skill in the art may appreciate that the confidence level of the positions of various components of skeleton models may depend on the degree of occlusion and the FOV of the depth sensing sensors. In response to generating the driver skeleton model, the system may determine a distance between the driver's hands in the driver skeleton model and a position of the steering wheel 104 in the vehicle interior portion.The system may determine that the driver's hands may be contacting the steering wheel 104 if the determined distance may be less than a predefined distance threshold, and may not be contacting the steering wheel 104 if the determined distance may be greater than the predefined distance threshold.

[0019] Further details of the system are given below in connection with Fig. 2 described.

[0020] The vehicle 100, the driver 102, and the driver spoofing detection system may implement and / or perform operations as described herein in the present disclosure in accordance with the operating manual and safety guidelines. Furthermore, any action taken by the driver 102 should comply with all regulations specific to the location and operation of the vehicle 100 (e.g., federal, state, county, city, etc.). The notifications provided by the vehicle 100 or the driver spoofing system should be treated as suggestions and followed only in accordance with any regulations specific to the location and operation of the vehicle 100.

[0021] Fig. 2 illustrates a block diagram of an exemplary driver behavior detection system 200 according to the present disclosure. During the description of Fig. 2 will be on Fig. 3-5 referred to.

[0022] The system 200 may include a vehicle 202, a user device 204, and one or more servers 206 communicatively coupled to one another via one or more networks 208. The vehicle 202 may be the same as the vehicle 100 described above in connection with Fig. 1. The user device 204 may be associated with the driver 102 and may include, but is not limited to, a mobile phone, a laptop, a computer, a tablet, a wearable device, or any other similar device with communication capabilities. The server(s) 206 may be part of a cloud-based computing infrastructure and associated with a telematics service delivery network (SDN) associated with the vehicle 202 and other vehicles (in Fig. 2 not shown), which may be part of a commercial vehicle fleet. In further aspects, the server(s) 206 may store instructions associated with one or more AI / ML-based algorithms that may enable the vehicle 202 to perform driver behavior detection, as described in detail below. The server(s) 206 may transmit the instructions associated with the AI / ML-based algorithms to the vehicle 202 over the network(s) 208 at a predefined frequency or when the vehicle 202 requests the instructions.

[0023] The network(s) 208 illustrate(s) an exemplary communication infrastructure in which the connected devices discussed in various embodiments of this disclosure may communicate. The network(s) 208 may be and / or include the Internet, a private network, a public network, or another configuration operating using any one or more of any known communication protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), Bluetooth ® , BLE ®, Wi-Fi based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard, ultra-wideband (UWB) and cellular technologies such as Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), High-Speed ​​Packet Access (HSPDA), Long-Term Evolution (LTE), Global System for Mobile Communications (GSM) and Fifth Generation (5G), to name just a few.

[0024] The vehicle 202 may include a variety of units, including, among others, a vehicle computer 210, a vehicle control unit (VCU) 212, and a driver spoofing detection system 214 (or system 214). The VCU 212 may include a plurality of electronic control units (ECUs) 216 arranged in communication with the vehicle computer 210.

[0025] The user device 204 may connect to the automotive computer 210 and / or the system 214 via the network 208, which may communicate via one or more wireless connections, and / or may connect using near field communication (NFC) protocols, Bluetooth ® protocols, WiFi, ultra-wideband (UWB), and other possible data connection and sharing techniques directly to the vehicle 202.

[0026] In some aspects, the automotive computer 210 and / or the system 214 according to the disclosure may be installed in an engine compartment of the vehicle (or elsewhere in the vehicle 202). Further, the automotive computer 210 may operate as a functional component of the system 214. The automotive computer 210 may be or include an electronic vehicle controller having one or more processors 218 and a memory 220. Furthermore, the system 214 may be separate from the automotive computer 210 (as in Fig. 2) or integrated as part of the vehicle computer 210.

[0027] The processor(s) 218 ​​may be arranged in communication with one or more storage devices arranged in communication with the respective computing systems (e.g., the memory 220 and / or one or more external databases located in Fig. 2 are not shown). The processor(s) 218 ​​may utilize the memory 220 to store programs in code and / or to store data for performing aspects according to the disclosure. The memory 220 may be a non-transitory computer-readable storage medium or a non-transitory computer-readable memory in which program code for driver spoofing detection is stored. The memory 220 may include any one or a combination of volatile memory elements (e.g., dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.) and any one or more non-volatile memory elements (e.g.,erasable programmable read-only memory (EPROM), flash memory, electronically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), etc.).

[0028] In some aspects, the VCU 212 may share a power bus with the automotive computer 210 and may be configured and / or programmed to transfer data between vehicle systems, connected servers (e.g., the server(s) 206), and other vehicles (in Fig. 2 not shown) operating as part of a vehicle fleet. The VCU 212 may include or communicate with any combination of the ECU 216, such as a body control module (BCM) 222, an engine control module (ECM) 224, a transmission control module (TCM) 226, a telematics control unit (TCU) 228, a driver assistance technologies (DAT) controller 230, etc. The VCU 212 may further include and / or communicate with a vehicle perception system (VPS) 232 that has connectivity to and / or controls one or more vehicle sensor systems 234.The vehicle sensor system 234 may include one or more vehicle sensors, including, but not limited to, a radio-based positioning and ranging (RADAR) sensor configured to detect and locate objects inside and outside the vehicle 202 using radio waves, seat belt buckle sensors, seat area sensors, an optical distance and speed measurement (LIDAR) sensor, door sensors, proximity sensors, temperature sensors, wheel sensors, ambient weather sensors, interior and exterior vehicle cameras, steering wheel sensors, an inertial measurements unit (IMU), etc.

[0029] In some aspects, the VCU 212 may control operational aspects of the vehicle and implement one or more sets of instructions received from the user device 204 from one or more sets of instructions stored on the memory 220, including instructions that serve as part of the system 214.

[0030] The TCU 228 may be configured and / or programmed to provide vehicle connectivity with wireless computing systems inside and outside the vehicle 202, and may include a navigation receiver (NAV receiver) 236 for receiving and processing a GPS signal, a BLE ® -Module (BLEM) 238, a WiFi transceiver, a UWB transceiver and / or other wireless transceivers (in Fig. 2 not shown) used for wireless communication (including cellular communication) between the vehicle 202 and other systems (e.g., a vehicle remote key, in Fig. 2 not shown), computers, and modules. The TCU 228 may be arranged in communication with the ECUs 216 via a bus.

[0031] The ECUs 216 may control aspects of vehicle operation and communication using inputs from human drivers, inputs from the automotive computer 210, the system 214, and / or via wireless signal inputs received over the wireless connection(s) from other connected devices, such as the user device 204, the server(s) 206, among others.

[0032] The BCM 222 generally includes an integration of sensors, vehicle performance indicators, and variable throttles associated with vehicle systems, and may include processor-based power distribution circuits that can control functions associated with the vehicle body, such as lights, windows, security, camera(s), audio system(s), speakers, windshield wipers, door locks and access control, and various convenience controls. The BCM 222 may also operate as a gateway for bus and network interfaces to communicate with remote ECUs (in Fig. 2 not shown).

[0033] The DAT controller 230 may provide Level 1 to Level 3 automated driving and driver assistance functionality, which may include, for example, active parking assistance, vehicle backup assistance, and adaptive cruise control, among other features. The DAT controller 230 may also provide aspects of user and environmental input usable for user authentication.

[0034] In some aspects, the automotive computer 210 may connect to an infotainment system 240. The infotainment system 240 may include a touchscreen interface portion and may include voice recognition features, biometric identification capabilities that may identify users based on facial recognition, voice recognition, fingerprint identification, or other biological identification means. In other aspects, the infotainment system 240 may be further configured to receive user instructions via the touchscreen interface portion and / or output or display notifications (including visual and / or audible alert notifications), navigation maps, etc., on the touchscreen interface portion.

[0035] The vehicle 202 may further include a first detection unit 242 and a second detection unit 244. The first detection unit 242 may include a steering wheel touch sensor (e.g., a capacitive touch sensor) and / or a steering wheel torque or rotation angle sensor. The first detection unit 242 may be configured to detect a first parameter associated with the steering wheel 104. In an example aspect, the first parameter may be associated with the capacitive sensing of the steering wheel touch sensor and / or a steering wheel torque or steering wheel rotation angle. In some aspects, the first detection unit 242 may be part of the vehicle sensor system 234. In other aspects, the first detection unit 242 may be separate from the vehicle sensor system 234.

[0036] The second detection unit 244 may be configured to receive one or more inputs associated with the driver 102 when the driver 102 may be located within the vehicle interior portion. The second detection unit 244 may include, among other things, one or more depth sensors, radar sensors, lidar sensors, TOF cameras, and / or the like. The inputs received by the second detection unit 244 may include, among other things, a depth image, a plurality of voxels, a point cloud, and / or the like associated with the driver 102 when the driver 102 may be located within the vehicle interior portion. In some aspects, the inputs may include depth data / information as a depth map / depth image, a point cloud, or an occupancy voxel directly or after processing. Further, the inputs may include information about a reflectivity of the object (e.g., return signal intensity).In some aspects, the depth information may be determined using direct or indirect time-of-flight principles. Depending on the capabilities of the sensors included in the second detection unit 244, a velocity estimate may be generated, which may be compared to inputs obtained from one or more vehicle sensors. For example, a radar may estimate depth and relative radial velocity, which may be used to estimate a velocity vector of a driver's hands / arms, which may be compared to the rotation rate of the steering wheel 104 (as one example).

[0037] In some aspects, the second detection unit 244 may be part of the vehicle sensor system 234. In other aspects, the second detection unit 244 may be separate from the vehicle sensor system 234.

[0038] The computing system architecture of the vehicle computer 210, the VCU 212 and / or the system 214 may omit certain computing modules. It is readily understood that the Fig. 2 is an example of a possible implementation according to the present disclosure and thus should not be considered limiting or exclusive.

[0039] In some aspects, system 214 may be integrated with and / or implemented as part of ECU 216. Whether integrated with vehicle computer 210 or ECU 216 or operating as a standalone computing system within vehicle 202, system 214 may include a transceiver 246, a processor 248, and computer-readable memory 250.

[0040] The transceiver 246 may be configured to receive information / inputs from one or more external devices or systems, e.g., the user device 204, the server(s) 206, and / or the like, via the network 208. For example, the transceiver 246 may be configured to receive instructions associated with one or more AI / ML-based algorithms from the server(s) 206, as described above. Further, the transceiver 246 may transmit notifications or signals to the external devices or systems. Additionally, the transceiver 246 may be configured to receive information / inputs from vehicle components, such as the infotainment system 240, the first detection unit 242, the second detection unit 244, and / or the like. Further, the transceiver 246 may receive signals (e.g.,Command signals) or notifications to the vehicle components, such as the infotainment system 240.

[0041] The processor 248 and the memory 250 may be the same as or similar to the processor 218 and the memory 220, respectively. In some aspects, the processor 248 may utilize the memory 250 to store programs in code and / or store data for performing aspects of the disclosure. The memory 250 may be a non-transitory computer-readable storage medium or a non-transitory computer-readable memory in which the driver spoofing detection program code is stored. In some aspects, the memory 250 may additionally store instructions / information / data obtained from the server(s) 206, the user device 204, and the first and second detection units 242 and 244.

[0042] In some aspects, the processor 248 may be an AI / ML-based processor that may be configured to execute the instructions associated with the AI / ML-based algorithms stored on the memory 250 (as acquired from the server(s) 206) to determine the location / position of the driver's hands in the vehicle interior portion, generate a digital driver skeleton model 306 (as in Fig. 3) and / or a probability that the driver's hands are contacting the steering wheel 104, and estimate the expected steering wheel rotation angle / torque based on the inputs obtained from the second detection unit 244. In some aspects, the processor 248 may perform the functions / actions described herein by using supervised or reinforcement machine learning and generating a "trained" machine learning module. In response to generating the machine learning module, the processor 248 may store the module on the memory 250 and use the instructions stored in the module to detect a possible driver steering wheel spoofing action when the driver 102 is driving the vehicle 100.

[0043] One of ordinary skill in the art can recognize that machine learning is an application of artificial intelligence (AI) in which the processor 248 may have the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on using data and algorithms to mimic the way humans learn. In some aspects, the machine learning algorithms may be generated to create classifications and / or predictions. Machine learning-based systems may be used for a variety of applications, including, but not limited to, speech recognition, image or video processing, statistical analysis, probabilistic prediction, and / or the like.

[0044] Machine learning can be of various types based on the data or signals available to the learning system. For example, the machine learning approach can include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Supervised learning is an approach that can be supervised by a human. In this approach, the machine learning algorithm can use labeled training data and defined variables. In the case of supervised learning, both the input and output of the algorithm can be predetermined / defined, and the algorithms can be trained to classify data and / or accurately predict outcomes.In the present disclosure, since the processor 248 (or the "AI / ML"-based processor) performs functions or makes predictions / estimates based on inputs obtained from the depth sensing sensors (where the inputs may include noise), in some aspects the system 214 may further include filters, such as a Kalman filter, particle filters, etc., and the processor 248 may obtain "filtered" inputs via the filters to perform the functions or make the predictions / estimations.

[0045] Broadly speaking, supervised learning can fall into one of two types: regression and classification. In classification learning, the learning algorithm can assist in dividing the dataset into classes based on different parameters. In this case, a computer program can be trained on the training dataset, and the computer program can categorize input data into different classes based on the training. Some well-known methods used in classification learning include logistic regression, k-nearest neighbors, support vector machines (SVM), kernel SVM, naive Bayes, decision tree classification, and random forest classification.

[0046] In regression learning, the learning algorithm can predict an output value that can be continuous or a real-world value. Some common methods used in regression learning include simple linear regression, multiple linear regression, polynomial regression, support vector regression, decision tree regression, and random forest regression.

[0047] Unsupervised learning is an approach that involves algorithms that can be trained with unlabeled data. An unsupervised learning algorithm can analyze the data itself and find patterns in input data. Furthermore, semi-supervised learning is a combination of supervised learning and unsupervised learning. A semi-supervised learning algorithm involves labeled training data; however, the semi-supervised learning algorithm can still find patterns in the input data. Reinforcement learning is a multi-stage or dynamic process. This model is similar to supervised learning but cannot be trained using sample data. This model can learn "as it runs" using trial and error. In reinforcement learning, a sequence of successful outcomes can be reinforced to develop the best recommendation or policy for a given situation.As described above, in some aspects, the processor 248 may perform the functions / acts described in the present disclosure using reinforcement machine learning.

[0048] In some aspects, to generate the trained machine learning module, the processor 248 may first "monitor" the driver's movement (e.g., a hand gesture, upper body movement, eye movement) within the vehicle 100 when the likelihood of spoofing may be low (e.g., when the driver 102 enters the vehicle 100, turns on the ignition, or operates the infotainment system 240). During this period, based on the inputs obtained from the second detection unit 244 and the instructions associated with the AI / ML-based algorithms stored on the memory 250, the processor 248 may estimate the location / position of the driver's hands, generate the digital driver skeleton model, estimate the likelihood of the driver's hands touching respective vehicle components (e.g., the infotainment system 240 or a vehicle ignition actuator), and / or the like. The processor 248 can then process “actual” data (e.g.When the driver 102 uses the infotainment system 240, the infotainment system 240 may send a driver interaction confirmation signal to the processor 248) with the estimated data described above to calibrate the machine learning module.

[0049] The processor 248 may also transmit a request (e.g., via the infotainment system 240 or the user device 204) to the driver 102 to perform a predefined action, e.g., to turn the steering wheel 104 left or right at a requested angle. When the driver 102 turns the steering wheel 104 in response to receiving the request, the processor 248 may estimate the steering wheel rotation angle / torque based on the inputs obtained from the second detection unit 244 and compare it to the rotation angle requested by the processor 248 or determined using inputs obtained from the first detection unit 242. Based on the comparison, the processor 248 may calibrate the machine learning module until the error in the detection is less than a predefined threshold. In this way, the processor 248 may "train" the machine learning module.Once the detection error is less than the predefined threshold, or when the processor 248 can have a "confidence" (e.g., a confidence level above a predefined confidence threshold) that the detection / estimation of driver hand position, probability, etc., described above, may be high, the processor 248 may treat the calibrated machine learning module as a "trained" machine learning module and store the trained machine learning module in the memory 250. The processor 248 may continue to update or improve the trained machine learning module based on new data, information, and insights.

[0050] In some aspects, during training of the machine learning module, the processor 248 may also consider one or more noise factors that may be present in the inputs acquired by the second detection unit 244, or interference / variances / inconsistencies in the inputs due to a driver's seating area position, driver size (e.g., the second detection unit 244 may only be able to accommodate the driver's head as opposed to the driver's head, shoulders, and / or upper arms), varying hand positions while driving or driving styles, etc. The processor 248 may also consider effects of the rate of change of vehicle speed on the estimated steering wheel torque while training the machine learning module.

[0051] In some aspects, processor 248 may additionally perform preprocessing to test or better interact with the comparison / correlation (described above), e.g., a Pearson correlation coefficient, Spearman rank correlation coefficient, and / or the like. Experimental data may be used to determine an appropriate threshold for the correlation coefficients described above. Edge cases, such as a driver driving with their chin or knees propped up, or a passenger driving the vehicle, etc., may also be considered in the training process.

[0052] Once the machine learning module is potentially trained, the processor 248 may use the module during vehicle operation, as described below.

[0053] In operation, the processor 248 may obtain data (or a "first parameter") associated with the capacitive sensing of the steering wheel touch sensor and / or the steering wheel torque or the steering wheel rotation angle from the first detection unit 242 when the driver 102 may be driving the vehicle 202. Example data / output obtained from the steering wheel touch sensor is shown in graphs 402, 502, and 504 in Fig. 4 and 5, respectively. In each of graphs 402, 502, and 504, the y-axis represents the output from the steering wheel touch sensor as the value "0" or "1," and the x-axis represents time. An output of "0" from the steering wheel touch sensor indicates that the first detection unit 242 detects no touch on the steering wheel 104, and an output of "1" indicates that the first detection unit 242 detects a touch on the steering wheel 104.

[0054] Furthermore, an exemplary steering wheel torque obtained by the first detection unit 242 is shown as graphs 404, 506 and 508 in Fig. 4 and 5, respectively. In each of graphs 404, 506, and 508, the y-axis represents the measured torque and the x-axis represents time.

[0055] In addition to obtaining the data from the first detection unit 242, the processor 248 may obtain the inputs from the second detection unit 244 described above. For example, the processor 248 may obtain depth images of the driver 102 in the vehicle interior portion. The processor 248 may use the obtained data and inputs to detect whether the driver 102 may be spoofing the vehicle 202 or the steering wheel 104. In some aspects, the processor 248 may obtain the data and inputs described herein to detect a possible spoofing action by the driver when a field of view (FOV) of the vehicle's interior camera may be blocked and the vehicle's interior camera may not be able to capture images of the driver's hands on the steering wheel 104, as in view 302 of Fig. 3. As shown in view 302, the FOV of the vehicle's interior camera may be obscured by a structure / object 304 (which may be part of the vehicle 202 or an external object / structure attached to the vehicle's interior portion). In this case, it may not be possible to determine whether or not the driver's hands are on the steering wheel 104 based on the images acquired from the vehicle's interior camera. In such cases, the processor 248 may use the data and inputs from the first and second detection units 242, 244 to detect whether the driver's hands are actually on the steering wheel 104 or whether the driver 102 may be spoofing the vehicle 202 or the steering wheel 104.

[0056] In response to obtaining the data and inputs from the first and second detection units 242, 244, the processor 248 may use the above-described trained machine learning module (or an end-to-end neural network model) to estimate a location of the driver's hands within the vehicle interior portion based on the inputs obtained from the second detection unit 244. In other words, the processor 248 may use the depth images of the driver 102 within the vehicle interior portion and the trained machine learning module to estimate the location of the driver's hands within the vehicle interior portion.

[0057] In some aspects, the processor 248 may additionally or alternatively estimate the location of the driver's hands in the vehicle interior portion by generating the digital driver skeleton model 306 using the depth image acquired by the second detection unit 244 and one or more known image processing algorithms / processes for generating 3-dimensional models based on 2-dimensional images. Examples of such image processing algorithms / processes include “Occlusion-tolerant and personalized 3D human pose estimation in RGB images” by A. Qammaz and A. Argyros (2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 6904-6911, doi: 10.1109 / ICPR48806.2021.9411956), “A Joint Model for 2D and 3D Pose Estimation from a Single Image” by Simo-Serra, Edgar et al.(2013 IEEE Conference on Computer Vision and Pattern Recognition (2013): 3634-3641) and “Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans” by Side Peng et al. (CVPR 2021), which are incorporated by reference into this paper.

[0058] In response to estimating the location of the driver's hands within the vehicle interior portion (using the trained machine learning module or by generating the driver's digital skeleton model 306), the processor 248 may estimate a probability that the driver's hands are contacting the steering wheel 104 and a steering wheel torque / angle (collectively referred to as a "second parameter" associated with the driver's interaction with the steering wheel 104) based on the estimated location of the driver's hands. In one example aspect, the processor 248 may estimate the probability that the driver's hands are contacting the steering wheel 104 by calculating a distance between the estimated location of the driver's hands within the vehicle interior portion and the position of the steering wheel 104 (which may be pre-stored on the memory 250) within the vehicle interior portion.An exemplary view of the driver's hands 308 near the steering wheel 104 is shown as view 310 in FIG. Fig. 3. If the calculated distance may be less than a first predefined distance threshold, the processor 248 may estimate that the probability that the driver's hands are touching the steering wheel 104 may be high. On the other hand, if the calculated distance may be greater than a second predefined distance threshold (which may be the same as or different from the first predefined distance threshold), the processor 248 may estimate that the probability that the driver's hands are touching the steering wheel 104 may be low.

[0059] Example estimated probabilities associated with the driver's hands 308 touching the steering wheel 104 are shown as graphs 312, 406 and 510 in Fig. 3, Fig. 4 and 5, respectively. In each of graphs 312, 406, and 510, the y-axis represents the estimated probabilities and the x-axis represents time.

[0060] Similarly, processor 248 may estimate the steering wheel torque by tracking the driver's hand movement within the vehicle interior portion over a period of time based on the estimated location of the driver's hands 308 within the vehicle interior portion. Example estimated steering wheel torques over time are shown as graphs 314, 408, and 512 in Fig. 3, Fig. 4 and 5, respectively. In each of graphs 314, 408, and 512, the y-axis represents the estimated steering wheel torque and the x-axis represents time.

[0061] In response to estimating the second parameter, processor 248 may correlate the estimated second parameter with the data / first parameter obtained by first detection unit 242 and, based on the correlation, determine whether a predefined condition may be met. In response to determining that the predefined condition may be met, processor 248 may determine that driver 102 may be spoofing vehicle 100 or steering wheel 104.

[0062] In an exemplary first aspect, the processor 248 may correlate the first and second parameters by comparing the "actual" steering wheel torque, as obtained by the first detection unit 242, with the "estimated" steering wheel torque described above and calculating a difference (e.g., an average difference over a predefined period of time) between the actual and the estimated steering wheel torque. If the calculated difference is greater than a predefined difference threshold, the processor 248 may determine that the predefined condition may be met and that the driver 102 may be spoofing the vehicle 100 or the steering wheel 104. As in Fig. 5, the pattern of estimated steering wheel torque over time in graph 512 differs substantially from the pattern of "actual" steering wheel torques over time depicted in graphs 506 and 508. In this case, processor 248 may determine that the difference between the estimated and actual steering wheel torque may be greater than the predefined difference threshold and may thus determine that driver 102 may be spoofing vehicle 100 / steering wheel 104. In one example aspect, graph 506 may be associated with a case where driver 102 may have placed object 106 on steering wheel 104, and graph 508 may be associated with a case where driver 102 may have used an electromechanical or electromechanical spoofing device.

[0063] On the other hand, the pattern of estimated steering wheel torque over time in graph 408 is similar to the pattern of actual steering wheel torque over time depicted in graph 404. In this case, processor 248 may determine that the difference between the estimated and actual steering wheel torque may be less than the predefined difference threshold, and thus may determine that driver 102 may not be spoofing vehicle 100 / steering wheel 104. In other words, processor 248 may determine that driver 102 may be steering / maneuvering steering wheel 104 using their hands.

[0064] In an exemplary second aspect, the processor 248 may correlate the first and second parameters by identifying a similarity or "correlation degree value" between the estimated probability associated with the driver's hands 308 touching the steering wheel 104 and the actual capacitive sensing data obtained by the first detection unit 242. In this case, the processor 248 may compare the estimated probability to the capacitive sensing data and identify a difference (e.g., an average difference over a predefined period of time) between the estimated probability and the capacitive sensing data.In one example aspect, an output of "1" (shown in graphs 402, 502, and 504) for capacitive sensing data may be treated by processor 248 as a true 100% probability that driver 102 may touch steering wheel 104.

[0065] In some aspects, the processor 248 may determine that the correlation degree value between the first and second parameters may be high when the difference between the estimated and actual probability described above may be low. As in Fig. 4, for example, between times "T1" and "T2," a difference between the estimated probability shown in graph 406 and the actual probability shown in graph 402 may be small. Within this time period, processor 248 may determine that the correlation degree value between the first and second parameters may be high. In response to such a determination, processor 248 may determine that driver 102 may not be spoofing vehicle 100 / steering wheel 104 and may be using their hands to control steering wheel 104.

[0066] On the other hand, the processor 248 may determine that the correlation degree value between the first and second parameters may be low (e.g., less than a threshold) if the difference between the estimated and actual probability described above may be high. As in Fig. 5, for example, the difference between the estimated probability shown in graph 510 and the actual probabilities shown in graphs 502 and 504 may be high. In response to determining that the difference may be high, processor 248 may determine that the correlation degree value between the first and second parameters may be low. In response to such a determination, processor 248 may determine that the predefined condition may be met and driver 102 may be spoofing vehicle 100 / steering wheel 104. In one example aspect, graphs 502 and 504 may be associated with instances where driver 102 may use object 106 or an electromechanical / electromagnetic spoofing device for spoofing.

[0067] In response to determining that the predefined condition may be met (i.e., the driver 102 may be spoofing), the processor 248 may perform a predefined action. For example, in response to determining that the predefined condition may be met, the processor 248 may transmit a notification to the infotainment system 240 and / or the user device 204 via the transceiver 246. The notification may include an audio and / or video notification or a request from the driver 102 to steer or hold the steering wheel 104 using their hands and / or to remove the spoofing device / object 106.

[0068] Although the foregoing description describes one aspect of detecting whether the driver 102 may be spoofing, those of ordinary skill in the art will recognize that the present disclosure may be applied to other applications, such as determining whether the driver 102 may be ready to assume control of the vehicle 202 (e.g., whether the driver 102 has one hand or both hands on the steering wheel 104).

[0069] Fig. 6 illustrates a flowchart of an exemplary method 600 for detecting driver behavior in the vehicle 100 according to the present disclosure. Fig. 6 may be described with continued reference to previous figures. The following process is exemplary and not limited to the steps described below. Furthermore, alternative embodiments may include more or fewer steps than shown or described herein, and may include these steps in an order that differs from the order described in the following exemplary embodiments.

[0070] The method 600 begins at step 602. At step 604, the method 600 may include obtaining the first parameter from the first detection unit 242 and the input from the second detection unit 244 by the processor 248. Examples of the first parameters and the input are described above in connection with Fig. 2 described.

[0071] At step 606, the method 600 may include estimating, by the processor 248, the second parameter associated with a driver interaction with the steering wheel 104 based on the input obtained from the second detection unit 244. At step 608, the method 600 may include correlating, by the processor 248, the first parameter and the second parameter. At step 610, the method 600 may include determining, by the processor 248, that the predefined condition may be met based on the correlation. At step 612, the method 600 may include transmitting a notification to the infotainment system 240 and / or the user device 204 in response to determining that the predefined condition may be met.

[0072] The method 600 may end at step 614.

[0073] In the foregoing disclosure, reference has been made to the accompanying drawings, which form a part hereof, and illustrate specific implementations in which the present disclosure may be practiced. It is understood that other implementations may be utilized and structural changes may be made without departing from the scope of the present disclosure. References in the specification to "one embodiment," "an embodiment," etc., indicate that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Further, such phrases are not necessarily all referring to the same embodiment.Furthermore, if a feature, structure, or characteristic is described in connection with one embodiment, those skilled in the art will recognize such a feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0074] Furthermore, the functions described in this document may be performed in one or more of hardware, software, firmware, digital components, or analog components, as appropriate. For example, one or more application-specific integrated circuits (ASICs) may be programmed to perform one or more of the systems and procedures described in this document. Certain terms used throughout the specification and claims refer to specific system components. It will be apparent to one skilled in the art that the components may be referred to by other names. This document does not intend to distinguish between components that differ in name but not in function.

[0075] It is also to be understood that the word "example" as used in this specification is intended to be non-exclusive and non-limiting. In particular, the word "example" as used in this specification indicates one of several examples, and it is to be understood that no undue emphasis or preference is placed on the particular example described.

[0076] A computer-readable medium (also referred to as a processor-readable medium) includes any non-transitory (e.g., physical) medium that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-transitory media and volatile media. Computing devices may include computer-executable instructions, where the instructions may be executable by one or more computing devices, such as those listed above, and may be stored on a computer-readable medium.

[0077] With respect to the processes, systems, methods, heuristics, etc. described in this specification, it is to be understood that although the steps of such processes, etc., have been described as occurring according to a certain ordered sequence, such processes could be implemented in practice such that the described steps are performed in an order that differs from the order described in this specification. Furthermore, it is to be understood that certain steps could be performed concurrently, that other steps could be added, or that certain steps described in this specification could be omitted. In other words, the descriptions of processes in this specification are for the purpose of illustrating various embodiments and should in no way be construed to limit the claims.

[0078] Accordingly, it is to be understood that the foregoing description is intended to be illustrative and not restrictive. From a reading of the foregoing description, many other embodiments and applications than the examples provided will become apparent. The scope should be determined not by reference to the foregoing description, but instead by reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that there will be future developments in the art discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. Overall, it is to be understood that the application is susceptible of modification and variation.

[0079] All terms used in the claims are intended to have their general meaning as known to one of ordinary skill in the art to which the technologies described in this specification apply, unless expressly stated otherwise herein. In particular, the use of the singular articles, such as "a," "an," "the," "the," "the," etc., is intended to refer to one or more of the specified elements, unless a claim expressly limits otherwise.Conditional language, such as, but not limited to, "may," "could," "may," or "may under certain circumstances," is generally intended to convey that certain embodiments may include certain features, elements, and / or steps, whereas other embodiments may not include them, unless specifically stated otherwise or clear from the particular context used. Thus, such conditional language is generally not intended to imply that features, elements, and / or steps are in any way required for one or more embodiments.

[0080] In one aspect of the invention, the input comprises at least one of a depth image, a plurality of voxels, and a point cloud.

[0081] In one aspect of the invention, the second parameter is at least one of a probability that hands of a vehicle occupant touch the steering wheel and a steering wheel torque.

[0082] In one aspect of the invention, the method includes: estimating a location of hands of a vehicle occupant in a vehicle interior portion based on the input; and estimating the probability that the vehicle occupant's hands contact the steering wheel and the steering wheel torque based on the location of the vehicle occupant's hands in the vehicle interior portion.

[0083] In one aspect of the invention, the method includes: comparing the steering wheel torque with the first parameter; and determining that the predefined condition is met based on the comparison.

[0084] According to the present invention, a non-transitory computer-readable storage medium is provided having stored thereon instructions that, when executed by a processor, cause the processor to: obtain a first parameter from a first detection unit and an input from a second detection unit, wherein the first detection unit is configured to detect the first parameter associated with a steering wheel, and wherein the second detection unit is configured to receive the input associated with a vehicle occupant; estimate a second parameter associated with a vehicle occupant interaction with the steering wheel based on the input; correlate the first parameter and the second parameter; determine that a predefined condition is met based on the correlation;and transmitting a notification in response to a determination that the predefined condition is met.; QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Occlusion-tolerant and personalized 3D human pose estimation in RGB images” by A. Qammaz and A. Argyros (2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 6904-6911, doi: 10.1109 / ICPR48806.2021.9411956

[0057] A Joint Model for 2D and 3D Pose Estimation from a Single Image“ von Simo-Serra, Edgar et al. (2013 IEEE Conference on Computer Vision and Pattern Recognition (2013): 3634-3641

[0057] Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans“ von Side Peng et al. (CVPR 2021

[0057]

Claims

[1] Vehicle comprising: a steering wheel; a first detection unit configured to detect a first parameter associated with the steering wheel; a second detection unit configured to receive an input associated with a vehicle occupant; a processor communicatively coupled to the steering wheel, the first detection unit, and the second detection unit, the processor being configured to: Obtained the input and the first parameter; estimating a second parameter associated with a vehicle occupant interaction with the steering wheel based on the input; Correlating the first parameter and the second parameter; Determining that a predefined condition is met based on the correlation; and Transmit a notification in response to a determination that the predefined condition is met. [2] The vehicle of claim 1, wherein the first detection unit is a steering wheel touch sensor and wherein the first parameter is associated with a capacitive sensing of the steering wheel touch sensor. [3] The vehicle of claim 1, wherein the first detection unit is a steering wheel torque sensor and wherein the first parameter is associated with a steering wheel torque or steering wheel rotation angle. [4] The vehicle according to claim 1, wherein the second detection unit is at least one of a depth sensing sensor, a radar sensor, a time-of-flight camera, and a lidar sensor. [5] The vehicle of claim 1, wherein the input comprises at least one of a depth image, a plurality of voxels, and a point cloud. [6] The vehicle of claim 1, wherein the second parameter is at least one of a probability that hands of a vehicle occupant touch the steering wheel and a steering wheel torque. [7] The vehicle of claim 6, wherein the processor is further configured to: Estimating a location of hands of a vehicle occupant in a vehicle interior portion based on the input; and Estimating the probability that the vehicle occupant’s hands touch the steering wheel, and the steering wheel torque based on the position of the vehicle occupant's hands in the vehicle interior section. [8] The vehicle of claim 7, wherein the processor is further configured to: Comparing the steering wheel torque with the first parameter; and Determine that the predefined condition is met based on the comparison. [9] The vehicle of claim 1, wherein the processor is further configured to: Generating a digital vehicle occupant skeleton model based on the input; and Estimating the second parameter based on the digital vehicle occupant skeleton model. [10] The vehicle according to claim 1, wherein the predefined condition is satisfied when a correlation degree value between the first parameter and the second parameter is less than a threshold value. [11] The vehicle of claim 1, wherein the processor transmits the notification to a user device or a human-machine interface of the vehicle. [12] A method for detecting driver behavior in a vehicle, the method comprising: Obtaining a first parameter from a first detection unit and an input from a second detection unit by a processor, wherein the first detection unit is configured to detect the first parameter associated with a steering wheel, and wherein the second detection unit is configured to receive the input associated with a vehicle occupant; estimating, by the processor, a second parameter associated with a vehicle occupant interaction with the steering wheel based on the input; Correlating the first parameter and the second parameter by the processor; Determining that a predefined condition is met by the processor based on the correlation; and Transmitting a notification by the processor in response to a determination that the predefined condition is met. [13] The method of claim 12, wherein the first detection unit is a steering wheel touch sensor and wherein the first parameter is associated with a capacitive sensing of the steering wheel touch sensor. [14] The method of claim 12, wherein the first detection unit is a steering wheel torque sensor and wherein the first parameter is associated with a steering wheel torque or steering wheel rotation angle. [15] The method of claim 12, wherein the second detection unit is at least one of a depth sensing sensor, a sensor for radio-based positioning and distance measurement (radar), a time-of-flight camera (TOF camera) and a sensor for optical distance and speed measurement (lidar).

Images