System and method for detecting driver behavior

Abstract

The present disclosure provides a system and method for detecting driver behavior. A vehicle includes a steering wheel, a first detection unit, a second detection unit, and a processor. The first detection unit may be configured to detect a first parameter associated with the steering wheel, and the second detection unit may be configured to capture an input associated with a vehicle occupant. The processor may be configured to obtain the input and the first parameter, and estimate a second parameter associated with an interaction of a vehicle occupant and the steering wheel based on the input. The processor may also be configured to correlate the first parameter and the second parameter, and determine that a predefined condition may be satisfied based on the correlation. The processor may also transmit a notification in response to determining that the predefined condition is satisfied.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for detecting driver behavior, and more particularly to systems and methods for detecting driver steering spoofing based on depth sensing. Background Art

[0002] Many modern vehicles include features that assist drivers in conveniently driving their vehicles. For example, vehicles include features that assist drivers during lane changes, reversing the vehicle, navigating traffic, etc. Modern vehicles also include features for detecting driver behavior in the vehicle, including driver inattention. Upon detecting that the driver may be inattentive, the vehicle can warn the driver to pay attention. There are situations where drivers spoof vehicle features that detect the driver's behavior in the vehicle. A system is needed to detect such situations. Summary of the invention

[0003] The present disclosure describes a vehicle configured to detect whether a vehicle driver may be deceiving the vehicle or the steering wheel. Specifically, the vehicle may be configured to detect whether the driver's hands are actually on the steering wheel, or whether the driver may be using an external object or a deception device to deceive the steering wheel. The vehicle may include a driver deception detection system ("system"), which may obtain input from a first detection unit and a second detection unit of the vehicle, and may determine that the driver may be deceiving the steering wheel based on the input obtained. The first detection unit may be, for example, one or more steering wheel sensors, and the second detection unit may be, for example, one or more vehicle interior time of flight (TOF) cameras, radio detection and ranging (radar) sensors, light detection and ranging (lidar) sensors, etc. In some aspects, when the driver's hands near the steering wheel may not be visible in the image captured by the vehicle interior camera, the system may determine that the driver may be deceiving the steering wheel based on the input obtained from the first detection unit and the second detection unit, but the driver's upper body portion may be visible in the depth image / map obtained from the second detection unit.

[0004] In some aspects, in order to determine possible driver's deceptive activity, the system may first estimate the position of the driver's hand in the interior part of the vehicle based on the input obtained from the second detection unit. In an exemplary aspect, the system may estimate the position of the driver's hand in the interior part of the vehicle by generating a digital driver skeleton model based on one or more artificial intelligence / machine learning-based image processing algorithms. In response to estimating the position of the driver's hand, the system may estimate the probability and steering wheel torque / rotation angle of the driver's hand touching the steering wheel based on the estimated driver's hand position. Then, the system may associate the estimated probability / steering wheel torque with the input obtained from the first detection unit. The system may determine that the driver may be deceiving the steering wheel based on the correlation. Specifically, the system may determine that the driver's hand may not actually be on the steering wheel based on the correlation. For example, when the correlation between the estimated probability / steering wheel torque and the input obtained from the first detection unit may be low (e.g., less than a predefined threshold), the system may determine that the driver's hand may not be on the steering wheel.

[0005] In response to determining that the driver may be deceiving the steering wheel, the system may transmit a notification to the vehicle HMI and / or the user device requesting the driver to take a hand of the steering wheel and remove the deceiving device / external object.

[0006] The present disclosure discloses a vehicle that detects whether a driver may be cheating on a steering wheel. When the vehicle detects that the driver may be cheating, the system may request the driver to use his hands to hold the steering wheel. In addition, the vehicle uses existing vehicle hardware and components to perform the detection, and therefore does not require any external hardware components to operate according to the present disclosure.

[0007] These and other advantages of the present disclosure are provided in detail herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Specific embodiments are described with reference to the accompanying drawings. The use of the same reference numerals may indicate similar or identical items. Various embodiments may utilize elements and / or components other than those shown in the accompanying drawings, and some elements and / or components may not be present in various embodiments. The elements and / or components in the accompanying drawings are not necessarily drawn to scale. Throughout this disclosure, singular terms and plural terms may be used interchangeably, depending on the context.

[0009] Figure 1 An exemplary vehicle is depicted in which techniques and structure for providing the systems and methods disclosed herein may be implemented.

[0010] Figure 2 A block diagram of an exemplary driver behavior detection system according to the present disclosure is depicted.

[0011] Figure 3 Depicted are an exemplary driver image, a driver skeletal model, and exemplary steering wheel touch probability and steering wheel torque according to the present disclosure.

[0012] Figure 4 Depicted is a first exemplary estimated parameter associated with driver and steering wheel interaction and input obtained from one or more vehicle sensors in accordance with the present disclosure.

[0013] Figure 5 Depicted is a second exemplary estimated parameter associated with driver and steering wheel interaction and input obtained from one or more vehicle sensors in accordance with the present disclosure.

[0014] Figure 6 A flow chart is depicted of an exemplary method for detecting driver behavior in a vehicle according to the present disclosure. DETAILED DESCRIPTION

[0015] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the disclosure are shown and which are not intended to be limiting.

[0016] Figure 1 An exemplary vehicle 100 is depicted in which techniques and structures for providing the systems and methods disclosed herein may be implemented. The vehicle 100 may take the form of any passenger or commercial vehicle, such as, for example, a sedan, a work vehicle, a crossover vehicle, a truck, a van, a minivan, a taxi, a bus, etc. The vehicle 100 may be a manually driven vehicle, and / or may be configured to operate in a partially autonomous mode, and may include any powertrain, such as, for example, a gasoline engine, one or more electrically actuated motors, a hybrid powertrain, etc. A vehicle occupant / user 102 (or driver 102) may be driving the vehicle 100. In an exemplary aspect, the driver 102 may drive the vehicle 100 by using and manipulating a vehicle steering wheel 104 (or steering wheel 104).

[0017] In some aspects, the vehicle 100 may include one or more steering wheel sensors (in Figure 2 104 ), the one or more steering wheel sensors may be configured to detect a touch or interaction of the driver's hand with the steering wheel 104. For example, the steering wheel sensor may include a capacitive touch sensor and / or a steering wheel rotation or torque sensor, which may be configured to detect a touch of the driver's hand with the steering wheel 104 (e.g., using capacitive sensing) and a rotation angle / torque of the steering wheel when the driver 102 rotates the steering wheel 104 (e.g., using a rotation or torque sensor).

[0018] The steering wheel sensor may be configured to continuously (or at a predefined frequency) detect driver's hand interaction, such as touch or movement, associated with the steering wheel 104 when the vehicle 100 may be in motion. The vehicle 100 may be configured to detect the driver's hand interaction, such as touch or movement, associated with the steering wheel 104 when the steering wheel sensor detects that the driver's hand may not be touching the steering wheel 104 and the vehicle 100 may be in motion. Figure 2 Infotainment system 240) and / or user device (shown in Figure 2 204), requesting the driver 102 to hold the steering wheel 104 with his hands. In response to hearing / seeing the notification, the driver 102 can hold the steering wheel 104 with his hands.

[0019] There may be situations where the driver 102 may be deceiving the vehicle 100 or the steering wheel sensor while the vehicle 100 may be in motion, such that the steering wheel sensor may detect that the driver's hands are on the steering wheel 104; however, in reality, the driver's hands may not be on the steering wheel 104. For example, the driver 102 may place an object 106 (e.g., a ball, etc.) on the steering wheel 104 (as shown in view 108) such that the steering wheel sensor may "detect" that the driver's hands are on the steering wheel 104; however, in reality, the object 106 may be on the steering wheel 104. In this case, the driver's hands may be involved in some other activity (e.g., operating a user device, or the hands may simply be resting on the driver's legs). In some aspects, the driver 102 may also attach a third-party electromechanical, electromagnetic, or digital deception device that may deceive the steering wheel sensor to detect that the driver's hands may be on the steering wheel 104.

[0020] To detect such situations where the driver is deceiving the vehicle 100 or the steering wheel 104, the vehicle 100 may include a driver deception detection system (in Figure 2 102 ), which can detect when the driver 102 may be attempting to deceive the vehicle 100. In some aspects, the driver deception detection system ("system") can be configured to detect when the driver 102 may be attempting to deceive the vehicle 100 from one or more vehicle interior depth sensing sensors (in Figure 2The depth sensing sensor obtains input from the steering wheel 104 and / or the driver's hands, and determines whether the driver 102 is deceiving the vehicle 100 based on the obtained input. The depth sensing sensor may include one or more vehicle interior cameras, including but not limited to a time-of-flight (TOF) camera, a gesture-controlled TOF camera, and the like. The depth sensing sensor may also include a radio detection and ranging (radar) sensor, a light detection and ranging (lidar) sensor, and the like. In some aspects, according to the present disclosure, the depth sensing sensor may not require the steering wheel 104 and / or the driver's hands to be in the sensor's field of view (FOV). In other words, even if the steering wheel 102 and / or the driver's hands are not in the sensor's FOV, the system may determine whether the driver 102 may be deceiving the vehicle 100 based on the input obtained from the depth sensing sensor.

[0021] In some aspects, the system may be an artificial intelligence (AI) / machine learning (ML) based system that may be configured to estimate the position of the driver's hands in the vehicle 100 (specifically, in an interior portion of the vehicle) based on input obtained from a depth sensing sensor (which may not include a view of the steering wheel 104 and / or the driver's hands, as described above). In response to estimating the position of the driver's hands in the interior portion of the vehicle, the system may estimate a probability that the driver's hands are touching the steering wheel 104 and a rotation angle or torque of the steering wheel when the driver 102 moves his hand while touching the steering wheel 104 based on the estimated position of the driver's hands. If the driver 102 may have adjusted the position of the steering wheel 104, the system may estimate the position of the steering wheel 104 by obtaining from the vehicle's control unit (in Figure 2 The controller 200 may obtain input from a VCU 212 (shown as VCU 212 in FIG. 1 ) and / or determine the adjusted position of the steering wheel by correlating the predicted position of the driver's hand with input obtained from one or more steering wheel sensors.

[0022] In some aspects, in addition to or in lieu of estimating probabilities as described above, the system may use a Bayesian or non-Bayesian approach (as part of AI / ML) to determine a "confidence value" or "confidence level" for detecting the driver's hand touching the steering wheel 104, which may be used for thresholding (e.g., only confidence levels above a predefined threshold may be considered for further action). The system may also use an end-to-end neural network to determine the confidence level.

[0023] The system may correlate or compare the estimated probability and steering wheel torque with the "actual" data / information associated with the capacitive sensing and steering wheel rotation angle / torque obtained from the steering wheel sensor. The system may also use other sensors for correlation, and the examples described herein should not be construed as limiting. When the estimated probability and steering wheel torque are not correlated with the actual data / information, the system may determine that the driver 102 may be deceiving the vehicle 100. In other words, when the correlation value between the estimated probability / steering wheel torque and the actual data / information may be less than a predefined threshold, the system may determine that the driver 102 may be deceiving the vehicle 100. In some aspects, the predefined threshold may be adjusted or adjusted based on one or more parameters, including but not limited to the operating mode of the vehicle, known noise factors (e.g., a TOF camera may detect multipath interference or low signal-to-noise ratio due to sun glare), etc. In response to such a determination, the system may output a notification to the vehicle HMI and / or user device, requesting the driver 102 to hold the steering wheel 104 with his hands (and remove the deception device / object 106).

[0024] In some aspects, the system can generate a digital driver skeleton model (in Figure 3 306) to estimate the position of the driver's hand in the interior portion of the vehicle. In some aspects, the system can use a partial depth view of the driver 102 obtained from the depth sensing sensor (with a view that may be blocked or outside the FOV of the sensor) to generate a skeleton model of the driver in the field of view. It will be appreciated by those of ordinary skill in the art that the confidence level of the position of the components of different skeleton models may depend on the degree of occlusion and the FOV of the depth sensing sensor. In response to generating the driver skeleton model, the system can determine the distance between the position of the driver's hand in the driver skeleton model and the steering wheel 104 in the interior portion of the vehicle. When the determined distance may be less than a predefined distance threshold, the system can determine that the driver's hand may be touching the steering wheel 104, and when the determined distance may be greater than a predefined distance threshold, the system can determine that the driver's hand may not be touching the steering wheel 104.

[0025] Combined with the following Figure 2 Describe the system in further detail.

[0026] The vehicle 100, driver 102, and driver fraud detection system implement and / or perform operations as described herein in the present disclosure in accordance with the owner's manual and safety guidelines. In addition, any actions taken by the driver 102 should comply with all rules specific to the location (e.g., federal, state, country, city, etc.) and operation of the vehicle 100. Notifications such as those provided by the vehicle 100 or the driver fraud detection system should be considered advisories and followed only in accordance with any rules specific to the location and operation of the vehicle 100.

[0027] Figure 2 A block diagram of an exemplary driver behavior detection system 200 according to the present disclosure is depicted. Figure 2 When Figures 3 to 5 .

[0028] System 200 may include a vehicle 202, a user device 204, and one or more servers 206 that are communicatively coupled to each other via one or more networks 208. Vehicle 202 may be combined with the above Figure 1 The vehicle 100 is the same as described. The user device 204 may be associated with the driver 102 and may include, but is not limited to, a mobile phone, a laptop, a computer, a tablet, a wearable device, or any other similar device with communication capabilities. The server 206 may be part of a cloud-based computing infrastructure and may be associated with and / or include a telematics service delivery network (SDN) that provides services to the vehicle 202 and other vehicles that may be part of a commercial vehicle fleet ( Figure 2 In another aspect, the server 206 may store instructions associated with one or more AI / ML-based algorithms that may enable the vehicle 202 to perform driver behavior detection, as described in detail later below. The server 206 may transmit instructions associated with the AI / ML-based algorithms to the vehicle 202 via the network 208 at a predefined frequency or when the vehicle 202 requests instructions.

[0029] The network 208 illustrates an exemplary communication infrastructure in which the connected devices discussed in the various embodiments of the present disclosure may communicate. The network 208 may be and / or include the Internet, a private network, a public network, or other configurations operating using any one or more known communication protocols, such as, for example, Transmission Control Protocol / Internet Protocol (TCP / IP), Wi-Fi, Ultra-Wideband (UWB) based on the Institute of Electrical and Electronics Engineers (IEEE) standard 802.11, and cellular technologies such as Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), High Speed ​​Packet Access (HSPDA), Long Term Evolution (LTE), Global System for Mobile Communications (GSM), and Fifth Generation (5G), to name a few examples.

[0030] The vehicle 202 may include a plurality of units including, but not limited to, a vehicle computer 210, a vehicle control unit (VCU) 212, and a driver fraud detection system 214 (or system 214). The VCU 212 may include a plurality of electronic control units (ECUs) 216 arranged to communicate with the vehicle computer 210.

[0031] The user device 204 may be connected to the vehicle computer 210 and / or the system 214 via a network 208, which may communicate via one or more wireless connections, and / or the user device may communicate using a near field communication (NFC) protocol, Protocol, Wi-Fi, ultra-wideband (UWB), and other possible data connection and sharing technologies to connect directly with the vehicle 202.

[0032] In some aspects, according to the present disclosure, the vehicle computer 210 and / or the system 214 can be installed in the vehicle engine compartment (or elsewhere in the vehicle 202). In addition, the vehicle computer 210 can operate as a functional part of the system 214. The vehicle computer 210 can be or include an electronic vehicle controller having one or more processors 218 and a memory 220. In addition, the system 214 can be separate from the vehicle computer 210 (e.g., Figure 2 ), or may be integrated as part of the vehicle computer 210.

[0033] The processor 218 may be configured to communicate with one or more memory devices (e.g., memory 220 and / or Figure 2 The processor 218 may utilize the memory 220 to store programs and / or store data in code for executing various aspects of the present disclosure. The memory 220 may be a non-temporary computer-readable storage medium or memory that stores driver deception detection program code. The memory 220 may include any one or combination of volatile memory elements (e.g., dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.), and may include any one or more non-volatile memory elements (e.g., erasable programmable read-only memory (EPROM), flash memory, electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), etc.).

[0034] According to some aspects, the VCU 212 can share a power bus with the vehicle computer 210 and can be configured and / or programmed to communicate between vehicle systems, connected servers (e.g., server 206), and other vehicles operating as part of a vehicle fleet (e.g., Figure 2 The VCU 212 may include or communicate with any combination of ECUs 216, such as, for example, a body control module (BCM) 222, an engine control module (ECM) 224, a transmission control module (TCM) 226, a telematics control unit (TCU) 228, a driver assistance technology (DAT) controller 230, etc. The VCU 212 may also include and / or communicate with a vehicle perception system (VPS) 232, which may be connected to and / or control one or more vehicle sensing systems 234. The vehicle sensing system 234 may include one or more vehicle sensors, including but not limited to radio detection and ranging (RADAR or "radar") sensors configured to detect and locate objects inside and outside the vehicle 202 using radio waves, seating area lock sensors, seating area sensors, light detection and ranging ("lidar") sensors, door sensors, proximity sensors, temperature sensors, wheel sensors, ambient weather sensors, vehicle interior and exterior cameras, steering wheel sensors, inertial measurement units (IMUs), etc.

[0035] In some aspects, VCU 212 may control aspects of vehicle operation and implement one or more instruction sets received from user device 204 , from one or more instruction sets stored in memory 220 , including instructions to operate as part of system 214 .

[0036] The TCU 228 may be configured and / or programmed to provide vehicle connectivity to wireless computing systems on and off the vehicle 202 and may include a navigation (NAV) receiver 236 for receiving and processing GPS signals, module (BLEM) 238, a Wi-Fi transceiver, a UWB transceiver, and / or may be configured to communicate between the vehicle 202 and other systems (e.g., a vehicle key fob, Figure 2 Other wireless transceivers (not shown), wireless communications (including cellular communications) between the computer and the module Figure 2 216). The TCU 228 may be configured to communicate with the ECU 216 via a bus.

[0037] The ECU 216 may control various aspects of vehicle operation and communications using input from a human driver, input from the autonomous computer 210 , the system 214 , and / or via wireless signal input received from other connected devices (such as the user device 204 , the server 206 , etc.) over a wireless connection.

[0038] The BCM 222 typically includes an integration of sensors, vehicle performance indicators, and varactors associated with vehicle systems, and may include processor-based power distribution circuits that control functions associated with the vehicle body, such as lights, windows, security devices, cameras, audio systems, speakers, wipers, door locks and access controls, and various comfort controls. The BCM 222 may also operate as a gateway for bus and network interfaces to communicate with remote ECUs ( Figure 2 (not shown) to interact.

[0039] The DAT controller 230 may provide Level 1 to Level 3 automated driving and driver assistance functions, which may include, for example, active park assist, vehicle reverse assist, and adaptive cruise control, etc. The DAT controller 230 may also provide aspects of user and environmental input that may be used for user authentication.

[0040] In some aspects, the vehicle computer 210 may be connected to an infotainment system 240. The infotainment system 240 may include a touch screen interface portion, and may include a voice recognition feature, a biometric recognition capability that may recognize a user based on facial recognition, voice recognition, fingerprint recognition, or other biometric recognition means. In other aspects, the infotainment system 240 may also be configured to receive user commands via the touch screen interface portion, and / or output or display notifications (including visual and / or audible alarm notifications), navigation maps, etc. on the touch screen interface portion.

[0041] The vehicle 202 may also include a first detection unit 242 and a second detection unit 244. The first detection unit 242 may include a steering wheel touch sensor (e.g., a capacitive touch sensor) and / or a steering wheel torque or rotation angle sensor. The first detection unit 242 may be configured to detect a first parameter associated with the steering wheel 104. In an exemplary aspect, the first parameter may be associated with capacitive sensing of the steering wheel touch sensor and / or a steering wheel torque or a steering wheel rotation angle. In some aspects, the first detection unit 242 may be part of the vehicle sensing system 234. In other aspects, the first detection unit 242 may be separate from the vehicle sensing system 234.

[0042] The second detection unit 244 may be configured to capture one or more inputs associated with the driver 102 when the driver 102 may be located in the interior portion of the vehicle. The second detection unit 244 may include, but is not limited to, one or more depth sensors, radar sensors, lidar sensors, TOF cameras, etc. The input captured by the second detection unit 244 may include, but is not limited to, a depth image, a plurality of voxels, a point cloud, etc. associated with the driver 102 when the driver 102 may be located in the interior portion of the vehicle. In some aspects, the input may include depth data / information as a depth map / image, a point cloud, or an occupied voxel directly or after processing. In addition, the input may include information about the reflectivity of the object (e.g., return signal strength). In some aspects, the depth information may be determined by direct or indirect time-of-flight principles. Depending on the capabilities of the sensors included in the second detection unit 244, an estimate of the speed may be generated, which may be compared with the input obtained from one or more vehicle sensors. For example, a radar may estimate depth and relative radial velocity, which may be used to estimate the velocity vector of the driver's hand / arm, which may be compared with the rotation rate of the steering wheel 104 (as an example).

[0043] In some aspects, the second detection unit 244 can be part of the vehicle sensing system 234. In other aspects, the second detection unit 244 can be separate from the vehicle sensing system 234.

[0044] The computing system architecture of the vehicle computer 210, VCU 212 and / or system 214 may omit certain computing modules. It should be readily understood that Figure 2 The computing environment depicted in FIG. 1 is an example of possible implementations according to the present disclosure and, therefore, should not be considered limiting or exclusive.

[0045] According to some aspects, system 214 can be integrated with and / or executed as part of ECU 216. System 214, whether integrated with vehicle computer 210 or ECU 216, or operating as a standalone computing system in vehicle 202, can include transceiver 246, processor 248, and computer readable memory 250.

[0046] The transceiver 246 may be configured to receive information / input from one or more external devices or systems (e.g., user device 204, server 206, etc.) via network 208. For example, the transceiver 246 may be configured to receive instructions associated with one or more AI / ML-based algorithms from the server 206, as described above. In addition, the transceiver 246 may transmit notifications or signals to the external device or system. In addition, the transceiver 246 may be configured to receive information / input from a vehicle component (such as an infotainment system 240, a first detection unit 242, a second detection unit 244, etc.). In addition, the transceiver 246 may transmit a signal (e.g., a command signal) or a notification to a vehicle component, such as an infotainment system 240.

[0047] The processor 248 and the memory 250 may be the same or similar to the processor 218 and the memory 220, respectively. In some aspects, the processor 248 may utilize the memory 250 to store programs and / or store data in code form to perform various aspects according to the present disclosure. The memory 250 may be a non-transitory computer-readable storage medium or memory storing driver fraud detection program code. In some aspects, the memory 250 may additionally store instructions / information / data obtained from the server 206, the user device 204, and the first detection unit 242 and the second detection unit 244.

[0048] In some aspects, the processor 248 can be an AI / ML-based processor that can be configured to execute instructions associated with an AI / ML-based algorithm stored in the memory 250 (e.g., obtained from the server 206) to determine the position / location of the driver's hands in the interior portion of the vehicle, generate a digital driver skeletal model 306 (e.g., Figure 3 ), and / or based on the input obtained from the second detection unit 244, the probability of the driver's hand touching the steering wheel 104 and the expected steering wheel rotation angle / torque are estimated. In some aspects, the processor 248 can perform the functions / actions described herein by using supervised or reinforced machine learning and generating a "trained" machine learning module. In response to generating a trained machine learning module, the processor 248 can store the module in the memory 250 and use the instructions stored in the module to detect possible driver steering wheel fraud activities when the driver 102 is driving the vehicle 100.

[0049] As will be appreciated by those of ordinary skill in the art, machine learning is the application of artificial intelligence (AI) using which a system or processor (e.g., processor 248) may have the ability to automatically learn and enhance based on experience without explicit programming. Machine learning focuses on using data and algorithms to mimic the way humans learn. In some aspects, machine learning algorithms may be created to perform classification and / or prediction. Machine learning-based systems may be used for a variety of applications, including but not limited to speech recognition, image or video processing and statistical analysis, probability determination, and the like.

[0050] Machine learning can be of various types based on the data or signals available to the learning system. For example, machine learning methods may include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Supervised learning is a method that can be supervised by humans. In this method, the machine learning algorithm can use labeled training data and defined variables. In the case of supervised learning, both the input and output of the algorithm can be specified / defined, and the algorithm can be trained to accurately classify the data and / or predict the results. In the present disclosure, since the processor 248 (or a processor based on "AI / ML") performs functions or makes predictions / estimations based on inputs obtained from depth sensing sensors (wherein the inputs may include noise), in some aspects, the system 214 may also include filters, such as Kalman filters, particle filters, etc., and the processor 248 may obtain "filtered" inputs via filters to perform functions or make predictions / estimations.

[0051] Generally, supervised learning can be of two types: “regression” and “classification”. In classification learning, a learning algorithm can help to divide a data set into multiple categories based on different parameters. In this case, a computer program can be trained on a training data set, and based on the training, the computer program can classify the input data into different categories. Some of the known methods used in classification learning include logistic regression, K-nearest neighbors, support vector machine (SVM), kernel SVM, naive Bayes, decision tree classification, and random forest classification.

[0052] In regression learning, the learning algorithm predicts output values ​​which may be of continuous nature or real valued. Some of the known methods used in regression learning include simple linear regression, multiple linear regression, polynomial regression, support vector regression, decision tree regression, and random forest regression.

[0053] Unsupervised learning is a method involving algorithms that can be trained on unlabeled data. An unsupervised learning algorithm can analyze data by itself and find patterns in the input data. In addition, semi-supervised learning is a combination of supervised learning and unsupervised learning. Semi-supervised learning algorithms involve labeled training data; however, semi-supervised learning algorithms can still find patterns in the input data. Reinforcement learning is a multi-step or dynamic process. The model is similar to supervised learning, but may not be trained using sample data. The model can "learn as it goes" by using trial and error. A series of successful results can be reinforced to develop the best recommendation or strategy for a given situation in reinforcement learning. As described above, in some aspects, the processor 248 can perform the functions / actions described in the present disclosure by using reinforcement machine learning.

[0054] In some aspects, to generate a trained machine learning module, the processor 248 may first “monitor” the driver’s movements (e.g., hand movements, upper body movements, eye gaze movements) in the vehicle 100 when the probability of deception may be low (e.g., when the driver 102 enters the vehicle 100 or turns on the ignition switch or operates the infotainment system 240). During this duration, the processor 248 may estimate the position / positioning of the driver’s hands, generate a digital driver skeleton model, estimate the probability that the driver’s hands touch corresponding vehicle components (e.g., the infotainment system 240 or the vehicle ignition actuator), etc. based on the input obtained from the second detection unit 244 and the instructions associated with the AI / ML-based algorithm stored in the memory 250. The processor 248 may then compare the “actual” data (e.g., when the driver 102 uses the infotainment system 240, the infotainment system 240 may send a driver interaction confirmation signal to the processor 248) with the estimated data described above to calibrate the machine learning module.

[0055] The processor 248 may also transmit a request to the driver 102 (e.g., via the infotainment system 240 or the user device 204) to perform a predefined action (e.g., to turn the steering wheel 104 left or right by a requested angle). When the driver 102 turns the steering wheel 104 in response to receiving the request, the processor 248 may estimate the steering wheel rotation angle / torque based on the input obtained from the second detection unit 244 and compare it to the rotation angle requested by the processor 248 or determined using the input obtained from the first detection unit 242. Based on the comparison, the processor 248 may calibrate the machine learning module until the error in the detection may be less than a predefined threshold. In this way, the processor 248 may "train" the machine learning module. Once the error in detection may be less than a predefined threshold, or when the processor 248 may have a "confidence" as described above that the detection / estimation of the driver's hand position, probability, etc. may be high (e.g., a confidence level above a predefined confidence threshold), the processor 248 may consider the calibrated machine learning module to be a "trained" machine learning module, and the trained machine learning module may be stored in the memory 250. The processor 248 may continue to update or enhance the trained machine learning module based on new data, information, and learnings.

[0056] In some aspects, when training the machine learning module, the processor 248 may also consider one or more noise factors that may be present in the input obtained from the second detection unit 244, or interference / bias / inconsistencies in the input due to the driver's seating area position, the driver's height (e.g., when the second detection unit 244 may only be able to capture the driver's head, rather than the driver's head, shoulders and / or upper arms), varying driver hand positions or driving styles, etc. The processor 248 may also consider the impact of the vehicle's speed change rate on the estimated steering wheel torque when training the machine learning module.

[0057] In some aspects, the processor 248 may additionally perform pre-processing to perform comparison / correlation (described above) tests or to work better with, for example, Pearson correlation coefficient, Spearman rank correlation coefficient, etc. Experimental data may be used to determine appropriate thresholds for the correlation coefficients described above. Edge cases (such as a driver driving with his chin or knee, or a passenger driving the vehicle, etc.) may also be considered during the training process.

[0058] Once the machine learning module can be trained, the processor 248 can use the module during vehicle operation, as described below.

[0059] In operation, when the driver 102 is driving the vehicle 202, the processor 248 can obtain data (or "first parameters") associated with the capacitive sensing of the steering wheel touch sensor and / or the steering wheel torque or the steering wheel rotation angle from the first detection unit 242. Exemplary data / outputs obtained from the steering wheel touch sensor are respectively Figure 4 and Figure 5 4. In each of the graphs 402, 502, and 504, the Y-axis depicts the output from the steering wheel touch sensor as a value of "0" or "1", and the X-axis depicts time. An output of "0" from the steering wheel touch sensor indicates that the first detection unit 242 has not detected any touch on the steering wheel 104, and an output of "1" indicates that the first detection unit 242 has detected a touch on the steering wheel 104.

[0060] In addition, the exemplary steering wheel torque obtained from the first detection unit 242 is Figure 4 and Figure 5 4, 506, and 508. In each of the graphs 404, 506, and 508, the Y-axis depicts the measured torque and the X-axis depicts time.

[0061] In addition to obtaining data from the first detection unit 242, the processor 248 may also obtain input from the second detection unit 244 described above. For example, the processor 248 may obtain a depth image of the driver 102 in the interior portion of the vehicle. The processor 248 may use the obtained data and input to detect whether the driver 102 may be deceiving the vehicle 202 or the steering wheel 104. In some aspects, such as Figure 3 As shown in view 302 of FIG. 304 , when the field of view (FOV) of the vehicle's interior camera may be obstructed and the vehicle's interior camera may not be able to capture an image of the driver's hands on the steering wheel 104, the processor 248 may obtain data and inputs described herein to detect possible driver deception activities. As shown in view 302, the FOV of the vehicle's interior camera may be obstructed by a structure / object 304 (which may be a part of the vehicle 202 or an external object / structure attached to the interior portion of the vehicle). In this case, it may not be possible to determine whether the driver's hands are on the steering wheel 104 based on the image obtained from the vehicle's interior camera. In such a case, the processor 248 can use data and inputs from the first detection unit 242 and the second detection unit 244 to detect whether the driver's hands are actually on the steering wheel 104 or whether the driver 102 may be deceiving the vehicle 202 or the steering wheel 104.

[0062] In response to obtaining data and input from the first detection unit 242 and the second detection unit 244, the processor 248 can use the trained machine learning module (or end-to-end neural network model) described above to estimate the location of the driver's hand in the interior portion of the vehicle based on the input obtained from the second detection unit 244. In other words, the processor 248 can use the depth image of the driver 102 in the interior portion of the vehicle and the trained machine learning module to estimate the location of the driver's hand in the interior portion of the vehicle.

[0063] In some aspects, the processor 248 may additionally or alternatively estimate the position of the driver's hands within the interior portion of the vehicle by generating a digital driver skeletal model 306 using a depth image obtained from the second detection unit 244 and one or more known image processing algorithms / processes for generating a 3-dimensional model based on a 2-dimensional image. Examples of such image processing algorithms / processes include, but are not limited to, "Occlusion-tolerant and personalized 3D human pose estimation in RGB images" by AQ mmaz and A. Argyros (2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 6904-6911, doi:10.1109 / ICPR48806.2021.9411956), "A Joint Model for 2D and 3D Pose Estimation from a Single Image" by Smo-Serra, Edgar et al. (2013 IEEE Conference on Computer Vision and Pattern Recognition (2013): 3634-3641), and "Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans" by Srada Peng et al. (CVPR 2021), all of which are incorporated herein by reference.

[0064] In response to estimating the position of the driver's hand in the interior portion of the vehicle (by using a trained machine learning module or by generating a digital driver skeletal model 306), the processor 248 can estimate the probability of the driver's hand touching the steering wheel 104 and the steering wheel torque / rotation angle (collectively referred to as "second parameters" associated with the driver's interaction with the steering wheel 104) based on the estimated position of the driver's hand. In an exemplary aspect, the processor 248 can estimate the probability of the driver's hand touching the steering wheel 104 by calculating the distance between the estimated position of the driver's hand in the interior portion of the vehicle and the position of the steering wheel 104 (which can be pre-stored in the memory 250). An exemplary view of the driver's hand 308 approaching the steering wheel 104 is shown in FIG. Figure 3 310. When the calculated distance may be less than the first predefined distance threshold, the processor 248 may estimate that the probability that the driver's hand touches the steering wheel 104 may be high. On the other hand, when the calculated distance may be greater than the second predefined distance threshold (which may be the same as or different from the first predefined distance threshold), the processor 248 may estimate that the probability that the driver's hand touches the steering wheel 104 may be low.

[0065] An exemplary estimated probability associated with the driver's hand 308 touching the steering wheel 104 is Figure 3 , Figure 4 and Figure 5 312, 406, and 510. In each of the graphs 312, 406, and 510, the Y-axis depicts the estimated probability and the X-axis depicts time.

[0066] In a similar manner, the processor 248 can estimate the steering wheel torque by tracking the movement of the driver's hand in the vehicle interior portion over a duration based on the estimated position of the driver's hand 308 in the vehicle interior portion. Figure 3 , Figure 4 and Figure 5 314, 408, and 512. In each of the graphs 314, 408, and 512, the Y-axis depicts the estimated probability and the X-axis depicts time.

[0067] In response to estimating the second parameter, the processor 248 may correlate the estimated second parameter with the data / first parameter obtained from the first detection unit 242 and determine whether the predefined condition may be met based on the correlation. In response to determining that the predefined condition may be met, the processor 248 may determine that the driver 102 may be deceiving the vehicle 100 or the steering wheel 104.

[0068] In an exemplary first aspect, the processor 248 may correlate the first parameter and the second parameter by comparing the "actual" steering wheel torque, as obtained from the first detection unit 242, with the "estimated" steering wheel torque described above and calculating a difference (e.g., an average difference over a predefined duration) between the actual steering wheel torque and the estimated steering wheel torque. When the calculated difference may be greater than a predefined difference threshold, the processor 248 may determine that the predefined condition may be satisfied and the driver 102 may be cheating on the vehicle 100 or the steering wheel 104. For example, if Figure 5 As shown, the pattern of estimated steering wheel torque over time in graph 512 is significantly different from the pattern of "actual" steering wheel torque over time depicted in graphs 506 and 508. In this case, processor 248 may determine that the difference between the estimated steering wheel torque and the actual steering wheel torque may be greater than a predefined difference threshold, and therefore may determine that driver 102 may be deceiving vehicle 100 / steering wheel 104. In an exemplary aspect, graph 506 may be associated with an instance when driver 102 may have placed object 106 on steering wheel 104, and graph 508 may be associated with an instance when driver 102 may have used an electromechanical or electromagnetic deception device.

[0069] On the other hand, the pattern of estimated steering wheel torque over time in graph 408 is similar to the pattern of actual steering wheel torque over time depicted in graph 404. In this case, the processor 248 may determine that the difference between the estimated steering wheel torque and the actual steering wheel torque may be less than the predefined difference threshold, and therefore may determine that the driver 102 may not be tricking the vehicle 100 / steering wheel 104. In other words, in this case, the processor 248 may determine that the driver 102 may be controlling / steering the steering wheel 104 with his hands.

[0070] In an exemplary second aspect, the processor 248 may correlate the first parameter and the second parameter by identifying a similarity or "correlation value" between the estimated probability associated with the driver's hand 308 touching the steering wheel 104 and the actual capacitive sensing data obtained from the first detection unit 242. In this case, the processor 248 may compare the estimated probability with the capacitive sensing data and may identify a difference between the estimated probability and the capacitive sensing data (e.g., an average difference over a predefined duration). In an exemplary aspect, an output of "1" for the capacitive sensing data (shown in graphs 402, 502, and 504) may be processed by the processor 248 as a 100% actual probability that the driver 102 may be touching the steering wheel 104.

[0071] In some aspects, when the difference between the estimated probability and the actual probability described above is likely to be low, the processor 248 may determine that the correlation value between the first parameter and the second parameter is likely to be high. Figure 4 As shown, between time "T1" and "T2", the difference between the estimated probability shown in graph 406 and the actual probability shown in graph 402 may be low. During this duration, processor 248 may determine that the correlation value between the first parameter and the second parameter may be high. In response to this determination, processor 248 may determine that driver 102 may not be deceiving vehicle 100 / steering wheel 104 and may control steering wheel 104 by hand.

[0072] On the other hand, when the difference between the estimated probability and the actual probability described above may be high, the processor 248 may determine that the correlation value between the first parameter and the second parameter may be low (e.g., less than a threshold value). Figure 5 As shown, the difference between the estimated probability shown in graph 510 and the actual probability shown in graphs 502 and 504 may be high. In response to determining that the difference may be high, processor 248 may determine that the correlation value between the first parameter and the second parameter may be low. In response to this determination, processor 248 may determine that the predefined condition may be satisfied and the driver 102 may be deceiving the vehicle 100 / steering wheel 104. In an exemplary aspect, graphs 502 and 504 may be associated with instances where the driver 102 may be deceiving using an object 106 or an electromechanical / electromagnetic deception device.

[0073] In response to determining that the predefined condition may be met (i.e., the driver 102 may be cheating), the processor 248 may perform a predefined action. For example, the processor 248 may transmit a notification to the infotainment system 240 and / or the user device 204 via the transceiver 246 in response to determining that the predefined condition may be met. The notification may include an audio and / or video notification or request to the driver 102 to control or hold the steering wheel 104 by hand and / or remove the deceptive device / object 106.

[0074] Although the above description describes aspects of detecting whether the driver 102 may be cheating, one of ordinary skill in the art will appreciate that the present disclosure may also be applied to other applications, such as determining whether the driver 102 may be ready to take over control of the vehicle 202 (e.g., whether the driver 102 has one or two hands on the steering wheel 104).

[0075] Figure 6 A flow chart depicts an exemplary method 600 for detecting driver behavior in a vehicle 100 according to the present disclosure. Figure 6The following process is exemplary and is not limited to the steps described below. Furthermore, alternative embodiments may include more or fewer steps than shown or described herein, and may include the steps in an order different from the order described in the following exemplary embodiments.

[0076] The method 600 starts at step 602. At step 604, the method 600 may include obtaining, by the processor 248, a first parameter from the first detection unit 242 and an input from the second detection unit 244. Figure 2 Examples of first parameters and inputs are described.

[0077] At step 606, the method 600 may include estimating, by the processor 248, a second parameter associated with the interaction of the driver and the steering wheel 104 based on the input obtained from the second detection unit 244. At step 608, the method 600 may include correlating, by the processor 248, the first parameter and the second parameter. At step 610, the method 600 may include determining, by the processor 248, that a predefined condition may be satisfied based on the correlation. At step 612, the method 600 may include transmitting, by the processor 248, a notification to the infotainment system 240 and / or the user device 204 in response to determining that the predefined condition may be satisfied.

[0078] Method 600 may end at step 614 .

[0079] In the above disclosure, reference has been made to the accompanying drawings that form part of the above disclosure, which illustrate specific implementations in which the present disclosure may be practiced. It should be understood that other implementations may be utilized and structural changes may be made without departing from the scope of the present disclosure. References to "one embodiment," "embodiment," "exemplary embodiment," etc. in this specification indicate that the described embodiment may include specific features, structures, or characteristics, but each embodiment may not necessarily include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same embodiment. In addition, when features, structures, or characteristics are described in conjunction with an embodiment, whether or not explicitly described, those skilled in the art will recognize such features, structures, or characteristics in conjunction with other embodiments.

[0080] In addition, where appropriate, the functions described herein may be performed in one or more of the following: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) may be programmed to perform one or more of the systems and programs described herein. Certain terms are used throughout the specification and claims to refer to specific system components. As will be appreciated by those skilled in the art, components may be referred to by different names. This document is not intended to distinguish between components that have different names but the same function.

[0081] It should also be understood that the word "example" as used herein is intended to be non-exclusive and non-limiting in nature. More specifically, the word "example" used herein indicates one of several examples, and it should be understood that no undue emphasis or preference is placed on the specific example described.

[0082] Computer-readable media (also referred to as processor-readable media) include any non-transitory (e.g., tangible) media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media may take many forms, including, but not limited to, non-volatile media and volatile media. A computing device may include computer-executable instructions, where the instructions may be executable by one or more computing devices (such as those listed above) and stored on a computer-readable medium.

[0083] With respect to the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes may be practiced by performing the described steps in an order different from the order described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating various embodiments and should in no way be construed as limiting the claims.

[0084] Therefore, it should be understood that the above description is intended to be illustrative and not restrictive. Upon reading the above description, many embodiments and applications other than the examples provided will be apparent. The scope should not be determined with reference to the above description, but should be determined with reference to the entire scope of the appended claims and equivalents to which such claims are entitled. It is expected and anticipated that the technology discussed herein will develop in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that the present application is capable of modification and change.

[0085] Unless explicitly indicated to the contrary in this article, all terms used in the claims are intended to be given their ordinary meanings as understood by the skilled person described herein. Specifically, unless the claims state an explicit limitation to the contrary, the use of singular articles such as "one", "the", "said" and the like should be interpreted as describing one or more of the indicated elements. Unless otherwise specifically stated or understood in other ways within the context when used, conditional language such as, in particular, "can", "may", "can" or "may" is generally intended to express that certain embodiments may include certain features, elements and / or steps, while other embodiments may not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require each feature, element and / or step in any way.

[0086] In one aspect of the invention, the input comprises at least one of a depth image, a plurality of voxels, and a point cloud.

[0087] In one aspect of the present invention, the second parameter is at least one of a probability that a vehicle occupant's hand touches the steering wheel and a steering wheel torque.

[0088] In one aspect of the present invention, the method includes: estimating the position of a vehicle occupant's hand in an interior portion of a vehicle based on an input; and estimating the probability that the vehicle occupant's hand touches the steering wheel and the steering wheel torque based on the position of the vehicle occupant's hand in the interior portion of the vehicle.

[0089] In one aspect of the invention, the method comprises: comparing the steering wheel torque with the first parameter; and determining that the predefined condition is met based on the comparison.

[0090] According to the present invention, a non-transitory computer-readable storage medium is provided, on which instructions are stored, which, when executed by a processor, cause the processor to: obtain a first parameter from a first detection unit and obtain input from a second detection unit, wherein the first detection unit is configured to detect a first parameter associated with a steering wheel, and wherein the second detection unit is configured to capture input associated with a vehicle occupant; estimate a second parameter associated with an interaction between the vehicle occupant and the steering wheel based on the input; correlate the first parameter and the second parameter; determine that a predefined condition is satisfied based on the correlation; and transmit a notification in response to determining that the predefined condition is satisfied.

Claims

1. A vehicle comprising: steering wheel; a first detection unit configured to detect a first parameter associated with the steering wheel; a second detection unit configured to capture an input associated with a vehicle occupant; a processor communicatively coupled to the steering wheel, the first detection unit, and the second detection unit, wherein the processor is configured to: obtaining the input and the first parameter; estimating a second parameter associated with an interaction of a vehicle occupant with the steering wheel based on the input; correlating the first parameter and the second parameter; determining, based on the correlation, that a predefined condition is satisfied; as well as A notification is transmitted in response to determining that the predefined condition is satisfied. 2 . The vehicle of claim 1 , wherein the first detection unit is a steering wheel touch sensor, and wherein the first parameter is associated with capacitive sensing of the steering wheel touch sensor. 3 . The vehicle of claim 1 , wherein the first detection unit is a steering wheel torque sensor, and wherein the first parameter is associated with a steering wheel torque or a steering wheel rotation angle.

4. The vehicle of claim 1, wherein the second detection unit is at least one of a depth sensing sensor, a radio detection and ranging (radar) sensor, a time of flight (TOF) camera, and a light detection and ranging (lidar) sensor.

5. The vehicle of claim 1, wherein the input comprises at least one of a depth image, a plurality of voxels, and a point cloud. 6 . The vehicle of claim 1 , wherein the second parameter is at least one of a probability that a vehicle occupant's hand touches the steering wheel and a steering wheel torque.

7. The vehicle of claim 6, wherein the processor is further configured to: estimating a position of a vehicle occupant's hand within the vehicle interior portion based on the input; and The probability that a vehicle occupant's hand touches the steering wheel and the steering wheel torque are estimated based on the position of the vehicle occupant's hand in the vehicle interior portion.

8. The vehicle of claim 7, wherein the processor is further configured to: comparing the steering wheel torque to the first parameter; and A determination is made based on the comparison that a predefined condition is satisfied.

9. The vehicle of claim 1, wherein the processor is further configured to: generating a digital vehicle occupant skeletal model based on the input; and The second parameter is estimated based on the digital vehicle occupant skeletal model.

10. The vehicle of claim 1, wherein the predefined condition is satisfied when a correlation value between the first parameter and the second parameter is less than a threshold value.

11. The vehicle of claim 1 , wherein the processor transmits the notification to a user device or a vehicle human machine interface.

12. A method for detecting driver behavior in a vehicle, the method comprising: obtaining, by a processor, a first parameter from a first detection unit and an input from a second detection unit, wherein the first detection unit is configured to detect the first parameter associated with a steering wheel, and wherein the second detection unit is configured to capture the input associated with a vehicle occupant; estimating, by the processor, a second parameter associated with an interaction of a vehicle occupant with the steering wheel based on the input; correlating, by the processor, the first parameter and the second parameter; determining, by the processor, that a predefined condition is satisfied based on the correlation; as well as A notification is transmitted by the processor in response to determining that the predefined condition is satisfied.

13. The method of claim 12, wherein the first detection unit is a steering wheel touch sensor, and wherein the first parameter is associated with capacitive sensing of the steering wheel touch sensor. 14 . The method of claim 12 , wherein the first detection unit is a steering wheel torque sensor, and wherein the first parameter is associated with a steering wheel torque or a steering wheel rotation angle.

15. The method of claim 12, wherein the second detection unit is at least one of a depth sensing sensor, a radio detection and ranging (radar) sensor, a time of flight (TOF) camera, and a light detection and ranging (lidar) sensor.

Images