SYSTEM AND METHODS FOR MONITORING DRIVER BEHAVIOR FOR VEHICLE FLEET MANAGEMENT IN A FLEET OF VEHICLES USING AN IMAGE DEVICE IN FRONT OF THE DRIVER.

Abstract

System and methods for monitoring driver behavior for vehicle fleet management in a fleet of vehicles using an image formation device in front of the driver.The systems and methods herein refer, in general, to the management of vehicle fleets to increase fleet safety and improve the performance of fleet drivers, and further refer to the monitoring of the operation of fleet vehicles using one or more driver-facing imaging devices located in the fleet vehicles to record the activities of fleet drivers and their passengers, store information related to the monitored activities, selectively generate warnings related to the monitored activities, and report the monitored activities to a central fleet management system for use in increasing the safety of fleet vehicles and helping to improve driver performance.

Description

DRIVER BEHAVIOR MONITORING SYSTEM AND METHODS FOR THE MANAGEMENT OF VEHICLE FLEET IN A FLEET OF VEHICLES USING AN IMAGING DEVICE IN FRONT OF THE DRIVER TECHNICAL FIELD

[0001] The modalities herein generally refer to the management of vehicle fleets to increase the safety of the fleet and improve the performance of the fleet drivers. More specifically, the particular embodiments relate to monitoring the operation of the fleet vehicles using one or more driver-facing imaging devices arranged on the fleet vehicles to record the activities of the fleet drivers and their passengers, and Reporting monitored activities to a central fleet management system for use in increasing the safety of fleet vehicles and helping to improve the performance of fleet drivers. CROSS REFERENCE WITH RELATED REQUESTS

[0002] This request is related to the U.S. request. Ser. No. 14 / 233,319, filed on July 12, 2012, titled: VEHICULAR FLEET MANAGEMENT SYSTEM AND MA. a.ZUZÓ / U IU900 METHODS TO MONITOR AND IMPROVE DRIVER PERFORMANCE IN A FLEET OF VEHICLES (Proxy File No.: 013097-000010), the contents of which are incorporated herein by reference in their entirety.

[0003] This request is also related to the U.S. Request. Ser. No., presented, entitled: SYSTEM AND METHODS OF MONITORING DRIVER BEHAVIOR FOR THE MANAGEMENT OF VEHICLE FLEET USING THE IMAGING DEVICE IN FRONT OF THE DRIVER (Proxy Files Nos.: PXE-BCVS-201711—US-01 and 013097 -000034), the contents of which are incorporated herein by reference in their entirety. BACKGROUND

[0004] Existing systems and methods in the field of vehicle fleet management focus on the specific characteristics of image capture systems and the transmission of file data within image capture systems. For example, the U.S. patent No. 7,671,762 to Breslau teaches a vehicle data transceiver system and method involving the transmission of data from one vehicle to another. Specifically, Breslau involves the transmission and reception of vehicle identification data and vehicle position data and includes the use of Sensor signals. MA. a.ZUZÓ / U IUSOD Global Position (GPS) and satellite transmission. MA.a.zuz ó / u i uaoo

[0005] Another existing technology is described in U.S. Pat. No. 6,389,340 to Rayner which teaches a circuit that terminates image capture upon the occurrence of an activation event and where the components of the system are housed within a rearview mirror of a vehicle such as a car or truck.

[0006] The U.S. patent. No. 7,804,426 to Etcheson teaches a system and method for selective review of event data comprising computer-assisted preparation of driving data for selective review in order to save time. Event data is captured continuously and sent to a data buffer. Event data is sent to an event listener when requested by a fleet manager or similar.

[0007] In the U.S. Application. Related Ser. No. 14 / 233,319, filed on July 12, 2012, entitled: VEHICLE FLEET MANAGEMENT SYSTEM AND METHODS FOR MONITORING AND IMPROVING DRIVER PERFORMANCE IN A VEHICLE FLEET, a system and method is described where the Vehicles are configured to collect the driver and vehicle event data, selectively compress and encode the collected driver and vehicle event data, and communicate the compressed and encrypted data wirelessly to one or more telematics service providers. One or more servers may poll this driver event data periodically, process it, and present multiple methods to end users by which they can view and analyze it. The described system allows fleet managers to use this driver event data, received via a report or notification, or extracted directly from a web-based portal, to monitor, correct and / or reward driver behavior. and implement driver training and education programs, or similar.

[0008] In addition to the above, systems having two forward-facing cameras as well as cameras facing the driver are also known. These systems typically continuously capture images of the road and the driver inside the vehicle, and store the images in a large buffer file, such as a first-in-first-out (FOFO) buffer, for example. Road and driver image data are sent to an event detector USOD when requested by a fleet manager or similar. That way, the driver's activities during any selected event can be determined by rewinding the recorded vehicle operation video to the appropriate time of the selected event's occurrence. MA. a.ZUZÓ / U IUUDD

[0009] It is desirable, however, to more intelligently control driver behavior by monitoring one or more particular behaviors rather than using raw images and / or using raw vehicle data collection.

[0010] Furthermore, it is desirable to analyze the one or more behaviors of the particular driver, preferably before an occurrence of any significant event, so that the driver or others, such as fleet managers or the like can be adequately warned beforehand, if possible. Furthermore, it is desirable that drivers can be additionally rated in relation to safety and other considerations, as well as ranked in relation to other drivers in the vehicle fleet, to motivate drivers to behave better, thereby improving safety and overall fleet performance. BRIEF DESCRIPTION OF EXEMPLARY MODALITIES

[0011] The embodiments herein are provided for new and improved driver behavior monitoring systems and methods for vehicle fleet management in a fleet of vehicles utilizing a driver-facing imaging device.

[0012] In embodiments herein, the systems and methods are provided using a camera in front of the driver to monitor the behavior of the driver directly in accordance with a detected position of the driver's head within the vehicle being operated by the driver. . The systems and methods are provided using the camera in front of the driver to monitor the driver's use of commercial vehicle mirrors, to monitor the driver's attention to the road, to monitor the position of the driver's head with respect to a proper driving position. the head, to monitor the driver's head posture metric, to monitor any impediments to the image collected by the camera in front of the driver and to monitor the driver's eyes on the road and to make adjustments to the warning system involuntary adaptive lane change of the associated vehicle. These driver behaviors can be monitored directly as well as others as appropriate. U90D necessary and / or desired in accordance with the modalities herein. MA. a.ZUZÓ / U I 1 / 900

[0013] In further embodiments herein, systems and methods are provided using a camera in front of the driver to monitor the driver's behavior indirectly according to detected aspects of the components inside the vehicle being operated by the driver. The systems and methods are provided using the camera in front of the driver to monitor the driver's proper use of the vehicle's seat belt, to monitor the proper positions of the driver's hands on the steering wheel, and to monitor the driver's compliance with policies. of the fleet regarding unauthorized passengers who are in the vehicle. These driver behaviors may be monitored directly, as well as others as necessary and / or desired in accordance with the modalities herein.

[0014] According to embodiments herein, systems, methods and logic, including various vehicle sensors and a camera in front of the driver, are provided to determine when a set of one or more predetermined conditions of a vehicle or are otherwise satisfied, determine a driver's head posture, learn or otherwise train the system on the average values ​​of the driver's head posture (pitch, yaw, roll, etc.) when the set of one or more predetermined conditions of the vehicle or otherwise satisfied and determine which occur due to deviations of the head postures from the average values. MA.a.zuz ó / u i uacD

[0015] According to the embodiments herein, systems, methods and logic are provided including various vehicle sensors and a camera in front of the driver for determining a driver's head posture, learning or otherwise training the system. in a head posture distribution and / or a head posture heat map, and determine any factual deviations of the driver's head posture from the head posture distribution and / or the average values ​​of the head heatmap.

[0016] According to the embodiments herein, systems, methods and logic are provided including various vehicle sensors to determine whether a set of one or more predetermined conditions conducive to determining violations or misbehavior of the driver or passengers are met. other are satisfied such as a state of the vehicle door, a change of speed, an unusual stopping place, a visible unauthorized passenger, or the like, and a camera facing the driver to obtain images of the vehicle cabin in response to the set of one or more predetermined vehicle conditions. MA.a.zuz ó / u i uaoD

[0017] According to the embodiments herein, systems, methods and logic are provided including various vehicle sensors and a camera in front of the driver for learning or otherwise training the system in the average values ​​of the appearance (images or template descriptions) of vehicle cabin items such as seat belt buckles, empty seats, steering wheel, door edges, mirror locations and determine if any changes or deviations occur from the average or learned values of the operational set of images or descriptions of the learned template.

[0018] According to the embodiments herein, systems, methods and logic are provided including various vehicle sensors and a camera in front of the driver to determine a vector of the driver's head posture, learn or otherwise train the driver. system on the average values ​​of the driver's head posture vector and selectively adapt other system values ​​as a function of the driver's head posture vector when persistent deviation of the driver looking at the road or the driver looking occurs The mirrors. uaoD

[0019] According to embodiments herein, the systems, methods and logic provide multi-factor authentication using multiple sensors and a camera in front of the driver for driver identity verification using image data of the driver in combination with and voice recognition data of the driver, such as by images of the driver using the camera in front of the driver, verification of a visual identity of the driver according to information from the driver database and image data of the driver that obtain voice recognition data from the driver by speaking a standardized passphrase, while in the camera field facing the driver verifying the identity of the driver's voice recognition by requesting that the driver say his or her name, leading to a template of standardized comparison, and recording the protocol in a local memory of the vehicle system. QQRH I Π / ΡΖΟΖ / Ε / ΥΙΛ

[0020] The term processor media as used herein refers to any microprocessor, discrete logic (e.g., ASIC), analog circuit, digital circuit, programmed logic device, memory device containing instructions, and so on. The term processor media also refers to logic that may include one or more gates, combinations of gates, other circuit components, hardware, firmware, software running on a machine, and / or combinations of each to perform a function. or an action, and / or cause a function or action of another logic, method and / or system, a software-controlled microprocessor, a discrete logic (e.g., ASIC), an analog circuit, a digital circuit, a programmed logic device , a memory device that contains instructions and so on. The term memory media as used herein refers to any non-transitory media that is involved in storing data and / or providing instructions to processing media for execution. Such non-transitory media can take many forms, including, but not limited to, volatile and non-volatile media. Non-volatile media include, for example, optical or magnetic disks. Volatile media includes dynamic memory, for example, and does not include transient signals, carrier waves, or the like. Common forms of computer-readable media include, for example, a floppy disk, floppy disk, hard disk, magnetic tape, or any other magnetic media, a CD-ROM, any other optical media, punched cards, paper tape, any another physical medium with hole patterns, a RAM, PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other non-transitory tangible medium that can be read by a computer.

[0021] Other embodiments, features and advantages of the exemplary embodiments will become apparent from the following description of the embodiments, taken together with the accompanying drawings, which show, by way of example, the principles of the exemplary embodiments. wow BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In the accompanying drawings which are incorporated in and constitute a part of the specification, embodiments of the invention are shown, which, together with a general description of the invention given above, and the detailed description given below, serve to exemplify the embodiments of this invention. QQRH I Π / ΡΖΟΖ / Ε / ΥΙΛ

[0023] Fig. 1 is a diagram of an overview of the fleet management system and user design according to the exemplary embodiment.

[0024] Fig. 2 depicts the operation of an exemplary fleet vehicle operating in a convoy and having a driver behavior monitoring system having a camera facing the driver according to one embodiment.

[0025] Fig. 3 is a schematic illustration of an exemplary embodiment of a portion of the data collection module of a driver behavior monitoring system having a camera facing the driver according to the exemplary embodiment;

[0026] Fig. 4 is a block diagram showing a computer system suitable for monitoring driver behavior directly in accordance with a detected position of the driver's head within the vehicle being operated by the driver and for monitoring the behavior of the driver indirectly according to the detected aspects of the components inside the vehicle that is being operated by the driver, according to an exemplary embodiment.

[0027] Fig. 4a is a block diagram showing the executable logical components of the driver behavior monitoring system having a camera in front of the driver according to the exemplary embodiment. MA.a.zuz ó / u i uacD

[0028] Fig. 5a is a schematic diagram showing a driver-facing imaging device according to an exemplary embodiment arranged in the cabin of an associated vehicle at a fixed location on the top of a windshield of the associated vehicle.

[0029] Fig. 5b is a diagram of one embodiment of the driver-facing imaging device of Fig. 5a formed as a driver-facing camera according to an exemplary embodiment.

[0030] Fig. 6a is a first example of a calibration image generated by the driver-facing camera of Fig. 5b and obtained during a first calibration operation of the driver behavior monitoring system.

[0031] Fig. 6b is an example of a second calibration image generated by the driver-facing camera of Fig. 5b and obtained during a second calibration operation of the driver behavior monitoring system.

[0032] Fig. 7 is an example of an image generated by the driver-facing camera of Fig. 5b and obtained by the driver behavior monitoring system during operation of the associated vehicle.

[0033] Fig. 8 is a flow chart showing an operating method of a driver behavior monitoring system having a camera in front of the driver for implementing a driver behavior monitoring and reporting strategy accordingly. with an exemplary modality. MA.a.zuz ó / u i uaco

[0034] Fig. 9 is a flow chart showing an operating method of a driver behavior monitoring system having a camera in front of the driver for implementing passenger detection, counting, monitoring and strategy of reports in accordance with an exemplary modality.

[0035] Fig. 9a is a flow chart showing an additional method of operation of a driver behavior monitoring system having a camera in front of the driver for the implementation of a passenger detection, counting, monitoring and strategy of reports in accordance with an exemplary modality.

[0036] Fig. 10 is a flow chart showing an operating method of a driver behavior monitoring system having a camera in front of the driver for the implementation of seat belt use detection, monitoring and the reporting strategy in accordance with an exemplary modality.

[0037] Fig. 10a is a flow chart showing details of a part of the operating method of a driver behavior monitoring system having a camera in front of the driver for the implementation of seat belt use detection, monitoring and reporting strategy of Fig. 10, according to an exemplary embodiment.

[0038] Fig. 10b is a flow chart showing additional details of a portion of the method of operation of a driver behavior monitoring system having a camera in front of the driver for uaoo the implementation of belt use detection security, monitoring, and reporting strategy of Fig. 10, according to an exemplary embodiment.

[0039] Fig. 11 is a flow chart showing an operating method of a driver behavior monitoring system having a camera in front of the driver for the implementation of detecting a hands on the steering wheel, monitoring and the reporting strategy in accordance with an exemplary modality.

[0040] Fig. 12 is an example of an image generated by the driver-facing camera of Fig. 5b and obtained by the driver behavior monitoring system during operation of the associated vehicle and showing a typical driver who has his hands on the wheel.

[0041] Fig. 13 is a flow chart showing an operating method of a driver behavior monitoring system having a camera in front of the driver for implementing a detection of driver attention to the road, monitoring and the reporting strategy in accordance with an exemplary modality. MA. a.ZUZÓ / U IUUOO

[0042] Fig. 14 is a flow chart showing a method of operation of a driver behavior monitoring system having a camera in front of the driver for the implementation of an obstructed view detection, monitoring and reporting strategy according to an exemplary modality.

[0043] Fig. 15 is a flow chart showing an operating method of a driver behavior monitoring system having a camera in front of the driver for implementing the detection that the driver's head is out of position , monitoring and reporting strategy according to an exemplary modality.

[0044] Fig. 15a is a flow chart showing an additional method of operation of a driver behavior monitoring system having a camera in front of the driver for the implementation of detecting that the driver's head is out of position, monitoring and reporting strategy according to an exemplary modality.

[0045] Fig. 16 is a schematic diagram showing the characteristics of the head of a conductor in order to MA.a.zuz ó / u i uacD determine a vector of the driver's head posture according to an exemplary embodiment.

[0046] Fig. 17 is a flow chart showing a method of operation of a driver behavior monitoring system having a camera in front of the driver for detecting, monitoring, and reporting whether the distribution of the posture of the driver's head is changing significantly or unacceptable by implementing a driver's road attention strategy in accordance with an exemplary modality.

[0047] Fig. 18 is an example of a head posture distribution map according to an exemplary embodiment.

[0048] Fig. 19 is a flow chart showing a method of comparing driver head posture histograms, and determining and reporting deviations and / or changes between driver head posture histograms. driver. MA. a.ZUZÓ / U IUUDD

[0049] Fig. 19a is a flow chart showing a method of comparing head posture statistics, and determining and reporting deviations between a driver's head posture and the desired, appropriate situation, statistics according to an exemplary modality.

[0050] Fig. 20 is a flowchart showing a method of comparing head posture histograms, and determining and reporting deviations between a driver's head posture and the desired, appropriate situation, the histograms according to an exemplary modality.

[0051] Fig. 21 is an illustration of the limits that apply to the use of the mirror according to an exemplary embodiment. DETAILED DESCRIPTION OF EXEMPLARY MODALITIES

[0052] In the following description of the present invention, reference is made to the attached figures that form a part thereof, and where the exemplary embodiments that show the principles of the present invention and how are shown, by way of illustration. it's practiced. Other embodiments may be used to practice the present invention and structural and functional changes may be made thereto without departing from the scope of the invention. USOD present invention.

[0053] Referring now to the drawings, wherein the representations are intended to illustrate exemplary modalities for monitoring driver behavior directly using a camera in front of the driver according to a detected position of the driver's head within the vehicle being operated by the vehicle, and to monitor the driver's behavior indirectly using a camera facing the driver according to the detected aspects of the components inside the vehicle being operated by the driver only, and not for purposes of limiting the Likewise, Fig. 1 shows an overview of a fleet management and reporting system 100 according to the exemplary embodiment. In the exemplary embodiment of the present invention, vehicles 110, such as trucks and cars and in particular fleet vehicles 112, are configured with one or more data collection and reporting devices 200 (Fig. 2) that generate the event data such as, in the example of a truck fleet, truck departure, truck stop, and safety event data, wherein one such system includes, for example, a change alert system lane tracking (LDW) 322 (Fig. 3) that generates signals indicative of one or more events and driver data and vehicle events with respect to the truck fleet example, MA. a.ZUZÓ / U IU900 errant or crossing truck lane. In addition, the secondary systems that will be described in more detail below with reference to Fig. 3 carried out by the vehicles or installed in the vehicle systems such as one or more video cameras, radar, transmission, engine, monitoring tire pressure and braking systems for example, can generate additional safety event data. Third-party systems that generate proprietary security events or data representative of detected security events may also be involved. For example, embodiments of the present invention may include software code implementing a Bendiz® Wingman® ACB System available from Bendix Commercial Vehicle Systems LLC that captures proprietary safety events and other data that relates to the proprietary safety events. and / or that relate to the operation of the vehicle by one or more operators or drivers of the vehicles. MA.a.zuz ó / u i uaoD

[0054] Continuing with reference to Fig. 1, these events and event data 120 are, in the exemplary embodiment, selectively sent over one or more wireless networks or wireless links 122 to one's network servers 132. or more service providers 130. The wireless service providers 130 use servers 132 (only one shown for ease of illustration) that collect the wireless data 120 provided by the trucks 112. Each also provides a web service through which the Users can report or download data. MA.a.zuz ó / u i uaoo

[0055] One or more servers 140 of the fleet management and reporting system 100 are configured to selectively download or otherwise retrieve data from collection servers 132 which may be third-party servers from one or more various telematics providers such as those available from PeopleNet Communications Corp. or Qualcomm Inc., for example. The one or more fleet management servers 140 and reporting system 100 are configured to initiate processing of vehicle events and vehicle event data in ways that will be described in more detail below. A web application 142 executable on one or more servers 140 of the fleet management and reporting system 100 includes a dynamic graphical user interface for fleet managers 160 and administrators 162 to view all information once it is processed. . The fleet management and reporting system 100 of the exemplary embodiment also includes one or more databases 150 configured to selectively store information about all events of vehicles 112 in the fleet 110 for one or more designated time intervals. , including raw and post-processed trip data. MA.a.zuz ó / u i uacD

[0056] According to the exemplary embodiment, the system administrators 162 are the users who are provided with interfaces to configure and manage fleets, monitor the performance of the platform, virtual alerts issued by the platform and view raw data of events and the records and / or views of subsequent processing. Fleet managers 160 can view event information for their respective fleet for internal processing. These events may arrive via user-initiated reports 170 executable in the web application 142 on the one or more servers 140, or via email or other notifications 172. Fleet managers 160 may, depending on policies and internal processes or for other reasons, also interact with individual drivers 164 regarding performance goals, corrections, reports or coaches.

[0057] The fleet management and reporting system 100 of the exemplary modality therefore offers a long list of functions and features for the end user. All have been designed to be driver-centric, so fleet managers can focus their attention on improving driver education, training and performance. One of the primary beneficial and novel uses of system 100 is the ease of access to driver-specific performance data and the ability to normalize each driver's performance for comparison to drivers in the fleet as a whole in order to identify to exemplary drivers to praise them as well as those who need training or any other corrective action.

[0058] Fig. 2 depicts the operation of an exemplary fleet vehicle operating in a basic convoy A that includes a host or leader vehicle 10 in traffic with a second vehicle or follower 20 in accordance with the present description. As shown, the follower vehicle 20 moves next to the lead vehicle 10 in an ordered convoy A along a highway 1. The follower vehicle 20 is provided with an electronic control system 12' that includes data collection and portion of a communication module 300' and a monitoring control portion 400' which will be described in more detail below. Likewise, the lead vehicle 10 is also provided with an equivalent electronic control system 12 that includes an equivalent data collection and communication module portion 300 and an equivalent monitoring control portion 400. In the exemplary embodiments to be described Herein, although each of the two or more vehicles comprising the various convoys to be described includes the same or equivalent electronic control system 12, 12', the same portion of the data collection and communication module or equivalent 300, 300' and the same or equivalent monitoring control portion 400, 400', other distinct control systems having the functionality described herein may be used equivalently as necessary or desired. uaoD

[0059] In the exemplary embodiment shown, the electronic control systems 12, 12' of the respective vehicles 20, 10 are configured to mutually communicate signals and exchange data with each other, and also to communicate signals and exchange data with various other communication systems including, for example, a remote wireless communication control system 250 and a remote satellite system 260. These remote systems 250, 260 may provide, for example, global positioning system (GPS) data to vehicles. 10, 20 as desired. Other information may be provided or exchanged between vehicles and remote systems, such as, for example, fleet management and control data from a remote fleet management facility, or similar (not shown). Although this functionality is provided, the embodiments herein find this remote communication, although useful, not necessarily essential where the embodiments herein are directed to directly monitor the behavior of the driver according to a detected position of the driver's head within of the vehicle being operated by the driver and to monitor the behavior of the driver indirectly according to the detected aspects of the components of the interior of the vehicle being operated by the driver without the need to consult or act under the direction or in accordance with the remote wireless communication system 250, remote satellite system 260, remote fleet management facility, Central Command Center (CCC), a Network Operations Center (NOC) or the like. MA. a.ZUZÓ / U IUSOD

[0060] In addition to the above, the electronic control systems 12, 12' of each vehicle 10, 20 operate to perform various vehicle to (single) vehicle (V2V Unicast) communications (communication between a broadcast vehicle and a single vehicle that responds), as well as the communication of various vehicles to (multiple) vehicles (V2V Broadcast) (communication between a broadcast vehicle and two or more vehicles that respond), and even more, as well as the communication of various vehicles to infrastructure (V2I ). Preferably, local V2V Unicast and V2V Broadcast communication follows the J2945 DSRC communications specification. In this sense, the vehicles that form basic convoy A can communicate with each other locally for self-sorting and separation into a convoy without the need for input from the CCC in accordance with the modalities herein. The vehicles forming basic convoy A may also communicate with one or more other vehicles locally without the need for input from the CCC for negotiation of the one or more other vehicles in the convoy in accordance with the modalities herein. The vehicles forming basic convoy A may additionally communicate with a fleet management facility remotely as may be necessary and / or desired to directly monitor driver behavior in accordance with a detected position of the driver's head within the vehicle. that is being operated by the driver and to monitor the behavior of the driver indirectly according to detected aspects of the components inside the vehicle that is being operated by the driver, in accordance with other exemplary embodiments herein. oops QQAH I Π / ΡΖΟΖ / Ε / ΥΙΛ

[0061] As noted above, preferably, local V2V Unicast and V2V Broadcast communication between vehicles, as will be described herein follows the J2945 DSRC communications specification. This specification does not currently define one-to-one vehicle communications. Rather, operationally, each communication-capable vehicle sends the necessary information by broadcast to all other communication-capable vehicles within its range, and the receiving vehicle decides whether it wants to process the received message. For example, only vehicles that are convoy capable and for which the driver indicates, through a switch or user interface, that he or she wants to join a convoy, that vehicle will begin broadcasting and listening to convoy protocol messages. All other vehicles in the area can ignore the convoy information. Accordingly, as used herein and for the purpose of describing exemplary embodiments, V2V Unicast communication will refer to communication between a broadcast vehicle and a response-only vehicle and V2V Broadcast communication will refer to communication between one broadcast vehicle and two or more responding vehicles. It is appreciated that the communication V2V Unicast also refers to direct one-to-one vehicle communication as the J2945 DSRC communications specification is developed further or using any one or more other standards, specifications or technologies now known or hereinafter developed. MA.a.zuz ó / u i uaco

[0062] Fig. 3 is a representation of a schematic block diagram showing details of the data collection of the tow vehicle and the communication module portion 300 of Fig. 2 according to an exemplary embodiment. In accordance with the principles of the exemplary embodiment shown, the tow vehicle data collection and communication module portion 300 may be adapted to detect, monitor and report a variety of operating parameters and conditions of the commercial vehicle and the driver interaction with them, and to selectively intervene and take corrective action as may be necessary or desired, such as, for example, to maintain the stability of the vehicle or to maintain the following distance of the vehicle with respect to other vehicles within a convoy. In the exemplary embodiment of Fig. 3, the data collection and communication module portion 300 may include one or more devices or systems 314 to provide input data indicative of one or more operating parameters or one or more operating conditions. a commercial vehicle. For example, the devices 314 may be one or more sensors, such as, but not limited to, one or more wheel rotation speed sensors 316, one or more acceleration sensors, such as various axle acceleration sensors 317 , a steering angle sensor 318, a brake pressure sensor 319, one or more vehicle load sensors 320, a drift speed sensor 321, a sensor or Lane Departure Warning System (LDW) 322 , one or more speed or engine condition sensors 323 and a tire pressure monitoring system (TPMS) 324. The tow vehicle data collection and communication module portion 300 may also use additional devices or sensors in the exemplary embodiment including, for example, a forward distance sensor 360 and a rear distance sensor 362. Other sensors and / or actuators or power generating devices or combinations thereof may be used or otherwise provided as well, and One or more devices or sensors can be combined into a single unit as may be necessary or desired.

[0063] The tow vehicle data collection and communication module portion 300 may also include a logic application arrangement such as a controller or processor 330 and control logic 331, in communication with the one or more devices or systems 314. The processor 330 may include one or more inputs for receiving input data from the devices or systems 314. The processor 330 may be adapted to process the input data and compare the raw or processed input data with one or more stored threshold values, or to process the input data and compare the raw or processed input data with one or more desired values ​​depending on the circumstances. The processor 330 may also include one or more outputs for delivery of a control signal to one or more vehicle systems 323 based on the comparison. The control signal may instruct systems 323 to intervene in the operation of the vehicle to initiate corrective action, and then report this corrective action to a wireless service (not shown) or simply store the data locally to be used to determine a driver quality. For example, the processor 330 may generate and send the control signal to an electronic engine control unit or drive device to reduce the throttle of the engine 334 and decelerate the vehicle. Additionally, the processor 330 may send the control signal to one or more braking systems of the vehicle 335, MA. a.ZUZÓ / U IU900 336 to selectively apply the brakes. In the tractor-trailer arrangement of the exemplary embodiment, the processor 330 may apply the brakes 336 on one or more wheels of a trailer portion of the vehicle through a trailer pressure control device (not shown), and brakes 335 on one or more wheels of a tractor portion of the vehicle 12, and then report this corrective action to the wireless service or simply store the data locally that will be used to determine a driver quality. A variety of corrective actions may be possible and multiple corrective actions may be initiated at the same time.

[0064] The controller 300 may also include a memory portion 340 for storing and accessing system information, such as system control logic 331 and control tuning. The memory portion 340, however, may be separate from the processor 330. The sensors 314 and processor 330 may be part of a pre-existing system or use components of a pre-existing system. For example, the Bendix® ABS-6™ Advanced Anti-lock Brake System Controller with the ESP® stability system available from Bendix Commercial Vehicle Systems LLC can be installed in the vehicle. The Bendix® ESP® system may use some or all of the sensors described in Fig. 3. The logical component of the system MA. a.ZUZÓ / U IUUOO Bendix® ESP® resides in the electronic control unit of the vehicle's anti-lock brake system, which may be used by the processor 330 of the present invention. Therefore, many of the components to support the tow vehicle controller 330 of the present invention may be present in a vehicle equipped with the Bendix® ESP® system, therefore not requiring the installation of additional components. The 330 Tow Vehicle Controller, however, can use independently installed components if desired. Additionally, an IMX,6 processor separate from the ESP system may perform the functions described herein. UUDD

[0065] The data collection and communication module portion 300 of the tow vehicle controller 12 may also include an input data source 342 indicative of a commercial vehicle configuration / condition. The processor 330 may detect or estimate the vehicle configuration / condition based on the input data and may select a control or sensitivity tuning mode based on the vehicle configuration / condition. The processor 330 may compare the operational data received from the sensors or systems 314 with the information provided by the tuning. System tuning may include, but will not be limited to: the nominal center of gravity of the vehicle, lookup maps and / or tables for the level of lateral acceleration for rollover intervention, lookup maps and / or tables of the yaw speed differential of the expected yaw speed for yaw control interventions, steering wheel angle tolerance, tire variation tolerance and brake pressure rates, magnitudes and maximums to be applied during the action corrective.

[0066] A vehicle configuration / condition may refer to a set of vehicle characteristics that may influence vehicle stability (roll and / or drift). For example, in a vehicle with a towed part, the input data source 342 may communicate the type of towed part. In tractor-trailer arrangements, the type of trailer being towed by the tractor can influence the stability of the vehicle. This is evident, for example, when towing multiple trailer combinations (doubles and triples). Vehicles with multiple trailer combinations may exhibit exaggerated response of the units in reverse when maneuvering (i.e., reverse amplification). To compensate for reverse amplification, the controller Tow vehicle USOD 330 can select a tune that makes the system more sensitive (i.e., intervene sooner than would occur for a single trailer condition). Control tuning may be, for example, specifically defined to optimize the performance of the data collection and communication module for a particular type of trailer that is being towed by a particular type of tractor. Therefore, the control tuning may be different for the same tractor towing a single trailer, a double trailer combination, or a triple trailer combination. MA. a.ZUZÓ / U IUUDD

[0067] The type of load the commercial vehicle is carrying and the location of the center of gravity of the load can also influence the stability of the vehicle. For example, moving loads such as liquid tankers with partially filled compartments and livestock can potentially affect the vehicle's rollover and rollover performance. Therefore, a more sensitive control tuning mode can be selected to account for a moving load. Additionally, a separate control tuning mode may be selectable when the vehicle is transferring a load whose center of gravity is particularly low or particularly high, such as with certain types of large machinery or low flat steel bars. QQAH I Π / ΡΖΟΖ / Ε / ΥΙΛ

[0068] Additionally, the controller 300 is operatively coupled with one or more imaging devices in front of drivers shown in the exemplary embodiment for simplicity and ease of illustration as a representation of a single camera in front of the driver 345 of one or more cameras. video cameras arranged in the vehicle such as, for example, a video camera at each corner of the vehicle, one or more cameras remotely mounted and in operative communication with the controller 330, such as a forward-facing camera (FFC) arranged in the vehicle in a manner that records images of the road in front of the vehicle, or, as in the exemplary embodiment, in the cabin of a commercial vehicle trained on the driver and / or trained inside the cabin of the commercial vehicle. In exemplary embodiments, the driver's behavior is monitored directly using the driver-facing camera 345 according to a detected position of the driver's head within the vehicle being operated by the vehicle, the details of which are explained below. . In other exemplary embodiments, the driver's behavior is monitored directly using the driver-facing camera 345 according to a detected head posture of the driver. For purposes of this description of exemplary embodiments and for ease of reference, head posture is the set of angles that describe the orientation of the driver's head, i.e., pitch (driver looking down or up), yaw ( driver looking left or right), and sway (the driver tilts his / her head to the left or right). In still further embodiments, the driver's behavior is indirectly monitored using the driver-facing camera 345 in accordance with detected aspects of the vehicle interior components being operated by the vehicle, the details of which are explained below. The driver-facing camera 345 may include an imaging device available from Ominivision™ as part / model number 10635, although other suitable, equivalent imaging devices may be used as necessary or desired.

[0069] Still further, the controller 300 may also include a transmitter / receiver (transceiver) module 350 such as, for example, a radio frequency (RE) transmitter that includes one or more antennas 352 for wireless communication of requests of automated deceleration, GPS data, one or more configuration and / or condition data of various vehicles, or similar ινΐΛ / azuz ó / u i uacD between the vehicles and one or more destinations, such as, for example, to one or Most wireless services (not shown) have a receiver and corresponding antenna. The transmitter / receiver (transceiver) module 350 may include various functional sub-portion parts operatively coupled with the convoy control unit including, for example, a communication receiver portion, a global position sensor (GPS) receiver portion, and a communications transmitter. For the communication of specific information and / or data, the communication receiver and transmitter parts may include one or more functional and / or operational communication interface parts as well. uaoD

[0070] The processor 330 is operative to communicate the acquired data to the one or more receivers in a raw data form, that is, without processing the data, in a processed form such as in a compressed form, in an encrypted form or both as may be necessary or desired. In this regard, the processor 330 may combine selected data values ​​of vehicle parameters into processed data representative of higher level vehicle condition data, such as, for example, data from acceleration sensors. Multi-axle sensor 317 can be combined with data from steering angle sensor 318 to determine data for excessive cornering speed events. Other hybrid event data that is related to the vehicle and the vehicle driver and that may be obtained from the combination of one or more raw data elements selected from the sensors includes, for example and without limitation, excessive braking event data. , curve excessive speed event data, lane change warning event data, excessive lane change event data, no lane change turn signal event data, loss of event data video tracking, LDW system disabled event data, distance alert event data, forward collision alert event data, haptic alert event data, collision mitigation braking event data, event data ATC, ESC event data, RSC event data, ABS event data, TPMS event data, engine system event data, average following distance event data, average fuel consumption event data, and of average events from ACC use. Importantly, however, and in accordance with exemplary embodiments described herein, controller 300 is operative to store image data acquired from the driver and / or vehicle interior in memory 340 and communicate MA. a.ZUZÓ / U IU900 selectively the data acquired from the driver and / or from the interior of the vehicle to one or more receivers through the transceiver 350. oops

[0071] In the exemplary embodiment shown, the tow vehicle controllers 12, 12' (Fig. 2) of the respective vehicles of the convoy are configured to mutually communicate signals and exchange data with each other and between their respective one or more towed vehicles, and also to communicate signals and exchange data with various other communication systems including, for example, a remote wireless communication system and a remote satellite system. These remote systems can provide, for example, global positioning system (GPS) data to vehicles as desired. Other information may be provided or exchanged between vehicles and remote systems, such as, for example, control and fleet management data may be received from a remote fleet management facility, or the like (not shown). and driver behavior data can be sent to the remote fleet management facility, a remote satellite system, a Network Operations Center (NOC), a Central Command Center (CCC) or similar.

[0072] The tow vehicle controller 300 of Fig. 3 is suitable for executing the modalities of one or more software systems or modules that implement trailer braking strategies and trailer braking control methods according to with the request. The example of the tow vehicle controller 22 may include a bus or other communication mechanism for communicating the information and a processor 330 coupled to the bus for processing the information. The computing system includes a main memory 340, such as a random access memory (RAM) or other dynamic storage device for storing information and instructions to be executed by the processor 330, and read-only memory (ROM) or other static storage device for storing static information and instructions for the processor 330. Other storage devices may also be suitably provided for storing information and instructions as necessary or desired. MA.a.zuz ó / u i uaoD

[0073] Instructions may be read into main memory 340 from another computer-readable medium, such as another storage device through transceiver 350. Execution of the instruction sequences contained in main memory 340 causes processor 330 perform the steps of the procedure described herein. In an alternative implementation, hardwired circuits may be used instead of or in combination with software instructions to implement the invention. Thus, implementations of exemplary embodiments are not limited to any specific combination of hardware and software circuitry.

[0074] According to the descriptions herein, the term computer-readable medium as used herein refers to any non-transitory medium that is involved in providing instructions to the processor 330 for execution. Such non-transitory medium may take many forms, including but not limited to, volatile and non-volatile media. Non-volatile media include, for example, optical or magnetic disks. Volatile media includes dynamic memory, for example, and does not include transient signals, carrier waves, or the like. Common forms of computer-readable media include, for example, a floppy disk, a floppy disk, hard disk, magnetic tape, or any other magnetic media, a CD-ROM, any other optical media, punched cards, paper tape, any another physical medium with hole patterns, a RAM, PROM, and EPROM, a FLASHEPROM, any other memory chip or cartridge, or any other tangible non-transitory medium from which a MA.a.zuz ó / u i uaoo computer. MA. a.ZUZÓ / U IUSOD

[0075] Furthermore, and also in accordance with the descriptions herein, the term logic, as used herein with respect to the figures, includes hardware, firmware, software running on a machine, and / or combinations of each one to perform a function or an action, and / or cause a function or action of another logic, method and / or system. Logic may include a software-controlled microprocessor, discrete logic (e.g., ASIC), an analog circuit, a digital circuit, a programmed logic device, a memory device containing instructions, and so on. The logic may include one or more gates, combinations of gates, or other circuit components.

[0076] Fig. 4 is a block diagram showing a computer system that monitors the behavior of the driver 400 suitable for the execution of the modalities of one or more systems or software modules that perform the monitoring of the driver's behavior and analyze the reports in accordance with this request. The exemplary system includes a bus 402 or other communication mechanism for communicating the information and a processor 404 coupled with the bus for processing the information. The computing system 400 includes a main memory, such as a random access memory (RAM) 406 or other dynamic storage device for storing information and instructions to be executed by the processor 404, and read-only memory (ROM) 408 or another static storage device for storing static information and instructions for the processor 404. A logical storage device 410 is also conveniently provided for storing instructions for execution by the processor and other information, including, for example, one or more calibration values. of the parameters monitored directly of the driver, such as the correct position of the driver's head, for example, and / or one or more calibration values ​​of the parameters monitored indirectly of the driver, such as the correct use of the seat belt, for example example. Additionally, operator interfaces are provided in the form of an input device 414 such as a keyboard or a voice recognition input that includes a microphone and logic that transforms human voice sounds into computer commands, a human readable display 412 to present information visible to the driver and a cursor control 416, such as a joystick or mouse or the like. MA.a.zuz ó / u i uaco

[0077] Exemplary embodiments described herein are related to the use of the computing system 400 for accessing, aggregating, manipulating and displaying information from one or more resources, such as, for example, from the camera in front of the driver 345.

[0078] According to one implementation, information from the camera facing the driver 345 is provided by the computing system 400 in response to the processor 404 executing one or more sequences of one or more instructions contained in the main memory 406. Said Instructions may be read into main memory 406 from another computer-readable medium, such as logical storage device 410. Logical storage device 410 may store one or more subsystems or modules to perform direct monitoring of driver behavior as shown. set forth herein and / or one or more subsystems or modules to perform indirect monitoring of driver behavior as set forth herein. Execution of the instruction sequences contained in main memory 406 causes processor 404 to perform the process steps described herein. In an alternative implementation, hardwired circuits can be used instead of or in combination with software instructions to MA.a.zuz ó / u i uyoo implement the invention. Thus, implementations of exemplary embodiments are not limited to any specific combination of hardware and software circuitry. wow

[0079] According to the descriptions herein, the term computer-readable medium as used herein refers to any non-transitory medium that is involved in providing instructions to the processor 404 for execution. Such non-transitory medium may take many forms, including, but not limited to, volatile and non-volatile media. Non-volatile media include, for example, optical or magnetic disks. Volatile media includes dynamic memory, for example, and does not include transient signals, carrier waves, or the like. Common forms of computer-readable media include, for example, a floppy disk, a floppy disk, hard disk, magnetic tape, or any other magnetic media, a CD-ROM, any other optical media, punched cards, paper tape, any other physical media with hole patterns, a RAM, PROM, and EPROM, a FLASHEPROM, any other memory chip or cartridge, or any other non-transitory readable media that can be read by a computer.

[0080] Additionally, and also consistent with the descriptions herein, the term logic, as used herein with respect to the figures, includes hardware, firmware, software running on a machine, and / or combinations of each one to perform a function or an action, and / or cause a function or action of another logic, method and / or system. Logic may include a software-controlled microprocessor, discrete logic (e.g., ASIC), analog circuit, digital circuit, programmed logic device, memory device containing instructions, and so on. The logic may include one or more gates, combinations of gates, or other circuit components. υυου

[0081] The computer system for monitoring driver behavior 400 includes a communication interface 418 coupled to the bus 402 that provides bidirectional data communication coupled to a network link 420 that is connected to the local network 422. For example, The communication interface 418 may be an Integrated Services Digital Network (ISDN) card or modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 418 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, communication interface 418 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information. IVIA / a ZUZÓ / U IU900

[0082] Network link 420 typically provides data communication over one or more networks to other data devices. For example, network link 420 may provide a connection over local network 422 to a host computer 424 supporting a database 425 storing internal proprietary data and / or to data equipment operated by a Service Provider. Internet (ISP) 426. ISP 426 in turn provides data communication services over Internet 428. Local network 422 and Internet 428 both use electrical, electromagnetic or optical signals that carry digital data streams. Signals across the various networks and signals on network links 420 and across communication interface 418, which carry digital data to and from the driver behavior monitoring computer system 400, are exemplary forms of waves. carriers that transport information.

[0083] The driver behavior monitoring computer system 400 can send messages and receive data, including program code, through the networks, network links 420 and the communication interface 418. In the exemplary embodiment connected to the Internet , the driver behavior monitoring computer system 400 is operatively connected to a plurality of external public, private, government or commercial servers (not shown) as one or more wireless services (not shown) configured to run a web application according to the exemplary modality that will be described below in greater detail. In the exemplary embodiment shown, the first server 430 is coupled with a database 450 that stores selected data received by a first wireless service, such as, for example, data from a first telematics provider, the second first server 432 is coupled to a database 452 that stores selected data received by a second wireless service, such as, for example, data from a second telematics provider, and the third server 434 is coupled to a database 454 that stores Selected proprietary data and executable code to make the web application. The driver behavior monitoring computer system 400 is operative to selectively transmit data to ινΐΛ / azuz ó / u i uacD the respective databases 450, 452, 454 via the Internet 428, ISP 426, the local network 422 and the communication interface 418, and / or to receive the selected data pushed from the databases 450, 452, 454, or by both means, according to exemplary embodiments. The received data is processed by processor 404 as it is received, and / or stored in storage device 410, or other non-volatile storage for subsequent data processing or manipulation. USOD

[0084] Although the driver behavior monitoring computer system 400 is shown in Fig. 4 as being connectable to a set of three (3) servers, 430, 432, and 434, those skilled in the art will recognize that the driver behavior monitoring computer system 400 may establish connections to multiple additional servers on the Internet 428. Each such server in exemplary embodiments includes HTTP-based Internet applications, which may provide information to the computer system. driver behavior monitoring 400 upon request in a manner consistent with these embodiments.

[0085] Selectively locate proprietary business data in database 425 within the firewall 440 is advantageous for numerous reasons including that it allows for fast comprehensive local queries without substantial network overhead. However, it is important to maintain data accuracy by performing update or refresh operations on a schedule based on the desired data characteristics or the data requirements of a particular query. MA. a.ZUZÓ / U IUUOO

[0086] The driver behavior monitoring computer system 400 suitably includes various subsystems or modules to perform direct and / or indirect monitoring of driver behavior as set forth herein. A primary objective of the present application is to provide better monitoring of driver behavior that allows fleet managers or the like to better manage their drivers. In this sense, Fig. 4a is a block diagram showing the executable logical components of the driver behavior monitoring system having a camera in front of the driver according to the exemplary embodiment. Referring now to that Figure, the logic stored in the storage device 410 (Fig. 4) is executable by the processor to perform driver behavior monitoring and reporting in accordance with the exemplary embodiment. Referring now to that figure, the logic stored in the storage device 410 (Fig. 4) is executable by the processor to monitor driver behavior and report, in accordance with the embodiment hereof. The logic stored in the storage device 410 includes the control logic 460 of the control logic stored in the non-transitory memory device. The control logic is executable by the processor to process the image data to determine an operating value of a parameter of a monitored condition of the associated vehicle, perform a comparison between a recommended range of values ​​of the parameter of the monitored condition of the associated vehicle and the operating value of the associated vehicle monitored condition parameter, and determining a compliance state of the vehicle operation according to a comparison result between the recommended value range and the operating value of the associated vehicle monitored condition parameter . The system processor can selectively generate the result data according to the result. UUDD

[0087] The logic stored in the storage device 410 further includes the facial detection logic 462 stored in the non-transitory memory device. The facial detection logic is executable by the processor to process the image data to locate one or more candidate areas of the face of an image captured by the imaging device 345 likely above a predetermined threshold stored in the memory device. transient of the system to be representative of one or more corresponding faces on the associated vehicle, and generate a set of face descriptors for each of the one or more candidate areas of the face.

[0088] The logic stored in the storage device 410 further includes the voice detection logic 464. The voice detection logic 464 is executable by the processor to identify a person associated with a set of face descriptors to each of the one or more candidate face areas according to received voice data representative of a recorded voice of one or more human passengers corresponding to the one or more candidate face areas.

[0089] The logic stored in the storage device 410 further includes the mouth movement logic 466. The mouth movement logic 466 is executable by the processor to identify a person associated with a set of face descriptors for MA.a.zuz ó / u i uaoo each of the one or more candidate areas of the face according to the voice data in combination with the received data representative of mouth movement images recorded from one or more corresponding human passengers to one or more candidate areas of the face. oops

[0090] The logic stored in the storage device 410 further includes the driver head detection logic 468. The driver head detection logic 468 is executable by the processor to process the image data to locate / determine a candidate head area of ​​an image captured by the imaging device 345 likely above a predetermined threshold stored in the non-transitory memory device to be representative of a head of an associated driver disposed in the associated vehicle, and label a portion of the image data corresponding to the candidate head area located / determined by the driver head detection logic as the driver head image data.

[0091] The logic stored in the storage device 410 further includes the driver head address logic 470. The driver head address logic is executable by the processor to process the head image data of the driver to determine an orientation direction of a head of an associated driver and generate the orientation direction data of the driver head, the orientation direction data of the driver head being representative of the determined orientation direction of the associated driver's head. uaoD

[0092] The logic stored in the storage device 410 further includes the driver head location logic 472. The driver head location logic is executable by the processor to process the driver head image data. together with the vehicle geometry data and the position data of the imaging device to determine a location of a driver's head with respect to one or more control structures of an associated vehicle and generate the head location data of the driver, which are representative of the determined location of the driver's head with respect to one or more control structures of the associated vehicle.

[0093] The logic stored in the storage device 410 further includes the driver face detection logic 474. The driver face detection logic is executable by the processor to process the image data together with the vehicle geometry data and the position data of the imaging device to determine one or more foreground objects in the imaging data and one or more background objects in the imaging data. The one or more foreground objects determined in the image data are arranged on the associated vehicle between the imaging device and the one or more background objects in the image data. The driver's face detection logic is executable by the processor to process a portion of the image data corresponding to the one or more foreground objects determined in the image data to selectively determine, from the image data , a face of the driver of the associated vehicle and generate one of: the driver's facial characteristic data representative of the selectively determined face of the associated driver, or the obstructed image data representative of an inability of the face detection location logic of the associated vehicle. driver to selectively determine the face of the driver of the associated vehicle from the image data. The driver's face detection logic is further executable by the processor to process the location data of the driver's head and a facial normal vector to selectively determine, a ινΐΛ / azuzó / u i useo from the image data, a face of the driver of the associated vehicle and generate one of: the driver's facial characteristic data representative of the selectively determined face of the associated driver, or the obstructed image data representative of an inability of the driver's face detection location logic to selectively determine the driver's face of the associated vehicle from the image data. MA.a.zuz ó / u i uacD

[0094] The driver-facing camera 345 of the exemplary embodiment is, preferably, a driver-facing video camera 510 arranged as shown in Fig. 5a on the top of the windshield 512 of the associated vehicle. In that position, the driver-facing video camera (DEC) 510 is more capable of taking images of the driver's head 520 and the area 530 surrounding the driver while also giving an advantageous view of the road ahead. for the forward facing camera. An alternative embodiment with the cameras spaced in front of the driver 345 and facing forward 346 is possible, in which case the forward focused camera (FFC) 346 is best positioned high on the windshield as shown and the camera in front of the driver 345 can be arranged in a separate housing and placed forward on the instrument panel or on the driver's side, either under the dashboard or up on the windshield as shown. Applicable vehicle unobstructed view requirements are typically met by these locations. A central vantage point is best for getting a full picture of the cabin. According to embodiments herein, one or more still and / or video images of the driver's head are used to directly monitor the driver's behavior in ways that will be described in greater detail below and, correspondingly, According to embodiments herein, one or more still and / or video images of the area 530 surrounding the driver are used to directly monitor the driver's behavior in a manner that will be described in greater detail below. MA.a.zuz ó / u i uaoD

[0095] Fig. 5b is a diagram showing the video camera in front of the driver 510 according to an exemplary embodiment herein. As shown, the driver-facing video camera 510 includes a housing 512 that supports a pair of lights, first 540 and second 542, arranged on opposite sides of a centrally located camera device 550. The pair of lights, first and second, 540, 542 are preferably infrared (IR) lights, such as IR LEDs, so that the driver and the area in the vehicle surrounding the driver may be illuminated for the purposes of recording images of the driver and the surrounding areas using the camera device 550 without obstructing the driver during the operation of the vehicle by either distracting or blinding the driver, or the like. The camera 550 is preferably angled somewhat toward the driver so that beneficial optical characteristics, such as higher resolution, near the central axis of the lens are favored. In the mode, the horizontal field of view of the lens is wide enough to see both the driver and the passenger. The horizontal field of view of the lens is also wide enough to view the driver, any passenger, and the interior of the vehicle's cabin to a large extent including, for example, the vehicle's side mirrors, as will be described in detail below.

[0096] Fig. 6a is a calibration image 600 obtained from the camera in front of the driver 345 showing an image of a driver 610, an image of a driver's seat 620 with the driver positioned thereon, an image of a properly worn seat belt 630, an image of a passenger side mirror 640 and an image of a driver's side view mirror 650. The calibration image 600 can be obtained from the images of a MA.a.zuz ó / u i uaoo human driver correctly positioned in the seat, with the seat belt correctly worn and with the driver's head being positioned in a direction that looks directly at the road ahead. In embodiments herein, one or more portions of the calibration image 600 may be used to monitor driver behavior directly using the driver-facing camera 345 according to a detected position of the driver's head within the vehicle that is being operated by the vehicle, and to monitor the behavior of the driver indirectly using the camera facing the driver 345 according to the detected aspects of the components of the interior of the vehicle that are being operated by the vehicle such as, for example, the detected aspects of the driver's seat 620, the seat belt 630, the left and right side view mirrors 640, 450 and other things that include the absence of any passenger in the calibration image 600. According to the modalities, the calibration image 600 can be obtained by taking the image of a human driver correctly positioned in the seat while the vehicle is moving at higher speeds such as, for example, greater than 40 mph, during which the driver's head posture data can be collected, thereby determining that the driver's head arrangement is facing forward. It can be assumed in modality that the average or most common (mode) of the MA. a.ZUZÓ / U IUSOD driver's head angles correspond to the 'looking forward, on the road' values ​​for this driver. It is noteworthy that a zero deflection angle can be taken as either looking directly at the camera, such as a driver's frontal view, or it can be taken as when looking forward, i.e. (typically) in line with the axis longitudinal of the driver's seat, facing forward, and the road.

[0097] Fig. 6b is a calibration image 602 obtained from the camera facing the driver 345 showing an image of the driver 610, an image of a driver's seat 620 with the driver positioned thereon, an image of a seat belt safety sensor 630' used incorrectly, an image of a passenger side mirror 640 and an image of a driver's side view mirror 650. The calibration image 602 can be obtained by placing the driver in the seat, with the belt safety device not used incorrectly and with the driver's head positioned in one direction to look directly at the road ahead. In embodiments herein, one or more portions of the calibration image 602 may be used to directly monitor driver behavior using the driver-facing camera 345 in accordance with a MA.a.zuz ó / u i uaco detected position of the driver's head inside the vehicle that is being operated by the vehicle and to indirectly monitor the behavior of the driver using the camera in front of the driver 345 according to the detected aspects of the components of the interior of the vehicle that is being operated by the vehicle, such as, for example, the detected aspects of the driver's seat 620, incorrect use of the seat belt, the seat belt buckle 632, the left and right side view mirrors 640, 650 and other things including the absence of any passenger in the calibration image 602.

[0098] Fig. 7 is an example of an image 700 obtained from the camera facing the driver 345 during operation of the vehicle such as, for example, while the vehicle is being driven, showing an image of the driver 710, an image of the driver's seat 720 with the driver positioned thereon, an image of the seat belt 730, an image of the passenger side mirror 740 and an image of the driver's side view mirror 750. The image 700 is in accordance with a embodiment of the present, which is obtained continuously as a video while the associated vehicle is being driven by the driver and is stored in the memory device as video data. Image 700 can also be obtained from UUDD is formed continuously as a sequence of photographic images taken over time and at predetermined intervals selected for example based on the speed or other operating characteristics of the vehicle while it is being driven by the driver and stored in the memory device 340 as data of a sequence of photographic images. In embodiments herein, one or more portions of the image 700 may be used to monitor driver behavior directly using the driver-facing camera 345 according to a detected position of the driver's head within the vehicle being operated. by the vehicle, and to monitor the driver's behavior indirectly using the driver-facing camera 345 according to detected aspects of the components of the interior of the vehicle that are being operated by the vehicle, such as, for example, detected aspects of the driver's seat. driver 720, the improperly worn seat belt, the seat belt buckle 732, the left and right side view mirrors 740, 750 and other things, including the presence of passengers 760, 762, and 764 in the image 700. MA.a.zuz ó / u i uaoo

[0099] As noted above, in embodiments herein, systems and methods are provided using the driver-facing camera 345 to monitor driver behavior directly in accordance with a detected position of the driver's head within the vehicle that It is being operated by the driver. The behavior of the driver being monitored includes, in the various modalities, one or more of: 1) a verification of the driver's appropriate use of the driver's side view mirror 750 and / or the passenger's side view mirror 74 0; 2) a verification of the driver's due attention to the road ahead; 3) a verification that the driver is not excessively reached by items beyond their consideration as a safe grip space, preferably an extension so that the driver is able to perform a maneuver without excessive movement of his body; and 4) a verification of the driver's head posture distribution metric.

[0100] Verification of the driver's appropriate use of the driver's side-view mirror 750 and / or the passenger's side-view mirror 740, of the driver's due attention to the road ahead, that the USOD driver is not excessively affected by items beyond their consideration as safe wingspan, and the driver's head posture distribution metric can be reported individually and / or collectively to an associated fleet management network , stored locally, or any combination of remote individual / collective reporting and / or local storage. Verification of the driver's due attention to the road ahead is used in one embodiment to adapt a lane departure warning (LDW) system to a given value of the driver's attention to the road. iviA / azu¿ó / u i uaoo

[0101] In additional embodiments herein and as noted above, systems and methods are provided using the driver-facing camera 345 to monitor the driver's behavior indirectly according to detected aspects of the components of the interior of the vehicle that is being operated by the driver. The behavior of the driver being monitored includes in the various modalities one or more of: 1) a verification that the driver is correctly wearing the seat belt; 2) a verification that the driver has proper hand placement on the steering wheel; and 3) a verification that the driver has no passengers, an appropriate passenger limit, and / or a verification that the detected passengers are authorized passengers. Using forward-facing camera (DFC) to monitor and report driver behavior

[0102] As noted above, exemplary embodiments herein are provided to monitor and report driver behavior directly through a camera in front of the driver according to a detected position of the driver's head within the vehicle being operated by the driver and to monitor and report the driver's behavior indirectly using a camera in front of the driver according to detected aspects of the components inside the vehicle being operated by the driver. In direct monitoring of driver behavior, the driver and / or the driver's head are located in the image obtained from the interior of the vehicle and the parameters of various driver behavior metrics are determined according to the driver's head located in the image. In indirect driver behavior monitoring, one or more vehicle components such as a seat belt or steering wheel are placed in the image obtained from the vehicle interior and the parameters of various driver behavior metrics are determined by inference. according to one or more located components of the vehicle in the image.

[0103] Fig. 8 is a flow chart showing a method 800 of implementing a driver behavior monitoring and reporting strategy according to an exemplary embodiment that includes a first set of steps 820 for monitoring driver behavior. driver indirectly using a camera in front of the driver according to the detected aspects of the vehicle interior components that are being operated by the vehicle and further including a second set of steps 830 to directly monitor the behavior of the driver using the camera in front of the driver. driver according to a detected position of the driver's head within the vehicle that is being operated by the vehicle. In the first set of steps 820 to indirectly monitor the driver's behavior, the vehicle cabin image data is collected and then analyzed in step 822. In the embodiment, the vehicle cabin image data is representative of the image 700 (Fig. 7) obtained from the υυου camera in front of the driver 345 during the operation of the vehicle. Thereafter, one or more actions are taken in step 824 based on the collected and analyzed cockpit image data. In the described embodiments, indirect monitoring of driver behavior does not depend on finding the location, position or posture of the driver's head in the image, but rather infers the driver's behavior from portions of the image in relation to the drivers. components of the vehicle that is being used by the driver, preferably being used in accordance with good driver behavior, such as proper use of seat belts.

[0104] Similarly in the second set of steps 830 to directly monitor driver behavior, a portion of the vehicle cockpit image data relative to the vehicle driver image is segregated in step 832 from the vehicle cockpit image data. images of the vehicle cabin collected in step 822. The segregated portion may be related to the driver's head, the driver's seat, the seat belt, the seat belt buckle, the one or more passengers, or any other item selected for monitoring as may be necessary and / or desired. From then on, it MA. a.ZUZÓ / U IU900 take one or more actions in step 834 based on the vehicle driver image portion of the cockpit image data. I. USING DFC TO MONITOR AND REPORT THE DRIVER BEHAVIOR INDIRECTLY

[0105] The behavior of the driver can be monitored, according to the embodiments described herein, using a camera in front of the driver to detect and monitor aspects of the components of the interior of the vehicle that are being operated by the vehicle, then infers the driver's behavior according to the monitored aspects of the components inside the vehicle. Indirectly monitored driver behavior is collected and stored locally in the vehicle and, in embodiments, may be reported to a central fleet management system. wow Passenger detection and counting

[0106] Drivers of commercial vehicles may have one or more unauthorized passengers accompanying the driver in the vehicle. Commercial vehicle fleet policy often prohibits or limits the passengers that can be present in their vehicles. Therefore, it would be desirable to detect if any unauthorized passengers are present in the vehicle. It would also be desirable to detect how many passengers are present in the vehicle. Furthermore, it would be desirable to identify the detected passengers present in the vehicle.

[0107] The exemplary embodiment, as shown for example in Fig. 9, provides a system and method for the detection, counting and identification of such passengers. An advantage of the exemplary modality is an ability to enforce the fleet policy, ensuring that the driver adheres to the fleet policy and that any violations of the fleet policy are recorded and reported appropriately. MA. a.ZUZÓ / U IUUOO

[0108] In the embodiment of method 900 shown in Fig. 9, the cockpit image data collection portion 822' includes a step 902 for determining a time of the cockpit image, and a step 904 collects vehicle operating data, such as, for example, vehicle speed data or the like. At step 906 the system logic finds one or more faces in the cockpit image data and also counts the number of faces found. In step 908 of the cockpit image data collection portion 822', system logic is executed to attempt to identify the one or more faces found in the cockpit image data. qqah ιη / ρζηζ / Β / γΐΛ

[0109] Next in the method 900 shown in Fig. 9, the action of taking the part 824 'includes a step 910 to determine if any of the faces located in the cockpit image data can be or have been identified. If one or more of the faces are identified, method 900 in step 920 stores an identification of the faces along with the vehicle state data collected in step 904. On the other hand, if any of the faces is not identified, method 900 in step 930 stores the determined number of faces along with the vehicle state data collected in step 904.

[0110] Furthermore, in method 900 of the embodiment, one or more of the identified faces, the determined number of faces, and / or the vehicle state data are stored locally in the system memory in the vehicle or transmitted in step 940 to a central fleet management system.

[0111] According to the exemplary embodiment, the driver-facing camera 345 uses wide-angle camera views to obtain an image 700 of the commercial vehicle cabin. This wide angle image is preferably not distorted afterwards to eliminate wide angle lens effects. The undistorted image data from the cockpit is inspected by logic 330 to first locate the faces in the image, and then count the located faces. Face detection algorithms, such as those by Viola-Jones, can be used to locate areas of candidate camera images that may be faces. Face descriptors are generated for these candidate camera image areas of located faces. The number of face areas detected, overlapping or not, and the corresponding face descriptors are generated. A threshold for facial similarity is established, below which faces are declared to be the same (via similar face descriptor vectors). Likewise, the detected faces can be compared with previously stored face descriptor vector data for drivers and passengers allowed to be in the vehicle. The face descriptor vector data of authorized drivers and permitted passengers may be stored locally in the driver behavior monitoring system or remotely in one or more databases associated with servers 142 (Fig. 1), of the central fleet management system. qqrh Ln / eznz / Β / γΐΛ

[0112] Tracking logic executed by the processor may be used to associate facial measurements with previous locations, thereby enabling person identification logic executed by the processor with a multi-area focus. Identified (or unidentified) persons are transmitted to the one or more fleet management servers 142 (Fig. 1), along with vehicle state data preferably sampled coincidentally with the person's identification. This can occur either while the vehicle is moving or while it is stopped. uaoD

[0113] The identified faces are compared to either of a database in the vehicle, or transmitted to a central management system 142 with a similar database 150 (Fig. 1). In the event that a person not registered as allowed in the vehicle is identified, a first step is taken to identify said person. If one or more identified persons is / are known to the database, a first type of event processing is performed by the driver behavior monitoring computer system. However, if the one or more identified persons is / are unknown to the database, a second type of event processing is performed by the driver behavior monitoring computer system.

[0114] The information is selectively transmitted to the fleet management system to be analyzed by a fleet manager 160 (Fig. 1) or the like. The information collected, analyzed and transmitted may include any one or more or others of: how many passenger(s) (i.e., non-driving) are present in a vehicle, whether these passengers are known or not, facial descriptors may be sent to an associated fleet management system if the identified passengers are not known, the gender of the passengers, a time of day of the image collection, a location of the vehicle at the time the cabin image was collected, snapshots of passengers, and snapshots of the vehicle interior / cabin as deemed necessary and / or desired. Unknown passengers may also be recorded by a microphone on input device 414 (Fig. 4) which may be present in the system when it is determined that the passenger is speaking. MA.a.zuz ó / u i uaoD

[0115] Fig. 9a shows an additional method 950 for detecting whether any unauthorized passengers are present in the vehicle, the number of passengers that are present in the vehicle, and the identities of the detected passengers present in the vehicle. In the embodiment, method 950 includes a series of steps that determine when passenger detection is performed, and what is detected and sent. Passenger visibility may typically be associated with the use of opening and closing the passenger door. In the example, passenger detection is performed only in response to selectable trigger events and is not performed otherwise. In the embodiment, a template image of the passenger door in open, closed, open, closed, and ajar conditions is used to detect the state of the passenger door as if it were open, closed, uncertain, or the like. The Figs. 6a, 6b and 7 for example show the driver's door (similar appearance to the passenger door) beyond the driver, and it is the edges of the driver, in a fixed location, that are used by the system according to the method 950 to determine if it is open or closed. MA.a.zuz ó / u i uacD

[0116] Method 950 is initiated in step 952 by the system of an example of the embodiment in which a series of circumstances or triggering events is determined in step 954 to proceed with method 950 to determine if any passenger is in the vehicle. If none of the trigger events are detected in step 954, the passenger detection module is not executed. However, detecting the occurrence of any one or more of the trigger events in step 754 will result in execution of the passenger detection module. In the exemplary embodiment, the activation events may include any one or more of the doors being opened (recently) and the vehicle having recently stopped; the door is being (recently) closed (at which point an image is stored) and the vehicle starts moving thereafter; when the vehicle has just started moving forward; when a predetermined time for execution arrives, such as a monitoring interval; when a stop has occurred in an unusual location, such as on a highway and the passenger door is opened. Other trigger events may be used and are contemplated in the modality. A black box data storage scheme can be used to retrieve appropriate images of passengers before the door opens or immediately after it closes. Suitability can be determined by finding a face facing forward toward the windshield where a passenger would appear.

[0117] When such circumstances occur, an image of the cabin is made in step 956 using the driver-facing camera 345 in the manner described above. All faces in this image are located at step 958 by the system logic. Facial descriptors are generated in step 960 for these faces. The descriptors are compared in step 962 with one or more than one MA. a.ZUZÓ / U IU900 in-vehicle database 340 (Fig. 3) and / or an off-vehicle database 450, 452, 454 (Fig. 4), and each face is labeled according to the comparison as known or unknown or, alternatively, labeled as allowed or not allowed. Any other suitable tag can be used as needed or desired.

[0118] Vehicle status information is collected and stored in step 964 and a passenger detection status report is then stored and / or sent in step 966 to a central database. This report contains one or more of the number of people who are present in the vehicle, their identity (with unknown John or Jane Doe status also possible), the image of the cabin, the location of the vehicle, the speed of the vehicle, the status of the door (possibly plural), the forward view, an audio recording if speech is detected from microphone or lip movement signals. UUDD

[0119] A system is provided for monitoring a condition of the permitted occupants of an associated vehicle during operation of the associated vehicle by an associated driver. The system includes an imaging device disposed in the associated vehicle, a control device, facial detection logic and control logic. The imaging device captures an image of the associated driver disposed in the associated vehicle. The imaging device also captures an image of an interior of the associated vehicle and generates image data representative of the captured image of the associated driver disposed in the associated vehicle and the interior of the associated vehicle. The control device includes a processor, an image data input operatively coupled to the processor, and a non-transitory memory device operatively coupled to the processor. The image data input is configured to receive the image data from the imaging device. The facial detection logic is stored in the non-transitory memory device and is executable by the processor to process the imaging data to locate one or more candidate face areas of the image captured by the imaging device likely above a threshold. predetermined stored in the non-transitory memory device to be representative of one or more corresponding human faces in the associated vehicle. The face detection logic is further executable by the processor to generate a set of face descriptors for each of the one or more candidate areas of the face. The stored control logic is also stored in the non-transitory memory device and is executable by the MA.a.zuz ó / u i uaoo processor to determine, based on the set of face descriptors generated for each of the one or more candidate face areas, a count of the vehicle occupants as an operational value of a parameter of the number of occupants of the permitted condition of the monitored occupants of the associated vehicle. The vehicle occupant count can be stored locally in the vehicle memory and / or transmitted to the central fleet management system. Calibrated seat belt use detection system

[0120] Many drivers do not regularly wear their seat belts, thereby compromising their own personal safety. For commercial vehicle drivers, however, not wearing a seat belt can also violate fleet policy.

[0121] Therefore, it is desirable to detect whether or not a driver is properly wearing his or her seat belt during vehicle operation. In this regard, belt use detection systems, methods and apparatus are provided as described below. MA. a.ZUZÓ / U IUSOD

[0122] Cameras are becoming ubiquitous in commercial vehicles for recording a digital loop video of the vehicles path ahead as they travel. The video is useful for accident reenactment purposes and for other commemorations of the vehicle's and driver's most recent activities in case mechanical or other problems arise. Cameras facing the driver have also been used to obtain images of the driver from time to time as necessary, for example, each time the vehicle is started so that the identity of the person in control of the vehicle can be determined at a later date. later moment. MA.a.zuz ó / u i uaco

[0123] In additional embodiments herein, camera-based systems, methods and apparatus are provided for detecting whether a seat belt is being worn. An example of the embodiment of a method for detecting whether the seat belt is worn is shown in Figs. 10, 10a and 10b. The characteristics expected from wearing a seat belt are looked for in an image 700 (Fig. 7) taken by the camera in front of the driver 345. These characteristics may include lines emanating from a point of origin or region within a predetermined portion of the image 700. Additionally or alternatively, these features may include the lines of the image within a range of angles. Additionally or alternatively, these features may include lines in the image with a range of colors between the lines, with no discontinuity, or if a discontinuity is present, where the lines end near the point of discontinuity approximately parallel to each other.

[0124] Fig. 10 is a flow chart showing an operating method of a driver behavior monitoring system having a camera in front of the driver for implementing seat belt usage detection, monitoring and the reporting strategy in accordance with an exemplary modality. Referring now to the figure, in the method 1000 of the embodiment, the cockpit image data collection part 822' includes a step 1012 that determines a time of the cockpit image and a step 1014 that collects the data of vehicle operation, such as, for example, vehicle speed data or similar. In step 1016 the system logic finds a seat belt origin point in the cabin image data and further determines a seat belt arrangement in step 1018. USOD

[0125] Below in method 1000 shown in the Fig. 10, the action taken by part 824' includes a step 1010 to determine if the driver's seat belt is used correctly. If the driver's seat belt is used properly, method 1000 in step 1020 stores an ON identification or seat belt ON status data. The ON identification or seat belt ON status data may be stored along with the cockpit image collected in step 1014 as necessary and / or desired. On the other hand, if the driver's seat belt is not used properly, method 1000 in step 1030 stores an OFF identification or seat belt OFF status data. Same as the ON identification above, the OFF identification or OFF seat belt status data may be stored along with the cockpit image collected in step 1014 as necessary and / or desired. MA.a.zuz ó / u i uaoo

[0126] Additionally, in method 1000 of the embodiment, one or more of the ON identification or the ON seat belt status data, the OFF identification or the OFF seat belt status data, and / or the image of the cabin collected in step 1014 are stored locally in the system memory in the vehicle or transmitted in step 1040 to a central fleet management system.

[0127] Fig. 10A is a flow chart showing details of a part of the operating method of a driver behavior monitoring system having a camera in front of the driver to implement the detection of seat belt use, the monitoring and reporting strategy of Fig. 10, according to an exemplary embodiment. Referring now to that figure, a calibration image 600 (Fig. 6a) of the driver 610 correctly wearing his seat belt is retrieved in step 1050 from a local memory of the system. The calibration image can be obtained in an initial step in which the driver is asked to first not wear the seat belt, and then in a second step, to wear the seat belt. Alternatively, a generic model of a correctly used seat belt 630 is retrieved in step 1050 from local memory. The image of the vehicle cabin obtained in step 1014 (Fig. 10) is compared in step 1052 against the calibration image 600 and / or against the generic model of a correctly used seat belt 630. UUDD

[0128] In step 1054 the system determines whether a seat belt is seen or otherwise detected in the image of the vehicle cabin obtained in step 1014. If a seat belt is seen in step 1054, the The system concludes at step 1056 that the driver is in fact wearing his or her seat belt. The method flow then returns to the action taken by part 824' (Fig. 10) of the method of operation of a driver behavior monitoring system according to the modality. However, if a seat belt is not seen in step 1054, a second examination is performed in step 1058 for the lightness or darkness covering the driver's body (under the head). If this area is dark, it is possible that the driver is wearing dark clothing, so no judgment can be made regarding his correct use of his seat belt. If light clothing is detected and the seat belt is not visible, the system concludes at step 1060 that the driver is not wearing the seat belt correctly. The method flow then returns to the action taking part 824' (Fig. 10) of the method of operation of a driver behavior monitoring system according to the modality. MA.a.zuz ó / u i uacD

[0129] Fig. 10b is a flow chart showing additional details of a portion of the method of operation of a driver behavior monitoring system having a camera in front of the driver to implement seat belt use detection, monitoring , and the reporting strategy of Fig. 10, according to an exemplary embodiment. Referring now to that figure, a calibration image 602 (Fig. 6b) of the driver 610 incorrectly wearing the seat belt is retrieved in step 1070 from a local memory of the system. Alternatively, a generic model of an incorrectly used seat belt 630' is retrieved in step 1070 from local memory. The image of the vehicle cabin obtained in step 1014 (Fig. 10) is compared in step 1072 with the calibration image 602 against and / or with the generic model of a correctly used seat belt 630'. υυου

[0130] In step 1074 the system determines whether a buckle 631' of an unfastened seat belt is seen or otherwise detected in the image of the vehicle cabin obtained in step 1014. If a buckle 631' of a Unbuckled seat belt is not seen in step 1074, the system concludes in step 1076 that the driver is wearing a jacket or the like. The method flow then returns to the action taken by part 824' (Fig. 10) of the method of operation of a driver behavior monitoring system according to the modality. However, if the buckle 631' of an unbuckled seat belt is seen or otherwise detected in step 1074 in the image of the vehicle cabin obtained in step 1014, the system concludes in step 1078 that the driver He's not wearing a jacket. The method flow then returns to the action taken by part 824' (Fig. 10) of the method of operation of a driver behavior monitoring system according to the modality. MA. a.ZUZÓ / U IU900

[0131] In one embodiment, a calibration image or model of the appearance of the seat belt is taken or established. A matched model of the seat belt buckle can be applied where the seat belt buckle can be visible. That is, a buckle 632 (Fig. 6b) should not be visible near the origin of the seat belt on the driver's shoulder. An alert or other action or function may be issued or otherwise begin upon detection of a seat belt that is not being worn by the driver.

[0132] The camera facing the driver 345 obtains an image 700 (Fig. 7) during the operation of the vehicle and, in this way, the camera can see or know the point of origin / region for the seat belt, which may be used to detect whether you are wearing a seat belt. Fig. 7 shows a user wearing a seat belt. These cameras see the point of origin of the seat belt and whether the seat belt is being used. The exemplary embodiment advantageously uses knowledge of the point of origin of the seat belt together with a calibration image 600 (Fig. 6a) of a driver 610 wearing the seat belt 630, or a generic model of the appearance of the seat belt ( angle, width, origin position, end location) in the image, to detect parallel lines within the appropriate width range, and starting and ending where expected. If the belt is not visible, the method of the exemplary embodiment is configured to determine whether the driver's jacket is dark, thus making a dark belt invisible, for example. In that case, the method first attempts to increase the line detection sensitivity, if this fails the method declares, for a benefit of double analysis, that the driver is wearing a seat belt. If a top is lighter and no dark belt is detected (dark relative to the light top), the modality method generates a signal that you are wearing a lighter top and no belt is detected for the top. storage in local memory and / or for transmission to the central fleet management system. wow

[0133] It is understood that the seat belt is visible as a band-shaped area of ​​different (contrasting) color, in contrast to the objects next to or behind it. When the seat belt is obscured by the person's scarf or face, a front edge line may be visible and continues upward to reconnect with the 'double-sided' segment. Even when the seat belt is obscured by the full extent of the driver's clothing or similar, the ends would still be visible and would continue and 'point' to each other approximately. It should also be noted that the seat belt is to the left / in front (in the image) of the person not present at the location, he was fastened. Thus, the system has an expectation of what the image of the belt, worn correctly, should look like: lines (parallel / simple / perhaps partially or completely obscured), running in an approximate direction, between two known points (regions) and within a certain part of the image. The system also has the expectation that the visible part of the belt looks like it does when worn behind the user. In this sense, the diagonal edges of the seat belt can be advantageously detected according to the present embodiments, using, for example, Kirsch filters or other equivalent or similar edge filters. uaoD

[0134] According to the embodiments herein, the system is not deceived or defrauded in a determination of good seat belt wearing behavior by a driver wearing a seat belt t-shirt (a shirt that has a graphic diagonal dark stripe that appears to be a seat belt he is wearing). In this mode, the system inspects the cabin image for a set of nearly parallel edges emanating from the top anchorage point of the seat belt. In another embodiment, the system inspects the cockpit image for continuous lines beyond the 'seat belt' (the fake printed seat belt) that the driver appears to be wearing. Even if the user fastens the belt behind himself, the system observes or otherwise detects a discontinuity between the real physical belt and the fake belt pattern printed on the shirt. The system, by searching for this break, is capable of detecting that the user driver is not using the seat belt correctly. MA. a.ZUZÓ / U IUUOO

[0135] Using knowledge of the point of origin (or interval), together with the calibration image 600 (Fig. 6a) of a driver 610 wearing the belt 630, or a generic model of the appearance of the seat belt ( angle, width, origin, end) in the image, also without a user present, the system is configured to detect parallel lines within the appropriate width range and the beginning and end where expected. If no belt is visible, the system checks whether the driver's jacket is dark (thus making a dark belt invisible, for example), in which case it first attempts to increase the detection sensitivity of the lines, if this fails the system declares, for the benefit of the analysis of doubt, that the driver is wearing the seat belt. If he is wearing a lighter top and the belt is not detected, the system signals this.

[0136] Alternatively, the system may detect the possibly visible (usually bright, and therefore likely clear and contrasting) buckle of the seat belt if the belt is not being used and is not fastened. The camera is, either from known geometric setup values, or in a calibration step (simply the signal where the origin point of the belt is), knows / shows where the buckle would be visible. If the seat belt is possibly not worn, the system can switch to this second mode and detect the presence of the buckle 632 (unengaged) at the point of origin as shown, for example, in Fig. 6b. The origin point is also typically fixed or snaps to a linear set of locations in the image. In one embodiment, a fixed patch is defined in the calibration image 602 (Fig. 6b) of the image where a buckle 631' of the seat belt not being used should appear, and that is where the system can search for the buckle. If the seat belt buckle is located in this area of ​​the fixed patch, the system concludes that the driver is not wearing a seat belt. Equivalently, for each driver, there is a fixed patch of the corresponding image where a correctly used buckle 631 appears (Fig. 6a). A matched set of templates of such properly fastened and unfastened seat belt images appears, can be stored and compared by the system with the actual image. Sufficient correspondence between an image in the stored sets and the DFC image patch corresponding to where the buckle can be worn, the mode system concludes that the driver is wearing, or not, his or her seat belt.

[0137] A system is provided for monitoring the use of a seat belt by a driver of a vehicle during the operation of the vehicle associated with the driver. The system includes an imaging device, a non-transitory memory device for storing safety model data comprising a recommended range of values ​​of a seat belt use parameter of a monitored seat belt used by the MA.a.zuz ó / u i uaoo condition of the associated driver of the associated vehicle, the control logic stored in the non-transitory memory device and an output. The imaging device captures an image of an interior of the associated vehicle along with an image of the associated driver disposed in the associated vehicle and generates image data representative of the captured images of the associated driver and the interior of the associated vehicle. The control logic is executable by the processor to process the image data to determine an operational value of the seat belt use parameter of the monitored seat belt use condition of the associated vehicle, perform a comparison between the range of values recommended seat belt use parameter of the monitored seat belt use condition of the associated vehicle and the operating value of the seat belt use parameter of the monitored seat belt use condition of the associated vehicle and determines a vehicle operation compliance state as one of a seat belt non-compliance state or a seat belt compliance state according to the comparison result.

[0138] In one embodiment, the non-transitory memory device stores a calibration image of a driver wearing a seat belt that has MA.a.zuz ó / u i uacD an origin point in relation to the image of the interior of the vehicle associated as the data of the safety model that comprises the recommended range of values ​​of the seat belt use parameter of the use condition of the associated vehicle monitored seat belt. Also in the embodiment, the control logic stored in the non-transitory memory device is executable by the processor to process the image data to determine, based on the calibration image having the point of origin, an arrangement of a belt in the image data as the operational value of the seat belt use parameter of the monitored seat belt use condition of the associated vehicle. MA.a.zuz ó / u i uaoD

[0139] In a further embodiment, the non-transitory memory device stores a generic model of a physical appearance of a fastened seat belt as the safety model data comprising the recommended range of values ​​of the seat belt use parameter of the associated vehicle's monitored seat belt usage condition. Also in the embodiment, the control logic stored in the non-transitory memory device is executable by the processor to process the image data to determine, based on the generic model of the physical appearance of a fastened seat belt, an arrangement of a seat belt in the image data as the operational value of the belt use parameter of the monitored seat belt use condition of the associated vehicle. UUDD

[0140] The exemplary mode system distinguishes between the type of non-use of the seat belt. These types may include, for example, buckling up behind the driver (or passenger), wearing an upper outer garment that says I'm wearing a seat belt, or simply not wearing a seat belt at all. Data related to the type of non-use is stored locally and / or transmitted to the central fleet management server, along with a photograph of the person or persons not wearing the seat belt in the vehicle. Detection of the driver's hands on the steering wheel

[0141] Many vehicle operators regularly fail to properly place their hands on the steering wheel while driving, thereby endangering their own personal safety and risking damage to the vehicle. For commercial vehicle drivers, however, incorrect, inconsistent, or lax hand placement on the wheel can also violate fleet policy. qqrh Ln / eznz / Β / γΐΛ

[0142] Therefore, it is desirable to detect whether or not a driver has correctly placed his or her hands on the steering wheel during vehicle operation. In this sense, systems, methods and apparatus for detecting the driver's hands on the steering wheel are provided, as described below.

[0143] Fig. 11 is a flow chart showing an operating method of a driver behavior monitoring system having a camera in front of the driver for the implementation of hand-on-the-wheel detection, monitoring and the reporting strategy in accordance with an exemplary modality.

[0144] Referring now to the figure, in method 1100 of the embodiment, the cockpit image data collection part 822' includes a step 1102 that determines a time of the cockpit image and a step 1104 that collects vehicle operational data, such as, for example, vehicle speed data or similar. At step 1106 the system logic finds a steering wheel shape in the cockpit image data and further searches the cockpit image data for short portions (approximately the hand width dimension) of the steering wheel that They are not visible in step 1108. MA. a.ZUZÓ / U IUSOD

[0145] Next in method 1100 shown in Fig. 11, the action taken by part 824' includes a step 1110 to determine whether the driver's hands are correctly on the steering wheel in the correct designated positions. If the driver's hands are correctly on the steering wheel in the correct designated positions, method 1100 in step 1120 stores a hands-ON identification or hands-ON steering wheel state data. The OVER identification or hands OVER steering wheel status data may be stored along with the cockpit image collected in step 1104 as may be necessary and / or desired. On the other hand, if the driver's hands are not correctly on the steering wheel or are on the steering wheel but not in the correct designated positions, method 1100 in step 1130 stores the Hands OFF identification or the Hands OFF steering wheel status data. . Same as the Hands ON identification above, the Hands OFF identification or Hands OFF steering wheel status data may be stored along with the cockpit image collected in step 1104 as may be necessary and / or desired.

[0146] Additionally in method 1100 of the modality, one or more of the hands ON identification or the hands ON steering wheel state data, the hands OFF identification or the hands OFF steering wheel state data, and / or the Cabin image collected in step 1104 is stored locally in system memory in the vehicle or transmitted in step 1140 to a central fleet management system.

[0147] Fig. 12 is an example of an image generated by the driver-facing camera of Fig. 5b and obtained by the driver behavior monitoring system during operation of the associated vehicle and showing a typical driver who has his hands on the wheel. The Official recommendations are that the driver 1210 has his left hand 1222 between clock positions 9 and 10 on the steering wheel, and his right hand 1220 between clock positions 2 and 3 on the steering wheel, which is visible in image 1202 of Fig. 12. Wide spacing is recommended due to the shape of the air bag as it expands when an accident occurs. In mode, the system seeks to find the positions of the driver's hands on the steering wheel in the image obtained by the camera in front of the driver 345. If you do not have your hands in the recommended positions, or do not have both hands on the steering wheel at all or not as frequently as required, is championed by the system as a MA.a.zuz ó / u i uaoo violation of the fleet policy, which is stored in local memory and / or transmitted to the central fleet management system. USOD

[0148] The modality takes advantage of the physical nature of steering wheels in commercial vehicles, which are almost always circular. Circular shapes are easily detected in images, even when viewed in a skewed view. The driver-facing camera 345 typically views both the driver and the steering wheel (if not the entire steering wheel, then a significant fraction of it) which appears as an ellipse. A Hough transform is used for ellipse detection (after lens distortion is accounted for) in an edge image of the camera facing the driver 345. Only the edge points in the original image need not be distorted. , which saves calculation time. The Hough transform returns the image where the steering wheel is located (elliptical shape in the undistorted image). Those border pixels are marked in the image corresponding to the steering wheel. Image pixels related to invisible portions of the steering wheel can also be marked with signals representative of information relating to what the steering wheel would be in the image, the view of it not being blocked. A mockup for the appearance of the complete steering wheel is provided here. 100 mode in the image, even though only a segment of the steering wheel is visible in the image. The driver's hands and arms may obscure portions of this image as can be seen in Fig. 12. ινΐΛ / azuz ó / u i uaco

[0149] In one embodiment, the area of ​​the image obtained by the camera in front of the conductor 345 that is searched for these edge points may be limited or otherwise reduced, which saves processing time and improves accuracy. This image search area reduction can be done based largely on knowledge of the optical and mechanical aspects of the camera and its physical installation geometry, or on an initial calibration step, when important features are found. of the truck cabin. For the purposes of helping to speed up the search for image elements, the search for the edge of the camera image in front of the driver 345 is limited in the modality in terms of both the part of the image examined and the instructions which edge must be present or are otherwise expected to be there in the reduced portion of the image to be examined (for example, the system does not expect a vertical edge at the top of the steering wheel 1230, this taken as shown). seen from the driver's point of view; in the picture the edge of the steering wheel is, in fact, approximately 101 vertical). QQRH ΙΠ / Ρ7Π7 / Ε / ΥΙΛ

[0150] The Hough transform is preferably executed on the undistorted edge image obtained from the camera in front of the driver 345 to detect the ellipses. A high edge sensitivity can be used as needed or desired, since the approximate location / appearance of the steering wheel is known, as there is only one ellipse, and it is within a limited size range. An alternative to the Hough transform is to store images of the flyer template and compare them to what is seen via DFC. The steering wheel portion of these images can be identified by the Hough transform in an initial calibration step, and then stored, after which template matching is performed to locate the steering wheel in an image, without needing to perform the transform. Hough again.

[0151] The modality therefore uses knowledge of the possible location of the steering wheel (-s, if adjustable), together with the detection of the ellipse by the Hough transform, to locate the steering wheel 1230 in an image of the camera facing the driver 1202 of the vehicle cabin. The contours of this detected ellipse 1232 are examined for the failing sections 1240, 1242, indicating the places where the 102 driver's hands 1220, 1222 are, respectively, on the steering wheel. That is, the hands are not detected directly; rather, the unseen parts of the steering wheel are taken as the location of the hands. MA.a.zuz ó / u i uacD

[0152] It can be seen, for example, in Fig. 12, that the driver's right hand 1220 interrupts the view of the steering wheel at 1240, but that on both sides thereof the steering wheel 1230 can be seen. Pixels on the invisible edge of the steering wheel 1230 are labeled, particularly on the right and top sides, and therefore the system determines where the driver's right hand 1220 is. The left hand 1222 is where the top section of the steering wheel is no longer visible. at 1242 looking left in the view shown. A knowledge of the color of the steering wheel may also be used according to the modality to help locate the steering wheel 1230 in the image 1202.

[0153] Additionally, the mode system may execute logic to follow the driver's hand movements relative to the steering wheel. For example, the system can look for active hand movement relative to the steering wheel (i.e. changing the position of the hand on the steering wheel), which can be used as a proxy for an attentive driver and recorded by the system as positive related events. with the 103 security. Episodes of an unchanging hand position on the steering wheel can be used to alert the driver or can be recorded by the safety system as relevant negative events. MA.a.zuz ó / u i uyoo

[0154] In one embodiment, one or more images of the stored template are used to determine where the steering wheel, if adjustable, may be located in the image. The one or more stored template images are compared to the address image 1230 when obtained by the camera facing the driver. The best matching template image effectively locates the steering wheel in the image. After this, the determination of the steering wheel 'seen in the gap' as described above is performed to locate the positions of the driver's hands 1220, 1222 on the steering wheel 1230 at the positions 1240, 1242 of the gaps determined in the image of the steering wheel 1232.

[0155] In addition to the above, the system of the modality can selectively perform a remapping of the steering wheel that appears elliptically in the image to a circle. This remapping corresponds to a re-projection of the steering wheel for a completely circular appearance. The blacked-out sections of the steering wheel 1240, 1242 by the driver's hands 1220, 1222 are also selectively transformed through this same remapping, and these positions 104 remapped hand angle spacing of the driver's hand can be determined. Good driver behavior suggests an angular separation of the driver's hand from about 180 degrees to about 120 degrees. The spacing of the driver's hand position on the steering wheel can be used to alert the driver or can be recorded by the system as positive or negative safety relevant events.

[0156] Fleet management or other policy violations such as: number of hands on the wheel, hand position, the percentage of time the driver holds the wheel, etc., can be detected, flagged, warned, recorded and / or measured. Variation in hand position can be used as a proxy for driver fatigue. II. USING A CAMERA IN FRONT OF THE DRIVER TO MONITOR AND REPORT DRIVER BEHAVIOR INDIRECTLY

[0157] Driver behavior may be directly monitored, according to the embodiments described herein, using an imaging device trained on the driver while the vehicle is being operated. The monitored driver behavior is collected and stored locally in the vehicle and, in the modalities, is 105 can report to a central fleet management system. MA. a.ZUZÓ / U IU900 Detecting driver attention to the road

[0158] Many drivers do not pay due attention to the road ahead. Drivers' eyes are often diverted from being directed towards the road due to various tasks to be performed while driving, such as, for example, checking the gauges on the instrument panel, checking other traffic using the vehicle's side mirrors , operating radios or other devices on or inside the vehicle cabin, and the like. This implies that the driver's eyes and, therefore, his attention are not always where they should be; that is, on the road, which has the tendency to adversely affect the safe operation of the vehicle, especially when drivers take their eyes off the road for a prolonged or extended period of time, or when their attention is frequently directed away from the road over time.

[0159] Therefore, it is desirable to detect whether a driver is paying due attention to the road ahead while driving the vehicle. In this sense, the driver's attention to the road detection systems are provided, methods and devices as shown. 106 described below. uaoD

[0160] According to one embodiment, generally, the driver-facing camera 345 of the driver behavior monitoring system is used to detect the direction in which the driver's head is facing and the system refers to this detected direction. and the location of the camera in front of the driver is oriented in such a way that the road can be seen correctly. The relative position between the camera facing the driver and the vehicle cabin structure may be based on one or more calibration images as necessary and / or desired. The systems, methods and apparatus of the modality are operable to transmit a signal to an associated central fleet management system when the driver is not oriented in such a way that the road can be correctly seen. Alternatively and / or additionally, the systems, methods, and apparatus of the embodiment are operable to store data representative of driver distraction in a local memory device when the driver is not oriented in such a way as to properly view the road. . Locally stored driver distraction data can be downloaded when the vehicle leaves the road, when the vehicle is undergoing maintenance, when the driver requests a download 107 similar. wow

[0161] The systems, methods and apparatus of the modalities monitor the driver's attention to the road according to a combination of a location of the driver's head and a facial normal vector of the driver's head. The location of the driver's head in relation to the vehicle cabin structure including, for example, the front windshield, and the facial normal vector of the driver's head are determined by the systems, methods and apparatus of the embodiments. This is beneficial, for example, when considering drivers of different heights operating the same vehicle at different times. For example, a short driver will have to look farther than a tall driver in order to clearly see the road ahead.

[0162] In the exemplary embodiment, a driver-facing camera 345 mounted on the windshield of a vehicle views the driver 520 (Fig. 5a) in the passenger cabin 530. The image taken by the camera 345 is analyzed to find the head of the driver and the way it is oriented is expressed in the exemplary embodiment as a facial normal vector 522. Standard methods for face location can be used for initial head location 108 of the driver, after which a shape regression is performed by the driver behavior monitoring system to determine where the facial landmarks are (e.g., nose, corners of the mouth, swallow points). Using these reference points, a generic model of the head is fitted, from which the facial normal vector 522 is derived, the details of which will be explained below. υυου

[0163] A monocular camera, however, cannot determine how far away an object is without more information. This being the case, the driver behavior monitoring system can determine the location of the driver's head in several ways, three of which will be described below.

[0164] According to a first method, known reference points on the driver's seat are used to measure the distance and / or height of the driver's seat, and from these distance and / or height measurements a approximate location of the driver's head. The known reference points on the driver's seat 620 (Fig. 6a) are preferably contained in the calibration image 600 (Fig. 6a).

[0165] According to a second method, one or more 109 calibration photographs are used to determine the location of the driver's head. For example, the driver may be asked to lean directly against the back of the seat completely, thereby producing a known position, in the reference snapshot 600 (Fig. 6a).

[0166] According to a third method, assuming that the driver 610 is sitting in the center of the seat 620 in the reference snapshot image 600 (Fig. 6a), his nose 611 will be in the vertical midplane of the seat 621, making that the head of the driver 520 is easily located in the image. Typical truck seats move up and down, forward and back, and their backrest reclines. The sides of the seat therefore move within a fixed plane in a certain approximation. A calibration image of the typical truck 600 is shown in Fig. 6a, and an operational image of the typical truck 700 is shown in Fig. 7. MA. a.ZUZÓ / U IUUOO

[0167] The driver-facing camera 345 may locate a point 622 (typically visible) on the side of the seat in the image such as for example in the upper left corner of the seat back over the driver's right shoulder or elsewhere such as the back of the 110 lower seat cushion just below a likely position of a driver's ID card on the right hip (not shown), and thus the mode's driver behavior monitoring system establishes a beam in space 3-D, emanating from the camera and passing through this seat point 622. In a monocular situation this would establish only the ray along which the seat point is located, and not exactly how far it is from this point 622 from camera 345. MA.a.zuz ó / u i uaoo

[0168] According to the embodiments herein, however, the ray intersects a known plane, and therefore defines a single point 622 in the 3-D space of the passenger cabin 530. After installation and a camera calibration, and if the seat location is known, the driver behavior monitoring system of the exemplary mode uses the entire 3-D coordinates of the seat calibration point. With this, the mode's driver behavior monitoring system can better establish the data used to determine where the driver's head is located in the 3-D space of the passenger cabin 530.

[0169] A similar principle can be applied according to a vehicle driver behavior monitoring system. 111 an additional modality for finding the tip of the driver's nose 611. In this modality, the driver behavior monitoring system assumes that the position of the driver's nose in the image is probably close to the vertical plane 621 that cuts the seat of the driver in half. This preamble is again translated into a line of intersection of a plane and the origin of the 3-D facial normal vector is therefore determinable in three dimensions. ινΐΛ / azuz ó / u i uacD

[0170] For the driver in front of the camera, the system adjusts to a head model of the driver's appearance, thereby obtaining a facial normal vector 522. The head model, which is generic, is rotated and expanded in 3-D space until it fits the distortion-free image of the driver's head as much as possible. The system thus has the three angles that characterize the head posture, within the limits of the generic head model, and a scaling factor. Driver head posture angles include, for example, a pitch angle of a driver's head (driver looking down or up), a deflection angle of a driver's head (driver looking left or right), and the roll angle of a driver's head (the driver tilts his head to the left or right). 112 QQRH ΙΠ / Ρ7Π7 / Ε / ΥΙΛ

[0171] The system, however, does not have or does not know the absolute distance 1640 (Fig. 16) of the driver from the camera, that is, the system does not have or does not know the location information of the driver's head at 3 -D (angles only) . To do this, typical pupillary distance limits can give the system a limit, where women have a mean pupillary distance of 61.7mm, and men have a mean pupillary distance of 64.0, both with a standard deviation of ~3.5mm. This produces a head distance within ~+ / -10% for ~95% of the general human population. That is, in the embodiment, the system first preferably searches for the gender of the driver, then takes the corresponding Interpupillary distance 1630 from the center of the eye 1620 to the center of the eye 1622 (Fig. 16) and relates the spacing of the eye to the head in the image at the distance of the camera. Since the system has the head posture angles, the system can obtain the interpupillary distance in pixels as if the driver were directly facing the camera. Then, using the pixel size, the system determines the interpupillary distance in meters and applies the focal length of the lens. Through similar triangles, the system calculates distance from the head to the camera as: 113 Distance from head to camera = (focal length of the UUDD lens * pupillary distance of gender) / (orientation of the camera in the Interpupillary distance of the image).

[0172] For example, if there are 20 pixels separating the pupils (or eye centers 1620, 1622, taken as proxies for the pupils), and the pixels are 4 microns in size, then there are 80 micrometers between the pupils. If, in addition, the focal length of the lens is 2 millimeters, and the gender of the driver is determined as male, then the distance from the driver's head to the camera is (2 mm x 64 mm / 80 micrometers) or 1.6 meters . Given the variability in eye spacing, one can allow for this uncertainty in the final head location, and 'soften' the criteria for out-of-position alerts.

[0173] With distance, the system is able to locate the driver's head in 3-D space and then use the direction of the facial normal vector to relate it to the vehicle cabin, mirrors, gauges, road, etc. . As the facial normal vector 522 typically originates at the tip of the nose 611, the distance from the camera to the head is known, and the angle of the head through the location of the tip of the nose in the image is also known. 114 is known, the system of the exemplary embodiment calculates location of the facial normal vector in space, and verifies that the facial normal vector points or is otherwise directed to the desired regions around the driver, such as mirrors, roads, adjacent lane when passing, etc. MA. a.ZUZÓ / U IU900

[0174] In general, the driver behavior monitoring system of the modality monitors the facial normal vector over time and compares the monitored facial normal vector with the correctly directed statistical predetermined facial normal vectors. The facial normal vector information is stored locally in the system memory along with the results for comparison over time. This data and results can be transmitted to the central fleet management system as necessary or desired.

[0175] Fig. 13 is a flowchart showing a method 1300 for monitoring the driver's attention to the road according to a combination of a location of the driver's head and a facial normal vector of the driver's head. An image of the vehicle cabin area is obtained in step 1310. A human head is detected in the image in step 1320. In step 1330, 115 determines the location of the human head with respect to the camera facing the driver 345 and / or in relation to the various components of the vehicle cabin. The facial normal vector of the detected human head is determined in step 1340. An estimated distance between the camera and the driver's head is determined in step 1350. Next, in step 1360, the driver's attention to the camera is monitored. road over time using the facial normal vector of the head in combination with the determined location of the head, where the determined location of the head is used as the base point of the facial normal vector for monitoring.

[0176] In a further embodiment, a self-calibration function may be performed by collecting statistics of where the driver is looking when driving at highway speeds over time. It can be assumed that the driver is predominantly facing forward when the vehicle is moving over a bit of speed, that is, the driver is most likely paying attention when moving quickly, and the direction of the most frequent or average normal vector will correspond to the straight path to follow. Therefore, the modality system collects normal vector statistics either by a histogram method, an average USOD 116 recursive stance angle method or a combination of the histogram and recursive average stance angle. In the histogram method, a histogram is created and filled for each of the sets of driver head posture normal vector angles that describe the orientation of the driver's head, that is, a pitch histogram ( the driver looking down or up), a drift histogram (the driver looking left or right), and a roll histogram (the driver tilting his head to the left or right). Normal vector statistics are collected for a predetermined time, such as 1 minute, after which the system takes the most complete histogram interval as corresponding to a forward-facing direction of the driver's head posture. . Alternatively, the system recursively averages the head posture angles and determines the average value as the representation of the driver's head facing forward, again allowing the averages to run long enough and only when the vehicle It moves fast enough. MA.a.zuz ó / u i uaoo Obstructed front-facing camera detection:

[0177] Knowing that the camera in front of the driver 345 of 117 In accordance with the terms hereof, they vigilantly observe drivers at all times during vehicle operation, some operators may choose to attempt to disable the camera for various reasons including, for example, to conceal violations or errors of the safety policy. the fleet, or similar. However, the functionalities of the driver-facing camera largely depend on a clear view of the driver, therefore detecting a clear view of the driver is highly desirable for the proper functioning of the detection and reporting of the driver's modalities. present. MA. a.ZUZÓ / U IUSOD

[0178] Therefore, it is desirable to detect whether or not a driver is trying to disable the camera in front of the driver. In this regard, obstructed front-facing camera detection systems, methods and apparatus are provided, as described below. An advantage of these embodiments is that proper operation of the camera facing the driver is ensured, thereby fully supporting the many functionalities of the various exemplary embodiments described herein.

[0179] According to one embodiment, generally, the driver-facing camera 345 of the vehicle monitoring system 118 driver behavior is used to detect the driver's head in the vehicle cabin during vehicle operation. In the embodiment, the camera facing the driver is complemented by face detection logic to determine the face of the driver of the vehicle. The exemplar mode logic is executed to monitor the continuous availability of a visible face, of approximately unchanged appearance, when the vehicle is in motion. The logic of the exemplary mode is executed to generate a signal of a detected loss of verification of the operator if the face is not visible and / or can be determined when the vehicle is moving. MA.a.zuz ó / u i uacD

[0180] In a further embodiment, the logic of the exemplary embodiment includes driver face finding functionality that is executed to utilize the foreground-background methods of object identification. The relatively static nature of the driver-facing camera 345 fixedly mounted to the vehicle roof support member 512 (Fig. 5a) allows foreground-background object identification methods to monitor the continued availability of the camera. face of a visible driver, of approximately unchanged appearance, when the vehicle is in motion. Initially, the background pixels; that is, the pixels that are considered to be 119 do not change due to only small changes in their value, they persistently cover a sufficiently high percentage of the region, or of the still image, where the driver's face is expected to be seen. However, when the background pixels begin to persistently cover a sufficiently high percentage of the region, or even the image, where the driver's face is expected to be seen, the system logic determines that the image is not an image. live view of the driver and that the camera can therefore be considered to be obstructed or otherwise blocked. If the face is not visible when the vehicle is moving, a loss of driver verification signal is generated and is selectively transmitted to the central fleet management system or stored locally in the system memory. UUDD

[0181] According to one embodiment, the driver face detection logic stored in a non-transitory memory device of the driver behavior monitoring and information system is executable by a processor of the system to process the location data of the driver. the driver's head and a facial normal vector determined as described above to selectively determine, from the image data, a face of the driver of the associated vehicle and generate one of: driver facial characteristic data representative of the 120 selectively determined face of the associated driver, or obstructed image data representative of an inability of the driver face detection location logic to selectively determine the face of the driver of the associated vehicle from the image data.

[0182] The driver behavior monitoring and information system of the embodiment includes an input operatively coupled to the processor, the input selectively receiving from the associated vehicle a vehicle motion signal and / or an active human control signal representative of motion. of the associated vehicle.

[0183] The control logic is executable by the processor to selectively generate, in response to the input receiving the vehicle motion signal and the obstructed image data being generated, obstructed view data representative of an obstruction between the imaging device and the associated driver disposed in the associated vehicle.

[0184] The driver behavior monitoring and reporting system of the modality also includes the uaoD logic 121 driver face detection stored in the non-transitory memory device. The driver face detection logic is executable by the processor to process the imaging data together with the vehicle geometry data and the position data of the imaging device to determine one or more foreground objects in the data. of images and one or more background objects in the image data, the one or more foreground objects determined in the image data being arranged on the associated vehicle between the imaging device and the one or more background objects in the image data.

[0185] The driver face detection logic is further executable by the processor to process the portion of the image data corresponding to the one or more determined foreground objects in the image data to selectively determine, from of the image data, a face of the driver of the associated vehicle and generate one of: driver facial characteristic data representative of the selectively determined face of the associated driver, or obstructed image data representative of an inability of the detection location logic driver face to selectively determine the driver face of the associated vehicle 122 from the image data.

[0186] The driver behavior monitoring and reporting system of the embodiment further includes an input operatively coupled to the processor, the input selectively receiving from the associated vehicle a vehicle motion signal representative of the motion of the associated vehicle. In the embodiment, the control logic is executable by the processor to selectively generate, in response to the input received by the vehicle motion signal and the obstructed image data being generated, obstructed view data representative of an obstruction. between the imaging device and the associated driver disposed in the associated vehicle.

[0187] Fig. 14 is a flow chart showing a method 1400 for monitoring the presence of a driver's face according to an exemplary embodiment. The time is determined in step 1402 and an image of the vehicle cabin is obtained in step 1404. The time may be associated with cockpit image data as necessary or desired. The cockpit image data is searched in step 1406 to find a human face, at the approximate location where the driver's face is expected to be. 123 QQRH ΙΠ / Ρ7Π7 / Ε / ΥΙΛ

[0188] A determination is made in step 1410 whether a driver's face is found in the cockpit image in step 1406. If the face is not found a further determination is made in step 1412 whether the vehicle is is moving. If the face is not found and the vehicle is moving, an alert signal is generated in step 1414, and the alert signal is selectively transmitted in step 1416 to the central fleet management system. Alternatively, the alert signal may be stored locally in the memory of the mode's driver behavior monitoring system. Driver's head out of position

[0189] Many vehicle operators reach for items while driving, such as, for example, control knobs on the dashboard, cups stowed in nearby cup holders, maps or other items stored in a center console pocket or in the door bag next to the driver's seat, or similar. This is normal ongoing behavior. However, it has been found that gaining access to distant objects while driving increases the chances of an accident by a factor of about eight (8). 124

[0190] Therefore, it is desirable to measure and alert an out-of-normal head position, as this correlates with overreaching. The driver's head position is used in the exemplary embodiment as a proxy for the driver's reach and, in particular, the driver's head position is used in the exemplary embodiment as a proxy for the driver's overreach thereby generating a representative signal of this monitored driver behavior.

[0191] The exemplary embodiment to be described herein provides a verification that the driver does not excessively reach for the items beyond the space considered safe grasping, preferably an extension of a reaching maneuver capable of being performed by the driver without movement. excessive body. Understanding the head position of the typical driver and alerting when the driver overreaches according to exemplary modalities is beneficial in helping prevent accidents caused by driver distractions.

[0192] The driver behavior monitoring system of the exemplary embodiment uses the camera facing the driver 345 to locate and measure the position of the driver's head. The logic that executes the monitoring system MA.a.zuz ó / u i uyoo Driver Behavior 125 uses recursive measurement equations to determine the mean and variance of the set of given driver head positions. The execution logic generates an alert or warning signal when a driver's head position deviates from the mean position by more than a predetermined number of standard deviations in any axis (xy- or z-) and when this occurs. deviation for a predetermined minimum period of time. The predetermined number of standard deviations and the predetermined minimum time to be out of position are parameters that can be adjusted or otherwise selectable by the operator or fleet system manager. Typical values ​​of these adjustable parameters can be two (2) standard deviations, essentially covering approximately 95% of a normally distributed variable, and for approximately 1-2 seconds. The driver's head out of position events are determined and recorded in the local memory device of the driver behavior monitoring system of the exemplary mode. The driver's out-of-position behavior is recorded by camera 345 and may be stored along with other data related to the operation of the vehicle at the time the driver's head was out of position such as, for example, speed data. of the MA. a.ZUZÓ / U IU900 126 vehicle or similar. An indication of head out of position combined with a high vehicle speed indication from the vehicle speed sensors may be used by the system to grade or otherwise flag the occurrence of head out of position more negatively than by For example, a head out of position indication in combination with a very low vehicle speed indication from the vehicle speed sensors. Stopping the vehicle to reach objects beyond the driver's safe grip space is graded or otherwise flagged by the driver behavior monitoring system in the exemplary mode considered to be good driver behavior. In contrast, continued operation of the vehicle at highway speeds, for example, while reaching for items beyond the space considered safe for the driver to grasp is graded or otherwise flagged by the driver behavior monitoring system of the exemplary embodiment. which is bad driver behavior. Other conditions of one or more vehicles may be monitored and combined with the driver's head position used in exemplary embodiments as a proxy for the driver's reach to determine a level of driver behavior. UUDD driver on a scale of good to bad. 127 qqrh Ln / eznz / Β / γΐΛ

[0193] Fig. 15 is a flow chart showing a method 1500 for monitoring the position of the driver's head used as a proxy for the driver's reach and, in particular, used as a proxy for the driver's overreach , according to an exemplary modality. The time is determined in step 1502 and an image of the vehicle cabin is obtained in step 1504. The time may be associated with the cabin image data as necessary or desired. The cockpit image data is searched in step 1506 to find a human head, preferably the driver's head. The location of the driver's head is stored in a local memory so that, in step 1508, the mean and variance of the driver's head position can be determined over a predetermined time interval.

[0194] A determination is made in step 1510 if the driver's head position is outside the mean and / or variance values ​​determined in step 1508. In one embodiment, the determination made in step 1510 of whether the driver's head position is outside the mean and / or variance values ​​determined in step 1508 includes determining whether the 128 driver's head position is outside the mean values ​​and / or variance for a predetermined period of time, which may be selectable by the operator or fleet manager. A head position alert signal is generated in step 1530 indicating that the driver's head position is outside the mean and / or variance values ​​for a predetermined period of time. A video image of the driver is recorded in step 1532 and the head position alert signal and the video image of the driver are selectively transmitted to the central fleet manager in step 1534. MA.a.zuz ó / u i uaoo

[0195] Alternatively, the head position alert signal and the driver's video image may be stored locally in the memory of the driver behavior monitoring system of the modality.

[0196] Fig. 15a is a flow chart showing a method 1550 for determining whether the driver's head is out of position according to an exemplary embodiment, with a particular approach to collecting statistics of a normal position of the driver's head. driver, such as, for example, while the vehicle is moving sufficiently fast and for a period of time 129 long enough, before evaluating the driver's head out of position according to the statistics collected. A timer is initialized in step 1560 and the driver's head posture statistics are collected in step 1562. Preferably, the driver's head posture statistics are collected when the vehicle is moving sufficiently fast and for enough time. time. The values ​​of the mean and variance of the driver's head posture need, in the exemplary embodiment, some time to develop before having any practical value such as, for example, on the scale of approximately one (1) minute at speed. Only after the mean and variance values ​​of the driver's head posture are collected and developed in step 1562 does the modality system know what 'regular' driving is for this driver, and only then The system performs the driver's head out of position test. This first test consists of taking images of the driver to obtain in step 1564 a current image of the driver. In step 1570 the comparison is made between the current measured head posture values ​​(deflection, pitch, roll, location) and the average values ​​of these driver head posture angles including, for example, a pitch. of the driver's head (driver looking down or up), 130 a deviation of the driver's head (driver looking left or right) and the sway of a driver's head (driver tilting his head to the left or right) developed in step 1562. If any of these deviates by more than a selectable number of standard deviations, preferably approximately two (2) standard deviations from the corresponding mean, the system considers the driver's head to be out of position. A timer is started at step 1572 when the head is out of position. If the timer value exceeds a threshold as determined in step 1574, an alert is issued in step 1580. When the head is not out of position, the timer is reset to zero in step 1582. MA.a.zuz ó / u i uacD

[0197] According to the exemplary embodiment, the control logic of the driver behavior monitoring and information system is executable by a processor of the system to determine, during a predetermined detection time, a central value of a facial normal vector of the driver of a vehicle and determining, over the predetermined detection time, a spread of the central value of the facial normal vector. The mean of the head position value of the facial normal vector can be determined and a variance of the facial normal vector can be determined. 131 be determined to produce a standard deviation of the driver's head position as the square root of the variance. USOD

[0198] A memory device stores, as the driver's road attention parameter of the safe attention model data, a range of recommended values ​​of a parameter of the driver's head out of position of the driver's attention condition. driver monitored as a selectable multiple of the determined standard deviation of the facial normal vector.

[0199] The control logic stored in the non-transitory memory device is executable by the processor of the driver behavior monitoring and reporting system to process the facial normal vector to determine an operating value of the driver's attention to road parameter of the monitored driver attention condition of the associated vehicle, and perform a comparison between the recommended range of values ​​of the driver head out of position parameter of the monitored driver attention condition of the associated vehicle and the determined operating value of the parameter of the driver's head position of the monitored driver of the attention condition of the monitored driver of the 132 associated vehicle. QQRH Ln / Pznz / Β / ΥΙΛ

[0200] The control logic stored in the non-transitory memory device is further executable by the processor of the driver behavior monitoring and reporting system to determine a driver distraction value as a driver distraction value.

[0201] The control logic may further determine the compliance state of the vehicle operation in a binary sense as one of a driver distraction state according to a first result of the comparison between the recommended value range and the operating value. determined from the driver's head out of position parameter of the monitored driver attention condition of the associated vehicle, wherein the processor generates the driver distraction data according to the first result or a driver distraction state according to a second result of the comparison between the recommended value range and the determined operating value of the driver's head position parameter out of position of the monitored driver attention condition of the associated vehicle. 133 Driver Head Posture Distribution Metric

[0202] An aspect of good driving behavior can be characterized as the driver being in his or her appropriate, individual driving position, that is, able to hold the steering wheel, able to see the road ahead, able to see the mirrors , positioned within reach of the pedals, and the like. Essentially, good body position within the vehicle usually leads to optimized driver performance. Deviations from these operating positions are associated with an increased risk of accidents, up to a factor of approximately eight (8) as noted above. Another aspect of good driving behavior can be characterized as the driver actually looking where he should when he drives. For example, mirrors must be used when reversing, so eyes off the road ahead under these conditions are acceptable, and eyes on one of the vehicle's mirrors are desired. Light traffic while moving forward could force the driver to scan the road ahead often with periodic scans of the mirrors, but paying particular attention to the road ahead. However, in heavy traffic situations, they will probably require more scans of the side mirrors than with MA.a.zuz ó / u i υυου 134 little traffic Lane changes are helpfully preceded by looking at the lane you are entering.

[0203] It is desired, therefore, to detect improper behavior or deviated head direction, in particular against the background of the current driving maneuver, and to use this as a monitored behavioral event. The driver can be alerted by the system when inappropriate or irregular behavior occurs. A signal can be generated when inappropriate or deviant behavior occurs and data representative of the signal can be stored locally in the vehicle-mounted monitoring system or transmitted to the central fleet management system. Still images or video images of the vehicle cockpit may be recorded when the improper or deviant behavior occurs and data representative of the cockpit images taken during the improper or deviant behavior may be stored locally in the vehicle monitoring system. mounted on the vehicle or transmitted to the central fleet management system. In one embodiment, the resolution / compression quality of the driver behavior recorded by the camera in front of the driver can be adjusted during the improper or deviant behavior to improve or otherwise increase the quality of the video to reflect 135 that this is a behavioral event of the driver's head posture. MA. a.ZUZÓ / U IUSOD

[0204] The driver behavior monitoring system of the modality determines a head posture of a driver using the camera in front of the driver, logic and a processor that executes the logic, determines a distribution of the head posture with time, and monitors the distribution of head posture, to alert the driver when it deviates from a desired or usual distribution. An alert signal may be generated and / or an alert event may be triggered to store data related to the alert signal indicating that the head posture deviates from the desired or usual distribution. The alert signal and / or data related to the alert signal may be transmitted to the central fleet management system.

[0205] Generally, the system observes the driver's head posture (facing direction) using the camera facing the driver 345. The spatial distribution of the driver's head posture is collected over time and generates a histogram 3 -D of the roll, pitch and deviation of the head that are generated. The driver behavior monitoring system is then capable 136 of verifying that there is a (desired and appropriate) change in the histogram when the driver is engaged in a vehicle reversing activity, when engaged in a turning activity (looks left when turning left, for example) and when performing other actions with the vehicle. By means of change detection methods, significant deviations from the driver's normal posture distribution can be detected from the collected head posture data, and the detected deviations can be marked, such as by generating of a signal of deviation of the driver's head posture. uaoD

[0206] In one embodiment, the histogram is operable on two time scales. That is, the histogram is operable on a long time scale, for learning or developing 'average' driver behavior or otherwise developing driver behavior from what is happening now. The two histograms are compared in modality.

[0207] Figure 16 is a diagram showing an image 1600 (not taken by the driver-facing camera of the embodiments) of a cockpit 1610 of an associated vehicle showing the driver-facing camera 345 in accordance with 137 the modality taking images of a correctly seated driver 1612 properly looking at the vehicle mirror 1650. The driver behavior monitoring system adjusts a head posture model shown in the drawing figure as a posture vector of the driver's head 1660 originating from the driver's nose 1624. This vector 1660 can be visualized as a rigidly fixed handle, connected to a generic 3-D model of the face. The face model is tilted, rotated, and angularly and scaled until it fits the observed face as closely as possible. The 3-D anchors that correspond to the handle are the head posture. It will be appreciated that the head posture model leverages and otherwise includes the driver's head location information, the driver's head roll information, and the driver's head heading and deflection information. MA.a.zuz ó / u i uacD

[0208] As described above, for the camera facing the driver 345, the system adjusts to a head model of the driver's appearance, thereby obtaining a facial normal vector 1660. The head model, which is generic, it is rotated and zoomed in 3D space until it fits the undistorted image of the driver's head as much as possible. The system of this 138 way has the three angles that characterize the posture of the head, within the limits of the generic head model, and a scale factor. Driver head posture angles include, for example, the pitch angle of a driver's head (driver looking down or up), the driver's head deflection angle (driver looking left or to the right), and a driver's head roll angle (driver tilting his or her head to the left or right).

[0209] The system, however, does not have or does not know the absolute distance 1640 (Fig. 16) from the camera 345 to the driver 1612, that is, the system does not have or does not know the location information of the driver's head in 3-D (angles only). Typical pupillary distance limits 1630 can give the system a limit, where women have a mean pupillary distance of 61.7mm, and men have a mean pupillary distance of 64.0, both with a standard deviation of ~3.5mm. This produces a head distance within ~+ / -10% for ~95% of the general human population. That is, in the modality, the system first preferably searches for the gender of the driver, then takes the corresponding Interpupillary distance 1630 from the center of the eye 1620 to the center of the 139 of the eye 1622 and relates the image of the eye spacing of the head to the distance from the camera. Since the system has the head posture angles, the system can determine or otherwise calculate the interpupillary distance 1630 in pixels as if the driver 1612 were directly in front of the camera 345. Then, using the size of pixel, the system determines the Interpupillary distance 1630 in meters, applies the focal length of the lens. Through similar triangles, the system calculates the distance between the camera 345 and the driver's head 1612 as: Head-to-camera distance = (lens focal length * gender Interpupillary distance) / (facing the camera in the image Interpupillary distance). UUDD

[0210] For example, if there are 20 pixels separating the pupils (or eye centers 1620, 1622, taken as proxies for the pupils), and the pixels are 4 microns in size, then there are 80 micrometers between the pupils. If, in addition, the focal length of the lens is 2 millimeters, and the driver's gender is determined as male, then the distance from the camera to the driver's head is (2 mm * 64 mm / 80 micrometers) or 1.6 meters .

[0211] With the distance, the system is able to locate the 140 driver's head in 3-D space and then use the direction of the facial normal vector 1660 to relate to the vehicle cabin, mirrors, gauges, road, etc. As the facial normal vector 1660 typically originates at the tip of the nose 1624, the distance from the camera to the head is known, and the angle of the head through the location of the tip of the nose in the image is also known. known, the system of the exemplary embodiment calculates the location of the facial normal vector in space and verifies that the facial normal vector points or is otherwise directed at or to the desired regions around the driver, such as mirrors, roads, lane adjacent to the driver. pass, etc. υυου

[0212] The system may collect data during a selectable period of time, such as, for example, during the last 120 seconds of the driver's head posture, entering this collected data into a multidimensional histogram stored in the local memory of the driver. system. It is preferred that a supplementary circular list with a pointer to the oldest input computational structure can form the data storage backbone that feeds this histogram.

[0213] The histogram can then be compared to a 141 safe condition observed. The observed safe condition may possibly be derived from the statistics of one or more accident-free time histories, or from one or more predetermined sets of statistics from the accident-free time history models. Furthermore, the histogram can be compared to a desired histogram of the fleet associated with the vehicle. Examples of histogram comparison are described, for example, in Serratosa F., Sanroma G., Sanfeliu A. (2007) A new algorithm to calculate the distance between multidimensional histograms In: Rueda L., Mery D., Kittler J. (eds) Advances in pattern recognition, image analysis and applications. CIARP 2007. Lecture Notes in Computer Science, vol 4756. Springer, Berlin, Heidelberg, the teachings of which are incorporated herein by reference.

[0214] Fig. 17 is a flow chart showing a method of operation of a driver behavior monitoring system having a camera in front of the driver to detect, monitor, and report whether the head posture distribution of the driver is changing significantly or unacceptable by implementing a driver attention strategy to the road in accordance with an exemplary modality. With MA.a.zuz ó / u i uacD 142 now referring to the figure, in the method 1700 of the embodiment, the driver image data collection part 832' includes a step 1702 for determining a time of the driver image, and a step 1704 collects the driver image . In step 1106 the system logic determines information relating to the operation of the vehicle such as, for example, vehicle speed data or the like, and the logic also determines the driver's head posture. The historical driver head posture data is updated in step 1708 with the newly acquired driver head posture. wow

[0215] A determination is made in step 1710 whether the collected historical data differs from a predetermined desired distribution during a given vehicle state. If the collected historical data does not differ from the predetermined desired distribution for the given vehicle state, no action is taken. However, if the collected historical data does differ from the predetermined desired distribution for the given vehicle state, then the method 1700 generates in step 1730 a head posture alert signal and / or generates the head posture alert data. the head posture. A video image of the driver is recorded or otherwise collected in step 1732, and the driver's posture alert signal 143 head and / or head posture alert data is selectively transmitted in step 1734 along with the driver's video image to a central fleet management system or the like. Alternatively, the driver's video image and head posture alert signal and / or head posture alert data may be selectively stored in a memory device of the driver monitoring system local to the vehicle. . MA.a.zuz ó / u i uyoo

[0216] Fig. 18 is an example of a head posture distribution map 1800 according to an exemplary embodiment. As shown in that figure, a visualization and analysis framework of head pose distribution can be realized in spherical coordinates, assigning named sites. The assigned locations may include, for example, a location of the vehicle radio 1822, a location of the footwell on the right and left side of the vehicle 1824 and 1826, a location of the driver's knees 1825, a location of a passenger in the vehicle 1828, a location of the left and right mirrors of the vehicle 1830 and 1832, a location of the vehicle's sun visor 1850, or a location of the road ahead 1850. A color-tinted heat map (i.e., histogram ) can indicate how often each location is oriented as shown in 144 that figure where the heat map that has the highest driver focus intensity is outlined with the x markers of the path ahead, presumably, often seen ahead of the vehicle. Portions of the map can be associated with labels - for example, when the radio station is being changed and the driver is not looking forward in the normal posture, and somewhat to the right, then the area of ​​the map that is being oriented may be marked as radius (or the probability that the radius increases). Similar labeling schemes can be used for the mirrors, this time activated by a series of flashing turn signals, and the driver turning left or right, in the direction of the turn signal. MA. a.ZUZÓ / U IUUOO

[0217] It should be noted that the safe driving position may vary, temporarily or in the long term. For example, the user may need to adjust a control that is further away (e.g., possibly a fan) or the user may change the seating position (e.g., to relieve back pain). Therefore we need to reset the histogram or mask the measurement values ​​when these changes occur, possibly temporary, possibly persistent. 145

[0218] Fig. 19 is a basic flowchart showing a method 1900 for comparing driver head posture histograms and determining and reporting deviations and / or changes between driver head posture histograms. driver according to a modality. Turning now to the Figure, method 1900 determines incorrect or deviated driver head steering behavior based on a driver head posture distribution metric. The method 1900 includes a start step 1910 that thereafter initiates a step 1912 of the associated vehicle driver and cockpit imaging system and obtains the driver image data. The driver's head posture is measured in step 1914 and a histogram of the driver's head posture is created for the last n seconds of the driver's head image capture in step 1916.

[0219] Next, in step 1920 the system determines whether the histogram shows a difference between the desired driver behavior and the actual driver behavior. If there is no difference between the desired driver behavior and the actual driver behavior, or if the difference is within predetermined limits, the system repeats step 1912 whereupon the system again takes images of the driver. MA. a.ZUZÓ / U IU900 146 and the cabin of the associated vehicle and obtains new image data from the driver. On the other hand, if there is a difference between the desired driver behavior and the actual driver behavior, or if the difference is outside the predetermined limits, the system initiates step 1922 after which the system generates the driver distraction signal. driver as determined based on the driver's head posture distribution metric. MA.a.zuz ó / u i uaoo

[0220] Fig. 19a is a flowchart showing a method 1950 for determining whether the driver's head is out of position according to an exemplary embodiment, with a particular approach to collecting statistics of a normal posture of the driver's head. driver such as, for example while the vehicle is moving sufficiently fast and for a sufficiently large period, before evaluating the driver's head posture according to the collected statistics. A timer is initialized in step 1960 and the driver's head posture statistics are collected in step 1962. Preferably, the driver's head posture statistics are collected when the vehicle is moving sufficiently fast and during enough time. The values ​​of the mean and variance of the posture 147 of the driver's head need, in the exemplary embodiment, some time to develop before they have any practical value such as, for example, on the scale of approximately one (1) minute at speed. Only after the mean and variance values ​​of the driver's head posture are collected and developed in step 1962 does the modality system know what this driver's 'regular' driving is, and only then does the system performs the driver's head posture test. This test consists of taking images of the driver to obtain in step 1964 a current image of the driver. In step 1970 a comparison is made between the values ​​of the current measured head posture (deflection, pitch, roll, location) and the average values ​​of these angles of the driver's head posture including for example the pitch of the head. driver's head (driver looking down or up), a deviation of the driver's head (driver looking left or right) and the sway of a driver's head (driver tilting his head to the left or to the right). right) developed in step 1962. If any of these deviate by more than a selectable number of standard deviations, preferably approximately two (2) standard deviations from the corresponding mean, the system considers the driver's head to be outside of USOD 148 position. A timer is started at step 1972, when the head is out of position. If the timer value exceeds a threshold as determined in step 1974, an alert is issued in step 1980. When the head is not out of position, the timer is reset to zero in step 1982. MA. a.ZUZÓ / U IU900

[0221] Fig. 20 is a flow chart showing a method 2000 for comparing head posture distribution maps and determining and reporting deviations between the actual map and a desired one, the appropriate situation, the according to an exemplary modality. The modality has a particular focus on collecting statistics from a normal position of the driver's head such as, for example, while the vehicle is moving fast enough and for a sufficiently long period, prior to evaluating the driver's head. driver out of position according to collected statistics. A timer is initialized in step 2060 and the driver's head posture statistics are collected in step 2062. Preferably, the driver's head statistics are collected when the vehicle is moving sufficiently fast and for a sufficient time. . The values ​​of the mean and variance of the driver's head posture in the 149 exemplary modality, they need a little time to stabilize before they have any practical value such as, for example, on the scale of approximately one (1) minute at speed. Only after the mean and variance values ​​of the driver's head posture are collected and developed in step 2062 does the modality system know what this driver's 'regular' driving is, and only then The system performs the driver's head out of position test. This first test consists of taking images of the driver to obtain a current image of the driver in step 2064. In step 2070 a comparison is made between the current measured head posture values ​​(deflection, pitch, roll, location) and a histogram of driver head posture angles including for example driver head pitch (driver looking down or up), driver head deflection (driver looking left or right) and a driver's head roll (driver tilting his head to the left or right), developed in step 2062. If any of these deviate by more than a selectable number of standard deviations, preferably about two (2 ) standard deviations from the corresponding mean, the system considers that the driver's head is out of position. a timer MA.a.zuz ó / u i uacD 150 is incremented at step 2072 when the head is out of position. If the timer value exceeds a threshold as determined in step 2074, an alert is issued in step 2080. When the head is not out of position, the timer is reset to zero in step 2082. Driver's eyes on the road with adaptive LDW warning margin

[0222] Drivers who do not properly look at the road when driving forward probably need more time to react to a dangerous situation. Therefore, it is desirable to adjust the warning parameters for a hazard detection system, such as a lane departure warning device or a radar-based distance keeping aid, so that the driver is warned of a more timely manner.

[0223] The exemplary mode system, therefore, couples the time that the driver is not looking at the road ahead with a greater alert margin parameter. A linear relationship can be used for example, such as: ινΐΛ / azuz ó / u i uaoo Alert Parameter = base alert parameter value + (factor * (the time elapsed since the driver last saw the road)). QQRH I Π / ΡΖΟΖ / Ε / ΥΙΛ 151

[0224] In the exemplary embodiment, the value of the resulting alert parameter then ends at some maximum value and / or number, which may be selectable by the driver, a fleet manager, or the like. The time since the driver last saw the road may, according to a further embodiment, have a 'grace period' value subtracted before it is used in the above equation. This beneficially allows the driver to briefly glimpse the distance, during which time the vehicle's warning systems do not change their parameterization. It is understood that an equivalent negative value version or an adjustment in a decreasing magnitude sense to the above equation may also be applied, as required by the application that is using the parameter.

[0225] The factor in the above equation can be adjusted within limits so that a desired behavior of the driver is maintained, for example, so that the headway time is kept greater than some minimum value by at least 95%. weather. This adjustment can be made by the driver or from a fleet command center, which can observe behavioral statistics relevant to driver safety. QQRH I Π / ΡΖΟΖ / Ε / ΥΙΛ 152 Verifying the use of the driver's mirror

[0226] Commercial vehicle drivers have many tasks to coordinate during vehicle operation. One of these tasks is to scan the vehicle's mirrors. When vehicle mirror scanning is not done correctly or not carried out frequently enough, the risk of a collision increases.

[0227] It is desirable, therefore, to provide a system, method and apparatus for verifying the sufficiency and suitability of use of the driver's mirror. According to one embodiment, the driver-facing camera 345 is used to verify the driver's proper use of the vehicle's mirrors.

[0228] The embodiments advantageously provide improvements in vehicle operation by helping to increase driving safety, both for commercial and other vehicles, as well as for other vehicles around the vehicle that have the systems, methods and apparatus for monitoring driver behavior of the embodiments hereof, including, in particular, the embodiment that provides verification of the use of the mirror. The modality provides additional characterization of the 153 driver such as, for example, biometric identification information and warns the driver and remote fleet management if any unsafe behavior occurs or is detected. MA.a.zuz ó / u i υυου

[0229] Algorithms for finding faces in images use a model of the human face. This model typically looks for facial 'references', that is, contrasts, distinct areas, such as the corners of the mouth, eyes, etc. When such a configuration of landmarks is found to be within the geometric expectations for human facial appearance, the face is found.

[0230] Landmark settings refer to the direction in which the face points (its 'pose') with respect to the camera. Posture can be summarized by a three-dimensional vector originating from the person's nose, as shown in Figs. 5a and 17 as a 3-D 522 head pose vector.

[0231] It can also be seen that the face has been positioned (chin, mouth, eyes, etc.), placing it within a certain volume in the passenger cabin. The tip of the nose is located in a ray emanating from the chamber, and, on average, 154 approximately centered the seat and pointing MA.a.zuz ó / u i uacD directly forward.

[0232] Fig. 21 is an illustration of the limits that apply to the use of mirrors according to an exemplary embodiment. The modality system refers to the facial posture vector 522, along with the head position, to see which direction the driver is facing (not necessarily the same direction he is looking or observing). Although simple vision by eye movement is only possible in the mirrors, the system examines the facial posture vector 522 over time to determine whether the driver is moving his head to look - as he should - in the mirrors. When the driver is not checking the mirrors often enough 2120 - or perhaps for too long 2110 (after all, one should primarily look ahead when driving forward, for example), an alert is issued, and may be activated a recording of safety events, and statistics on driver behavior can be collected.

[0233] The camera modality system facing the driver geometry is known, along with thus can use 345 (whose position and the location of the driver's head and posture, to increase safety, make 155 follow the policy, look for signs of fatigue, and collect MA. a.ZUZÓ / U IUSOD the driver behavior and safety statistics.

[0234] A particular case of verifying the use of the mirror is that of changing lanes. Good driving practice states that the mirror associated with the lane being changed will be used before making the lane change. Therefore, when the turn signal is set, for example, the example mode system performs a test to use the mirror before changing lanes. The test may be, for example, determining whether the driver looked in the appropriate mirror for a sufficient time (between the upper 2110 and lower 2120 bands) before changing lanes. Equivalently, if the turn signal is not set, but the lane is changed (an event detectable by a lane departure warning system), and the mirror is not seen, then this event of the mirror not being used before When a lane change is detected, it is also activated.

[0235] A similar test for mirror use can be performed when a driver is stopped and blinking right. This is a classic, dangerous situation for any cyclist located on the right side of a vehicle. 156 commercial, where they can be crushed by the truck when turning. Therefore, one can enforce correct mirror use by verifying that the driver has looked to the right before the vehicle moves again, i.e. creating a visual interlock on the vehicle's movement. It is understood that the left-hand side version may also be similarly deployed in regions where left-hand side traffic is the norm.

[0236] According to one embodiment, a system for monitoring an attention condition of the driver of an associated vehicle during operation of the associated vehicle by an associated driver is provided. The system includes an imaging device disposed in the associated vehicle, a control device including a processor, and an output operatively coupled to the processor. The imaging device captures an image of the associated driver disposed in the associated vehicle and the interior of the associated vehicle, and generates image data representative of the captured image of the associated driver disposed in the associated vehicle and the interior of the associated vehicle.

[0237] The control device includes a data input 157 of images operatively coupled with the processor, a non-transitory memory device operatively coupled with the processor, the detection logic of the driver's head stored in the non-transitory memory device, the address logic of the driver's head stored in the non-transitory memory device and the control logic stored in the non-transitory memory device. UUDD

[0238] The image data input receives the image data from the imaging device. The non-transitory memory device stores vehicle geometry data representative of positions relative to one or more associated vehicle structures and safe attention model data comprising a range of recommended values ​​of a driver attention parameter. to the road of the monitored driver's attention condition of the associated vehicle.

[0239] The driver head detection logic is executable by the processor to process the imaging data to locate / determine a candidate head area of ​​the image captured by the imaging device likely above a predetermined threshold stored in the non-transitory memory device to be 158 representative of the head of the associated driver disposed in the associated vehicle, and labeling a portion of the image data corresponding to the candidate head area located / determined by the driver head detection logic as image data of the driver's head. υυου

[0240] The driver head detection logic is executable by the processor to process the driver head image data to determine an orientation direction of the associated driver head and generate the orientation direction data of the driver. driver's head, the orientation direction data of the driver's head being representative of the determined orientation direction of the associated driver's head.

[0241] The control logic is executable by the processor to process the driver's head orientation direction data together with the vehicle geometry data and the position data of the imaging device to determine an operating value of the parameter driver attention to road condition of the associated vehicle's monitored driver attention condition, and perform a comparison between the recommended range of values ​​of the 159 driver attention to road parameter of the monitored driver attention condition of the associated vehicle and the determined operating value of the driver attention to road parameter of the monitored driver attention condition of the associated vehicle. MA.a.zuz ó / u i uaoo

[0242] The control logic is further executable by the processor to determine a state of compliance of the operation of the vehicle according to a result of the comparison between the recommended value range and the determined operating value of the driver's attention parameter to the Monitored driver attention road condition of the associated vehicle.

[0243] The control logic may, according to an example, determine the compliance state of the vehicle operation as one of a driver distraction state according to a first result of the comparison between the recommended value range and the determined operational value of the driver's attention to road parameter of the monitored driver attention condition of the associated vehicle, wherein the processor generates the driver distraction data according to the first result, or a driver attention state According to a second result of the 160 comparison between the recommended value range and the determined operating value of the driver's attention to road parameter of the monitored driver's attention condition of the associated vehicle. MA. a.ZUZÓ / U IUUOO

[0244] The one output selectively receives the driver distraction data from the processor and generates a driver distraction signal representative of the determined operating value of the driver's attention to road parameter of the monitored driver's attention condition while outside the recommended value ranges of safe model data.

[0245] According to another exemplary embodiment, the control logic is executable by the processor to process the driver's head orientation detection data, together with the vehicle geometry data and the position data of the tracking device. images to determine an operational value of the driver's attention parameter on the road of the monitored driver attention condition of the associated vehicle, correlating the parameter of the driver's attention on the road of the associated vehicle, with an operational value of a parameter of the monitored lane departure warning (LDW) condition of the associated vehicle and determine a value of 161 adjustment to modify the setting of an LWD system of the associated vehicle according to the driver attention to road parameter of the monitored driver attention condition of the associated vehicle correlated with the operating value of the parameter of the monitored condition LDW of the vehicle associated. The output is operatively coupled to an input in the LDW system of the associated vehicle and selectively receives the set value to modify the LDW parameter and delivers the set value to the associated vehicle.

Claims

1. A safety system that monitors a steering control condition of an associated vehicle during operation of the associated vehicle by an associated driver, the system comprising: an imaging device disposed in the associated vehicle, the imaging device capturing an image of a portion of the interior of the associated vehicle together with an image of a portion of the driver steering the associated vehicle, and generating image data representative of the captured images of the associated driver and the interior of the associated vehicle; and a control device comprising: a processor; an image data input operatively coupled with the processor, the image data input receiving the image data from the imaging device;a non-transient memory device operatively coupled with the processor, the non-transient memory device storing safe model data comprising a recommended range of values ​​for a steering wheel usage parameter of the monitored steering wheel control condition; and control logic stored in the non-transient memory device, the control logic being executable by the processor to: process image data to determine an operating value for the steering wheel usage parameter of the monitored steering wheel control condition of the associated vehicle; perform a comparison between the recommended range of values ​​for the steering wheel usage parameter of the monitored steering wheel control condition of the associated vehicle and the operating value of the steering wheel usage parameter of the monitored steering wheel control condition of the associated vehicle;and determine a vehicle operation compliance status as one of: a steering policy non-compliance status according to a first result of the comparison between the recommended value range and the operating value of the steering wheel usage parameter of the monitored steering control condition of the associated vehicle, wherein the processor generates steering policy non-compliance data according to the first result, or a steering policy compliance status according to a second result of the comparison between the recommended value range and the operating value of the steering wheel usage parameter of the monitored steering control condition of the associated vehicle;and an output operatively coupled with the processor, selectively receiving the steering wheel policy violation data from the processor and QQRH Ln / Pznz / B / YIL 164 generating a steering wheel usage policy violation signal representative of the operating value of the monitored steering wheel usage parameter of the control condition is outside the recommended range of the safe model data.; 2. The safety system according to claim 1, wherein: the output selectively receives steering wheel compliance data from the processor and generates a steering wheel usage policy compliance signal representative of the operating value of the steering wheel usage parameter of the monitored steering wheel control condition that is within the recommended range of the safe model data.

3. The safety system according to claim 1, wherein: the non-transient memory device stores steering wheel interface policy data representative of a steering wheel hand position rule for the associated driver gripping a steering wheel during operation of the associated vehicle as safe model data comprising the range of recommended values ​​of the steering wheel usage parameter of the monitored steering wheel control state of the associated vehicle; and the control logic stored in the non-transient memory device is executable by the processor to process the image data to: QQRH IP / RP7P7 / E / YIL 165 locate, in the image data, an image of the associated steering wheel of the associated vehicle;Determine, based on the image of the associated steering wheel in the image data, one or more positions of the associated driver's hands on the associated steering wheel of the associated vehicle as the operating value of the steering wheel usage parameter of the monitored steering wheel control of the associated vehicle.

4. The safety system according to claim 3, wherein the control logic is executable by the processor to process the image data to: locate, in the image data, the image of the associated steering wheel of the associated vehicle by detecting an ellipse in the image data as the steering wheel; and determine one or more positions of the associated driver's hands on the associated steering wheel of the associated vehicle as the operating value of the steering wheel usage parameter by: marking pixels of the detected ellipse; and determining one or more discontinuities or breaks in the marked pixels as the one or more positions of the associated driver's hands on the associated steering wheel.

5. The safety system according to claim 3, wherein the control logic is executable by the processor to process the Image data to detect the ellipse in the image data as the steering wheel qqah ιη / ρζηζ / Β / γΐΛ 166 by: QQRH Ln / Pznz / Β / YIL masking a predetermined portion of the image data not directed to the steering wheel portion of the associated vehicle; distorting the unmasked portion of the image data directed to the steering wheel portion of the associated vehicle; and Hough Transform processing the unmasked and undistorted portion of the image data directed to the steering wheel portion of the associated vehicle.

6. The safety system according to claim 3, wherein: the image-forming device captures a series of images of the interior portion of the associated vehicle together with a series of images of the associated driver portion disposed in the associated vehicle, and generates image data representative of the captured series of images of the associated driver and the interior of the associated vehicle; and the control logic stored in the non-transient memory device is executable by the processor to process the image data to: locate, in the image data, images of the associated steering wheel of the associated vehicle during a predetermined period of time;Determining, based on the images of the associated steering wheel in the image data, a ratio between the hands of the associated driver that are on the associated steering wheel of the associated vehicle and the hands of the associated driver that are off the associated steering wheel of the associated vehicle as an operating value of the steering wheel usage parameter of the monitored steering wheel control condition of the associated vehicle.

7. A safety system that monitors the steering wheel usage condition of an associated vehicle during operation of the associated vehicle by an associated driver, the system comprising: an imaging device disposed in the associated vehicle, the imaging device capturing an image of the interior of the associated vehicle and generating image data representative of the captured image; and a control device comprising: a processor; an image data input operatively coupled with the processor, the image data input receiving the image data from the imaging device; a non-transient memory device operatively coupled with the processor,storing in the non-transient memory device secure model data comprising a recommended range of values ​​for a hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle; QQRH Ln / Pznz / B / YIL 168 control logic stored in the non-transient memory device, the control logic being executable by the processor to: process the image data to determine an operating value for the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle by: inspecting the image data to determine a steering wheel shape representative of a steering wheel of the associated vehicle; and inspecting the image data to determine any discontinuity in the determined steering wheel shape, any discontinuity being representative of a,both or neither hands of the associated driver holding the steering wheel of the associated vehicle; perform a comparison between the recommended range of values ​​of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle and the operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle; and determine the vehicle operation compliance status as one of: a steering wheel usage policy non-compliance status according to a first comparison result indicating that the operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition QQRH IP / RP7P7 / E / YIL 169 is outside the recommended range of values ​​for the hand placement parameter on the steering wheel,wherein the control logic is executable by the processor to generate steering wheel policy noncompliance data according to the determined steering wheel policy noncompliance state, or a steering wheel policy compliance state according to a second comparison result indicating that the operating value of the hand placement parameter on the steering wheel of the monitored steering wheel use condition is within the recommended range of values ​​for the hand placement parameter on the steering wheel, wherein the control logic is executable by the processor to generate steering wheel policy compliance data according to the determined steering wheel policy compliance state; and an output operatively coupled with the processor,The output received by the steering wheel uses policy compliance or non-compliance data from the processor and selectively generates: a steering wheel usage policy non-compliance signal representative of the steering wheel usage policy non-compliance data received from the processor, or a steering wheel usage policy compliance signal representative of the steering wheel usage policy compliance data received from the processor.

8. The safety system according to QQAD IP / GZRPZ / B / YIL 170 claim 7, wherein the control logic of the control device is executable by the processor to determine the operating value of the hand placement parameter on the steering wheel by: applying a Hough Transform to the image data to detect an ellipse shape in the image data; associating the detected ellipse shape with a steering wheel shape representative of the steering wheel of the associated vehicle; and inspecting the image data to determine any discontinuity in the determined ellipse shape, any discontinuity being representative of one, both, or neither hand of the associated driver holding the steering wheel of the associated vehicle.

9. The safety system according to claim 8, wherein: the control device processor executes the control logic to determine the steering wheel template data by: applying the Hough Transform to the image data; detecting the ellipse shape in the image data; and selecting pixels in the image data corresponding to the selected ellipse as steering wheel template data; the non-transient memory device stores the steering wheel template data; the image formation device functions to capture an additional image of the interior of the associated vehicle and generate additional image data representative of the additional captured image;The control logic of the control device is executable by the processor to process the additional image data to determine a current operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle by: comparing the additional image data with the ellipse shape detected in the steering wheel template data stored in the non-transient memory device to determine in the additional image data a current shape of the steering wheel representative of a current steering wheel position of the associated vehicle; and inspecting the additional image data to determine any discontinuity in the current determined shape of the steering wheel, any discontinuity being representative of one, both or neither hand of the associated driver holding the steering wheel of the associated vehicle.

10. The safety system according to claim 9, wherein: the imaging device operates for a first period of time to capture a plurality of additional images of the interior of the associated vehicle and generate a plurality of additional image data sets, each being representative of the plurality of additional images captured;The control logic of the control device is executable by the processor to process the plurality of additional image data sets to determine a current average operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle by: comparing each additional image data set of the additional image data sets with the ellipse shape detected in the steering wheel template data stored in the non-transient memory device to determine in each additional image data set a current steering wheel shape representative of a current steering wheel position of the associated vehicle;inspect each additional image data set to determine any discontinuity in the current determined shape of the steering wheel, any discontinuity being representative of one, both, or neither hand of the associated driver holding the steering wheel of the associated vehicle; and average any discontinuity over the first time period as the operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition. QQRH IP / RP7P7 / E / YIL 173; 11. The security system according to claim 8, wherein: the non-transient memory device stores a plurality of steering wheel template data sets, each steering wheel template data set being representative of an image obtained by the imaging device of the steering wheel of the associated vehicle arranged in a particular different position with respect to the interior of the associated vehicle; the imaging device operates for a first period of time to capture a plurality of additional images of the interior of the associated vehicle and generate a plurality of additional image data sets, each being representative of the plurality of additional images captured;The control logic of the control device is executable by the processor to process the plurality of additional image data sets to determine a current average operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle by: comparing each additional image data set of the additional image data sets with the ellipse shapes detected in the plurality of steering wheel template data sets stored in the non-Lransl Loria memory device to deLerininar in each additional image data set a current steering wheel shape representative of a current steering wheel position of the associated vehicle;inspect each additional image data set to determine any discontinuity in the current determined shape of the steering wheel, any discontinuity being representative of one, both, or neither hand of the associated driver holding the steering wheel of the associated vehicle; and average any discontinuity over the first time period as the operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition.

12. The safety system according to claim 7, wherein: the image-forming device operates for a first period of time to capture a plurality of additional images of the interior of the associated vehicle and generate a plurality of additional image data sets, each being representative of the plurality of additional images captured; the control logic of the control device is executable by the processor to process the plurality of additional image data sets to determine a current average operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle; determining in each additional image data set a current shape of the steering wheel representative of a current steering wheel position of the associated vehicle;inspect each additional image data set to determine any discontinuity in the current determined shape of the steering wheel, any discontinuity being representative of the movements of the associated vehicle driver's hand intermittently holding the steering wheel; and determine from any discontinuity during the first time period an attention of the associated vehicle driver as an additional operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition.

13. The safety system according to claim 12, wherein: the control logic of the control device is executable by the processor to process the determined attention of the driver of the associated vehicle as driver attention data; and the non-transient memory device stores the driver attention data.

14. A method for monitoring a steering wheel usage condition of a partner vehicle during operation of the partner vehicle by a partner driver, the method comprising: capturing, by means of an imaging device disposed in the partner vehicle, an image of the interior of the partner vehicle and generating image data representative of the captured image; receiving the image data at an image data input of a control device comprising a processor and a non-transient memory device to store control logic and safe model data comprising a recommended range of values ​​for a steering wheel hand placement parameter of the monitored steering wheel usage condition of the partner vehicle; executing the control logic by the processor to: process the image data to determine an operating value for the steering wheel hand placement parameter of the monitored steering wheel usage condition.of the associated vehicle by: inspecting image data to determine a steering wheel shape representative of a steering wheel of the associated vehicle; and inspecting image data to determine any discontinuity in the determined steering wheel shape, any discontinuity being representative of one, both, or neither hand of the associated driver holding the steering wheel of the associated vehicle; performing a comparison between the recommended range of values ​​of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle and the operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle; and determining the vehicle operation compliance status as one of: 5 a steering wheel usage policy non-compliance status according to a first result of the comparison indicating that the operating value of the parameterHand placement on steering wheel of the monitored steering wheel usage condition is outside the recommended range of values ​​of the 10 hand placement on steering wheel parameter, wherein the control logic is executable by the processor to generate steering wheel usage policy non-compliance data according to the determined steering wheel usage policy non-compliance state, or 15 a steering wheel usage policy compliance state according to a second comparison result indicating that the operating value of the hand placement on steering wheel parameter of the monitored steering wheel usage condition is within the recommended range of values ​​of the 20 hand placement on steering wheel parameter, wherein the control logic is executable by the processor to generate steering wheel usage policy compliance data according to the determined steering wheel usage policy compliance state; and 25 receive compliance or non-compliance data ofSteering wheel usage policy from the processor via an output of the coupled control device QQRn I η / R7P7 B / YIL 178 operatively with the processor, and selectively generate: a representative steering wheel usage policy non-compliance signal from the steering wheel using the policy non-compliance data received from the processor, or a steering wheel usage policy compliance signal representative of the steering wheel usage policy compliance data received from the processor.

15. The method according to claim 14, wherein executing the control logic to determine the operating value of the hand placement parameter on the steering wheel comprises executing the control logic to determine the operating value of the hand placement parameter on the steering wheel by: applying a Hough Transform to the image data to detect an ellipse shape in the image data; associating the detected ellipse shape with a steering wheel shape representative of the steering wheel of the associated vehicle; and inspecting the image data to determine any discontinuity in the determined ellipse shape, any discontinuity being representative of one, both, or neither of the associated driver's hands holding the steering wheel of the associated vehicle.

16. The method according to claim 15, wherein: the execution of the control logic by the processor comprises executing the control logic by the processor to determine the steering wheel template data by: applying the Hough Transform to the image data; detecting the ellipse shape in the image data; and selecting pixels in the image data corresponding to the detected ellipse as steering wheel template data; storing the control logic in the non-transient memory device comprises storing the steering wheel template data in the non-transient memory device; the image capture by the image-forming device comprises operating the image-forming device to capture an additional image of the interior of the associated vehicle and generating additional image data representative of the additional image captured;Executing the control logic of the control device by the processor comprises executing the control logic to process the additional image data to determine a current operating value of parameter 25 hand placement on steering wheel of the associated vehicle steering wheel usage condition; comparing the additional image data with the QQRH Ln / Pznz / B / YIL 180 ellipse shape detected in the steering wheel template data stored in the non-transient memory device to determine in the additional image data a current steering wheel shape representative of a current steering wheel position of the associated vehicle; and inspecting the additional image data to determine any discontinuity in the determined current steering wheel shape, any discontinuity being representative of one, both, or neither hand of the associated driver holding the steering wheel of the associated vehicle.

17. The method according to claim 16, wherein: operating the imaging device comprises operating the imaging device 15 for a first period of time to capture a plurality of additional images of the interior of the associated vehicle and generate a plurality of additional image data sets, each being representative of the plurality of additional images captured;20 The execution of the control logic of the control device by the processor comprises executing the control logic to process the plurality of additional image data sets to determine a current average operating value of the parameter 25 hand placement on steering wheel of the monitored steering wheel usage condition of the associated vehicle by: comparing each additional image data set QQAH I η / R7P7 B / YIL 181 of the additional image data sets with the ellipse shape detected in the steering wheel template data stored in the non-transient memory device to determine in each additional image data set an actual steering wheel shape representative of a current steering wheel position of the associated vehicle;inspect each additional image data set to determine any discontinuity in the current determined shape of the steering wheel, any discontinuity being representative of one, both, or neither hand of the associated driver holding the steering wheel of the associated vehicle; and average any discontinuity over the first time period as the operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition.

18. The method according to claim 15, further comprising: storing a plurality of steering wheel template data sets in the non-transient memory device, each steering wheel template data set being representative of an image obtained by the imaging device of the steering wheel of the associated vehicle arranged in a particular different position relative to the interior of the associated vehicle; operating the imaging device for a first period of time to capture a plurality of additional images of the interior of the associated vehicle and generating a plurality of additional image data sets, each being representative of the plurality of additional images captured;execute the control logic of the control device by the processor to process the plurality of additional image data sets to detect and remove a current average operating value of the hand placement parameter on the steering wheel from the monitored steering wheel usage condition of the associated vehicle by: comparing each additional image data set from the additional image data sets with the ellipse shapes detected in the plurality of steering wheel template data sets stored in the non-transient memory device to determine in each additional image data set a current steering wheel shape representative of a current steering wheel position of the associated vehicle;inspect each additional image data set to determine any discontinuity in the current determined shape of the steering wheel, any discontinuity being representative of one, both, or neither hand of the associated driver holding the steering wheel of the associated vehicle; and average any discontinuity over the first Llempo period as the operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition QQRH IP / RP7P7 / E / YIL 183.

19. The method according to claim 14, further comprising: operating the imaging device for a first period of time to capture a plurality of additional images of the interior of the associated vehicle and generating a plurality of additional image data sets, each being representative of the plurality of additional images captured; executing the control logic of the control device by the processor to process the plurality of additional image data sets to determine a current average operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition of the associated vehicle by: determining in each additional image data set a current shape of the steering wheel representative of a current steering wheel position of the associated vehicle;inspect each additional image data set to determine any discontinuity in the current determined shape of the steering wheel, any discontinuity being representative of the movements of the associated vehicle driver's hand intermittently gripping the steering wheel; and determine from any discontinuity during the first time period an additional operating value of the hand placement parameter on the steering wheel of the monitored steering wheel usage condition.

20. The method according to claim 19, further comprising: executing the control logic of the control device by means of the processor to process the determined attention of the driver of the associated vehicle as driver attention data; and storing the driver attention data in the non-transient memory device.

21. A system that monitors a permitted occupant condition of a fleet policy during operation of an associated vehicle by an associated driver, the system comprising: an imaging device disposed in the associated vehicle, the imaging device capturing an image of the interior of the associated vehicle including an image of the driver and any passengers disposed in the associated vehicle, and generating image data representative of the captured image; and a control device comprising: a processor; an image data input operatively coupled with the processor, the image data input receiving the image data from the imaging device; a non-transient memory device operatively coupled with the processor;face detection logic stored in the non-transient memory device, the face detection logic being executable by the processor to: process the image data to locate one or more candidate face areas from the image captured by the imaging device likely above a predetermined threshold stored in the non-transient memory device will be representative of one or more corresponding faces on the associated vehicle;and generate a set of face descriptors for each of the one or more candidate face areas, control logic stored in the non-transient memory device, the control logic being executable by the processor to: determine, based on the set of face descriptors generated for each of the one or more candidate face areas, a probable face count as an operational value of an occupant count parameter of the monitored allowed occupant condition of fleet policy during the operation of the associated vehicle.; 22. The system according to claim 21, wherein: the non-transient memory device stores: fleet policy compliance model data comprising a recommended range of values ​​for the occupant count parameter of the monitored permitted occupant condition of the associated vehicle; and the control logic stored in the non-transient memory device is further executable by the processor to: process the set of face descriptors generated for each of one or more candidate face areas to determine a vehicle occupant count as the operating value of the occupant count parameter of the monitored permitted occupant condition of the fleet policy during the operation of the associated vehicle;Perform a comparison between the recommended range of values ​​of the 10 occupant number parameter of the monitored permitted occupant condition and the operating value of the occupant number parameter of the monitored permitted occupant condition;and determine a vehicle operation fleet policy compliance status 15 as one of: a non-compliance status according to a first result of the comparison between the recommended range of values ​​and the operating value of the occupant quantity parameter of the monitored permitted occupant condition of the fleet policy during the operation of the 20 associated vehicle, wherein the processor generates non-compliance data according to the first result, or a compliance status according to a second result of the comparison between the recommended range of values ​​and the operating value of the occupant quantity parameter of the 25 monitored permitted occupant condition of the fleet policy during the operation of the associated vehicle.; 23. The system according to claim QQRn I η / R7P7 B / YIL 187 22, further comprising: an output operatively coupled with the processor, the output selectively receiving the non-compliance data from the processor and generating a non-compliance signal representative of the operating value of the 5 occupant quantity parameter of the monitored permitted occupant condition is outside the recommended range of the safe model data.

24. The system according to claim 22, wherein: the facial detection logic is operable to 10 process the set of face descriptors generated for each of one or more candidate face areas to determine one or more identities of one or more human persons inside the associated vehicle based on a comparison of the set of face descriptors with 15 information from the driver database stored in the non-transient memory device.

25. The system according to claim 22, wherein: the facial detection logic is operable to process the set of face descriptors generated for each of the one or more candidate face areas to determine one or more human faces within the associated area, vehicle; and the control logic is operable to selectively transmit, based on one or more determined faces not matching the information in the driver database stored in the non-transient memory device. QQRH Ln / Pznz / B / YIL 26. The system according to claim 188 22, further comprising: voice detection logic operable for identifying a human person associated with the set of face descriptors generated for each of one or more candidate face areas based on a comparison of received voice data representative of a recorded voice of the human person with information from the driver database stored in the non-transient memory device.

27. The security system according to claim 26, further comprising: mouth movement logic operable to identify a human person associated with the set of face descriptors for each of one or more candidate face areas according to voice data in combination with mouth movement data received representative of recorded mouth movement images of one or more human passengers corresponding to one or more candidate face areas.

28. The system according to claim 22, wherein: the control logic is further executable by the processor to determine the vehicle fleet policy compliance status over time, and to store the determined vehicle fleet policy compliance status over time in a compliance log data file on the non-transient memory device. QQAD Ln / Pznz / B / YIL 29. The system according to claim 189 22, wherein: the control logic is further executable by the processor to transmit the determined state of compliance with the vehicle's fleet operating policy through a system output device to an associated fleet management system 5 to determine in the associated fleet management system a fidelity of compliance with the fleet policy.

30. The system according to claim 22, wherein: the face detection logic is executable by the processor to: process the image data received from the image-forming device without distorting the image data to remove wide-angle lens effects from the image data; and process the undistorted image data to locate one or more candidate face areas, and generate the set of face descriptors from one or more candidate face areas.

31. The system according to claim 22, wherein: the control logic is executable by the processor to: process the set of face descriptors by comparing the set of face descriptors with historical face descriptors stored in the non-transient memory device to determine the vehicle operation fleet policy compliance status based on the set of face descriptors that match the historical face descriptors comprising the authorized occupants of the associated vehicle. QQRH I η / R7P7 B / YIL 190 32. The system according to claim 31, wherein: the historical face descriptors are stored in one or more non-transient memory devices and / or in an associated database of an associated fleet management system in operational communication with the system.

33. The system according to claim 22, further comprising: processor-executable tracking logic for determining locations of the associated vehicle during operation of the associated vehicle, and for associating the set of face descriptors with the determined locations of the associated vehicle during operation of the associated vehicle.

34. The system according to claim 22, wherein: the non-transient memory device stores template data representative of images of a passenger door of the associated vehicle that is in one or more of a closed condition, an open condition and / or an uncertain condition; and the control logic is executable by the processor to determine a state of the passenger door of the associated vehicle by comparing the image data received from the image-forming device of the interior of the associated vehicle, including the image of the driver and any passengers disposed in the associated vehicle, with the template data.

35. The system according to claim 34, wherein: the control logic is executable by the processor to selectively determine the probable face count based on the determined state of the associated vehicle passenger door.

36. The system according to claim 22, wherein: the control logic is executable by the processor to determine a triggering event based on one or more conditions of one or more physical components of the associated vehicle; and the control logic is executable by the processor to selectively process the image data based on an occurrence of the triggering event to locate one or more candidate face areas from the image data received from the imaging device.

37. The system according to claim 36, wherein: the control logic is executable by the processor to determine the triggering event based on an occurrence of one or more conditions of one or more physical components of the associated vehicle comprising one or more of: a door of the associated vehicle opening; the associated vehicle stopping; the door of the associated vehicle closing and the vehicle recently moving forward; the vehicle just moving forward; and / or the vehicle stopping in an unusual location.

38. The system according to claim 36, wherein: the control logic is executable by the processor to selectively process the set of face descriptors based on the occurrence of the triggering event by comparing the set of face descriptors qqrh Ln / rznz / B / γAL 192 with historical face descriptors stored in the transient memory device to determine the vehicle operation fleet policy compliance status based on the set of face descriptors 5 that match the historical face descriptors comprising the authorized occupants of the associated vehicle.

39. The system according to claim 32, wherein: the control logic is executable by the processor to determine an activation event based on one or more conditions of one or more physical components of the associated vehicle; the face detection logic is operable based on an occurrence of the triggering event to process the set of face descriptors generated for each of the one or more candidate face areas to determine one or more human faces inside the associated vehicle; the control logic is executable by the processor based on the occurrence of the triggering event to: determine the conditions of the associated vehicle comprising a vehicle speed condition and a door status condition; store in the non-transient memory device the audio obtained from inside the associated vehicle;to store the determined operating state of the vehicle, fleet policy compliance, the determined conditions of the 25 associated vehicle, the audio, the image of the driver and passengers available in the associated vehicle obtained by the image formation device, in a QQRH I η / R7P7 B / YIL 193 compliance record data file on the non-transient memory device.; 40. The system according to claim 39, wherein: the control logic is further executable by the processor to transmit the determined compliance record data file through a system output device to an associated fleet management system to determine in the associated fleet management system a fleet policy compliance fidelity.

Images