Mobile Device Usage Monitoring for Commercial Vehicle Fleet Management

Abstract

The driver monitoring system includes at least two cameras configured to record images of a driver within a cab of a vehicle. A first camera is located on a driver's side of the cab and a second camera is located on a passenger's side of the cab. At least one of the at least two cameras is located at a mirror replacement display or a mirror replacement display location. A controller is in communication with the at least two cameras and configured to determine a driver's posture based on images from the at least two cameras. At least one of the images originates from the first camera and at least one of the second camera.

Description

[Technical Field]

[0001] This application relates to driver monitoring, and in particular to recording images depicting a driver during an abnormal driving event.

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This disclosure is a continuation-in-part of U.S. Patent Application No. 16 / 845,228, filed April 10, 2020, which claims priority to U.S. Patent Application No. 17 / 145,891, filed January 11, 2021, which claims priority to U.S. Provisional Application No. 62 / 833,252, filed April 12, 2019, which is incorporated herein by reference in its entirety. [Background technology]

[0003] Commercial vehicle safety costs are rising dramatically, with distracted driving becoming a leading cause of accidents. Cell phone use is believed to be the leading cause of these types of distracted driving accidents. Summary of the Invention

[0004] In one exemplary embodiment, a driver monitoring system includes at least two cameras configured to record images of a driver within a cab of a vehicle, wherein a first camera is located on a driver's side of the cab and a second camera is located on a passenger's side of the cab, and at least one of the at least two cameras is located at a mirror replacement display or a mirror replacement display position; and a controller in communication with the at least two cameras, the controller configured to determine a posture of the driver based on images from the at least two cameras, at least one of the images originating from the first camera and at least one of the second camera.

[0005] In another example of the above driver monitoring system, the at least two cameras include a third camera facing the rear of the cab and defining a field of view that includes the driver's face when the driver is looking forward.

[0006] In another example of any of the above driver monitoring systems, the first camera is integrated into the driver's side mirror replacement monitor and defines a field of view that includes at least the driver's face when the driver is looking out the driver's side window, as well as at least one arm of the driver, one hand of the driver, and one shoulder of the driver.

[0007] In another example of any of the above driver monitoring systems, the second camera is incorporated into a passenger side mirror replacement monitor and defines a field of view that includes the driver's face when the driver is looking out the passenger side window, as well as at least the other of the driver's arms, the driver's hands, and the driver's shoulders.

[0008] In another example of any of the above driver monitoring systems, the controller is operable to obtain the at least one particular image from a rolling video buffer recorded within a time frame corresponding to the abnormal event, the rolling video buffer including video feeds originating from the at least two cameras.

[0009] In another example of any of the above driver monitoring systems, the controller is further configured to determine a line of sight of the driver based at least in part on an analysis of at least two simultaneous images from the at least two cameras.

[0010] In another example of any of the above driver monitoring systems, the controller is configured to supplement the attentive / inattentive posture detection with the gaze detection.

[0011] In another example of any of the above driver monitoring systems, the controller is further configured to analyze the driver's posture determined from the posture detector, thereby determining whether the posture is an attentive posture or an inattentive posture.

[0012] In another example of any of the above driver monitoring systems, the analysis to determine whether the posture is an attentive posture or an inattentive posture uses a machine learning algorithm.

[0013] In another example of any of the above driver monitoring systems, the at least two cameras include a third camera positioned generally in front of the driver.

[0014] In one exemplary embodiment, a driver monitoring system includes an eye-tracking system including a plurality of cameras configured to record images of a driver within a vehicle cab and determine a gaze direction of the driver using the recorded images, and a controller in communication with the eye-tracking cameras configured to detect a potential distracted driving event based on the driver's gaze direction deviating from a predefined warning driver region for an amount of time that exceeds a predefined time threshold.

[0015] Another example of the above driver monitoring system further includes a posture tracking system configured to determine a posture of the driver based in part on body part positions, the body part positions relative to at least one of vehicle components and other body part positions.

[0016] In another example of any of the above driver monitoring systems, the body part includes at least one of the driver's hand, arm, torso, and face.

[0017] In another example of any of the above driver monitoring systems, the plurality of cameras includes at least two cameras, and at least one of the at least two cameras is positioned within a mirror replacement monitor.

[0018] In another example of any of the above driver monitoring systems, the controller is configured to provide at least one image to a convolutional neural network, the convolutional neural network being trained to identify an inattentive driving posture using a first data set including images from a position of a first camera of the at least two cameras and a second data set including images from a position of a second camera of the at least two cameras.

[0019] In another example of any of the above driver monitoring systems, the neural network is configured to analyze recorded images and identify occurrences of the driver's use of a mobile device in the recorded images, and the neural network is configured to cause the driver monitoring system to at least one of transmit the recorded images to a fleet manager and store the recorded images in a local repository of abnormal driving images.

[0020] In another example of any of the above driver monitoring systems, the controller is configured to acquire additional images depicting the driver from the eye-tracking camera or another camera at random intervals, and to transmit the additional images to a fleet manager, store the additional images in the local repository of abnormal driving images, or both.

[0021] An exemplary method of monitoring a driver includes recording images of a driver in a cab of a vehicle using a driver monitoring system including at least two cameras, including a first camera and a second camera, wherein each of the first camera and the second camera is incorporated into a mirror replacement monitor; detecting an abnormal driving event of the vehicle based on input from at least one vehicle sensor; acquiring at least two specific images from the driver monitoring system depicting the driver during the abnormal event, wherein each of the at least two specific images is from a respective one of the at least two cameras; and performing at least one of transmitting the specific images to a fleet manager and storing the specific images in a local repository of abnormal driving images.

[0022] Another example of the above exemplary method of monitoring a driver further includes determining the driver's posture in the recorded images based at least in part on which of at least the first camera and the second camera includes the driver's face; detecting a potential distracted driving event based on the driver's posture depicted in a particular one of the recorded images being consistent with an inattentive posture for an amount of time that exceeds a predefined time threshold; and, in response to detecting the distracted driving event, performing at least one of sending the particular image to a fleet manager and storing the particular image in a local repository of abnormal driving images. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a schematic diagram of an example fleet management system.

[0024] [Figure 2] FIG. 2 is a schematic diagram of components of an exemplary driver monitoring system for each vehicle of FIG. 1.

[0025] [Figure 3]FIG. 3 is a more detailed schematic diagram of the electronic control unit of FIG. 2.

[0026] [Figure 4] 1 is a flowchart of an exemplary method for monitoring a driver.

[0027] [Figure 5] 10 is a flowchart of an example of another method for monitoring a driver.

[0028] [Figure 6] 1 is a schematic diagram of an example of a vehicle cab.

[0029] [Figure 7] 10 is a flowchart of an example of another method for monitoring a driver.

[0030] [Figure 8] FIG. 1 is a schematic diagram of an exemplary vehicle including a mirror replacement system adapted to facilitate driver monitoring. DETAILED DESCRIPTION OF THE INVENTION

[0031] The embodiments, examples, and alternatives set forth in the claims and in the following description and drawings include any of their various aspects or individual features, which may be employed independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible.

[0032] 1 illustrates schematically an example fleet management system 10 including a fleet 12 of vehicles 14A-N operable to communicate with a fleet manager 22 over a wide area network ("WAN") 16, such as the Internet. The vehicles 14 are operable to record images depicting the drivers of the vehicles 14 and to store or transmit those images, optionally along with associated event data describing how the vehicles 14 are operating (e.g., acceleration events, steering events, braking events, close-in collisions, etc.).

[0033] In one example, the vehicles 14A-N transmit images and / or event data to the fleet manager 22 by transmitting images to the fleet management server 18, which is accessible by the computing device 20 of the fleet manager 22 that oversees the fleet 12. In one example, the vehicles 14A-N can transmit images and / or event data to the fleet manager 22 by transmitting to the computing device 20 of the fleet manager 22, bypassing the fleet management server 18. In one example, in addition to or instead of transmitting images to the fleet manager 22, the vehicles 14 store images in a local repository within the vehicle 14. In one example, whether a given image is transmitted over the WAN 16 or stored in the local repository is based on whether the vehicle 14 is currently able to connect to the WAN 16. In the example of FIG. 1 , the vehicles 14 are trucks, although it is understood that other commercial vehicles, such as delivery vans, may be used.

[0034] FIG. 2 is a schematic diagram of components of an exemplary driver monitoring system 24 provided in each vehicle 14. In the example of FIG. 2, the driver monitoring system 24 includes an electronic control unit (ECU) 30 operably connected to a telematics module 32, a cab camera 34, an acceleration sensor 36, a steering angle sensor 38, and a braking sensor 40. While three sensors 34-38 are described, it is understood that fewer or more sensors may be used. For example, the ECU 30 may be operably connected to an exterior camera 42 operable to record images of the vehicle 14's surrounding environment, an object detection sensor 44 operable to detect objects external to the vehicle 14, a wireless activity detector 45 operable to detect use of a wireless device by the driver, an electronic display 46, a vehicle speaker 48, and / or a Bluetooth® module 50.

[0035] In one example, the electronic display 46 and speaker 48 are part of a driver information system (“DIS”) that provides information about vehicle conditions (e.g., speed, engine RPM, etc.). In this example, the electronic display 46 may be part of a vehicle instrument cluster. As another example, the electronic display 46 may be a center console display that is part of an infotainment system that provides a combination of vehicle and entertainment information (e.g., current radio station, climate control, etc.). In one example, the ECU 30 is integrated into a DIS ECU (not shown) or telematics module 32. In some examples, a posture detector 31 is included within the ECU 30 of the driver monitoring system 24. The posture detector 31 analyzes images from multiple cameras, including the cab camera 24, to determine the driver's overall posture and triggers one or more responses based on the determined posture. Images from multiple cameras may be analyzed individually and used to validate each other, or the images from multiple cameras may be combined into a single image to provide a more complete set of information for the ECU 30 to analyze in determining the driver's posture.

[0036] In the example of Figure 2, ECU 30 is operatively connected to components 31-50 via a vehicle data bus 52, which may be a controller area network ("CAN") bus. Of course, it will be understood that Figure 2 is an example, and that ECU 30 may be connected to certain ones of components 32-50 via other connections other than vehicle data bus 52.

[0037] FIG. 3 is a more detailed schematic diagram of the ECU 30. Referring now to FIG. 3, the ECU 30 includes a processor 60 operatively connected to a memory 62 and a communication interface 64. The processor 60 includes one or more processing circuits, such as a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), or the like. The memory 62 may include one or more types of memory, such as a read-only memory (ROM), a random-access memory, a cache memory, a flash memory device, an optical storage device, or the like. The memory 62 includes a local repository 66 of abnormal driving images and may optionally also include a convolutional neural network (“CNN”) 67, a driver attention model 68, and / or a driver gaze model 69. In some examples, a posture detector 31 is also included within the memory 62. The CNN 67, in some examples, is operable to detect whether the driver is utilizing a mobile device in the cab of the vehicle 14. As used herein, “mobile device” refers to a handheld electronic device, such as a mobile phone, a smartphone, a tablet, a personal media player, or the like. While illustrated as part of the ECU 30, it is understood that the CNN 67 may instead be stored external to the vehicle 14, such as on the fleet management server 18. The communication interface 64 provides communication between the ECU 30 and other components (e.g., a wired connection to the vehicle data bus 52). In another example, the CNN 67 is operable to detect the driver's overall posture and trigger one or more responses within the ECU 30 based on whether the posture corresponds to attentive or inattentive driving. As used herein, attentive driving refers to posture, gaze, or other operator state that corresponds to the operator recognizing an existing condition and responding appropriately thereto. Inattentive driving refers to posture, gaze, or other operator state that does not correspond to the operator recognizing an existing condition and responding appropriately thereto. By way of example, a driver may be considered inattentive if they are not eating, using a cell phone, or otherwise engaged in a driving-related task.

[0038] Referring now to FIG. 2 and with continued reference to FIG. 3, the cab camera 34, which may be an eye-tracking camera, is configured to record images of the driver within the cab of the vehicle 14, and each of the sensors 36-44 is configured to detect abnormal vehicle driving events based on predefined criteria corresponding to distracted driving. In an alternative example, the video feed from the cab camera 34 may be provided to the posture detector 31. The posture detector 31 is configured to utilize a CNN 67 to track the driver's posture, including arm position and orientation, hand position and orientation, torso twist, relative orientation and / or position of the hands and face, arm and hand position relative to the steering wheel, and any number of similar posture metrics that combine to define the posture the driver is currently assuming. The determined posture is then correlated with a number of trained postures and identified as either attentive or inattentive. In yet another example, the CNN 67 can combine posture detection with gaze detection from the cab camera 34 to further improve its ability to distinguish between attentive and inattentive postures.

[0039] In some examples, detection of an abnormal driving event by one of the sensors 36-44 causes the ECU 30 to acquire a specific image taken by the cab camera 34 or a posture identified by the posture detector 31 depicting the driver during the abnormal event. The ECU 30 transmits the specific image or identified posture to the fleet manager 22 using the telematics module 32 and / or stores the specific image or identified posture in the local repository 66. An abnormal driving event may include sudden acceleration, sudden deceleration, a sudden change in steering trajectory, an unexpectedly long period of constant steering stability, a deviation from the expected road position determined by satellite navigation, or other similar events. Distracted driving may occur for a variety of reasons, including, but not limited to, cell phone use, eating, drowsiness, or operating an entertainment system.

[0040] The acceleration sensor 36 is configured to detect abnormal acceleration events that may be indicative of distracted driving, such as sudden acceleration or deceleration of the vehicle 14. Predefined criteria for the acceleration sensor 36 may include, for example, an acceleration rate that exceeds a predefined acceleration threshold or a deceleration rate that falls below a predefined deceleration threshold.

[0041] The steering angle sensor 38 is configured to detect abnormal steering events, such as a rapid change in steering wheel angle, which may indicate a turn. For example, predefined criteria for the steering angle sensor 38 may include a change in steering angle exceeding a predefined angle threshold within a predefined time period while the vehicle 14 is traveling at a speed exceeding a predefined speed threshold, which may indicate a turn as a result of distracted driving.

[0042] Brake sensor 40 is configured to detect abnormal braking events, such as rapid braking of vehicle 14, and may be configured, for example, to measure changes in vehicle speed and / or control signals sent to the vehicle braking system.

[0043] The object detection sensor 44 may be, for example, a LIDAR ("light detection and ranging") or RADAR ("radio detection and ranging") sensor. The object detection sensor 44 may be used alone or in combination with the ECU 30 to detect near misses where a collision was narrowly avoided.

[0044] Telematics module 32 includes a wireless transceiver operable to transmit images over WAN 16. In one example, telematics module 32 is configured to use predefined protocol standards, such as one or more 802.11 standards and / or one or more cellular standards (e.g., GSM, CDMA, LTE, etc.).

[0045] The wireless activity detector 45 includes an antenna configured to detect wireless signals and associated processing circuitry for determining whether the detected wireless signals represent mobile device use within the cab of the vehicle 14 based on one or more predefined thresholds. Criteria used by the processing circuitry of the wireless activity detector 45 may include any one or combination of signal strength, signal duration, and a mobile device identifier. Some examples of a mobile device identifier may include an International Mobile Subscriber Identity ("IMSI"), an Internet Protocol ("IP") address, or a Media Access Control ("MAC") address, and if a mobile device identifier associated with a driver is detected, there is an increased likelihood that the signal transmission corresponds to mobile device use by this driver rather than a pedestrian or the driver of a nearby vehicle.

[0046] In one example, signal duration is used to distinguish between background activity, such as handovers between adjacent cells, where the driver is not actually using the mobile device, and active use of the mobile device (e.g., phone calls, video streaming, etc.), where the signal duration is likely to exceed a predefined signal length threshold.

[0047] In one example, wireless activity detector 45 is configured to limit its monitoring to frequency bands associated with known telecommunications standards, such as GSM band(s), CDMA band(s), LTE band(s), WiMax band(s), WiFi band(s), etc. In one example, wireless activity detector 45 includes multiple antennas, each tuned for a particular frequency band or set of frequency bands, and / or includes one or more antennas configured to sweep multiple such frequency bands.

[0048] In one example, wireless activity detector 45 is configured to detect based at least in part on signal strength, as signals detected from mobile devices in the vehicle cab are likely to be stronger than signals from mobile devices in adjacent vehicles.

[0049] In one example, the cab camera 34 is a video camera operable to provide a rolling buffer of a predefined period (e.g., 30 seconds) that overwrites itself if not backed up, and the ECU 30 is operable to capture images from frames of the rolling video buffer within a time frame corresponding to an abnormal driving event. This may also store video leading up to an abnormal driving event to provide an opportunity to view what happened during and before the abnormal driving event.

[0050] In one example, ECU 30 is configured to record additional images depicting the driver from cab camera 34 at random intervals that may occur outside of an abnormal driving event, transmit the additional images to fleet manager 22, store the additional images in local repository 66 of abnormal driving images, or both. This random sampling can provide an additional level of compliance with the driver.

[0051] In one example, ECU 30 is configured to adjust predefined criteria used to determine an abnormal driving event based on at least one of traffic density, weather conditions, and object detection in the vicinity of vehicle 14. For example, in adverse weather conditions (e.g., rain, snow, icy roads) and in areas with heavy traffic or high pedestrian traffic, the thresholds used to determine what constitutes abnormal driving may be lowered from default values ​​to more stringent criteria, particularly if vehicle 14 is a large commercial truck.

[0052] The determination of whether adverse weather conditions exist may be based, for example, on a weather forecast received by ECU 30. The determination of whether vehicle 14 is in a high traffic or pedestrian area may be based, for example, on traffic reports received by ECU 30 and / or on object detection from object detection sensor 44 or external camera 42.

[0053] The predefined criteria used to detect abnormal driving events may also be selected based on the driver's experience level, which may provide stricter criteria for less experienced drivers and more lenient criteria for more experienced drivers.

[0054] In one example, the ECU 30 is configured to continuously identify the driver's posture using posture detector 31 and trigger an inattentive driver response when a posture corresponding to inattentive driving is detected. As an example, the response may include an audio, visual, tactile, or other sensory alert provided to the driver. In another example, the inattentive driver response may be a command to store an image of the driver in an inattentive posture generated from an in-vehicle camera for supervisor review and / or further analysis.

[0055] In one example, the physical implementation described herein facilitates tracking and monitoring of a vehicle operator's posture by the ECU 30. When the ECU 30 detects an abnormal driving event, the ECU 30 reviews images from multiple video feeds to determine the driver's posture. Alternatively, posture is continuously monitored by the posture detector 31. Once posture is determined, the ECU 30 determines whether the driver is distracted based at least in part on posture.

[0056] As an example, if a sudden deceleration event occurs, ECU 30 may trigger a review to determine the operator's posture. If the operator's posture is forward-facing and the orientation of the hands / arms is consistent with both hands gripping the steering wheel, ECU 30 may determine that the driver is not distracted and that the sudden deceleration is the result of an external event (e.g., an animal attempting to cross the road). In contrast, if the operator's posture includes one hand off the steering wheel or holding a cell phone and the torso and / or head rotated to face the identified hand, ECU 30 may determine that the driver was distracted based on the identified posture. Similarly, if the posture includes a food-like object in one or both hands and the object is positioned near the mouth, the operator's posture corresponds to eating and is an inattentive posture. The above examples are non-limiting examples of posture determination, and it is understood that a practical implementation may utilize any number of identified postures or relative positions within a posture to determine whether a driver is distracted in a single system.

[0057] In instances where continuous monitoring is performed, attitude detection, such as in the example above, is determined continuously or periodically without the need for a prerequisite abnormal operating event.

[0058] In another example, video feeds and other sensor information may be provided via a network connection to a remote controller / processor that makes decisions at an operations center instead of being onboard the vehicle, and the decisions are made in the same manner.

[0059] 4 is a flowchart of one exemplary method 100 for monitoring a driver. ECU 30 monitors one or more vehicle sensors (e.g., sensors 36-44) for abnormal driving events (step 102). If an abnormal driving event is not detected (“NO” at step 104), ECU 30 continues monitoring for abnormal driving events. If an abnormal driving event is detected (“YES” at step 104), ECU 30 acquires a specific image from cab camera 34 depicting the driver during the abnormal event (step 106). ECU 30 transmits the specific image to the fleet manager and / or stores the image in local repository 66 of abnormal driving images, and then resumes monitoring vehicle sensors for abnormal driving events (step 102).

[0060] In some embodiments, the cab camera 34 is an eye-tracking camera configured to record images of the driver in the cab of the vehicle 14 and determine the driver's gaze direction in the recorded images. Such cameras are commercially available from SmartEye (https: / / smarteye.se / ) and EyeSight (http: / / www.eyesight-tech.com / ). In one example, the cab camera 34 detects gaze by directing infrared or near-infrared light at the user's eyes and then measuring the reflection of that infrared light from the driver's eyes. The gaze direction can be ascertained based on the angle of reflection. In another example, the cab camera 34 infers the driver's gaze direction by determining a gaze vector from the overall shape of the driver's head and / or the symmetry of the driver's face in the recorded images. Both of these techniques are well known to those skilled in the art and will not be discussed in detail herein. In one example, the cab camera 34 is integrated into a driver information system and / or an instrument cluster.

[0061] In other examples, the cab camera 34 may provide images to a posture detector 31 configured to determine the driver's posture, including which direction the driver is facing, the position of their arms / hands relative to the direction the driver was facing, the position of their arms / hands relative to the steering wheel, and similar directions or positions. Posture tracking may be either rule-based using static rules established during configuration (e.g., if there are no hands on the steering wheel, the driver is inattentive) or machine learning-based using a machine learning algorithm such as a convolutional neural network (CNN) that learns distracted and non-distracted postures from a training set. In some examples, posture detection uses images from at least two different cameras to provide a complete view of the vehicle operator. For example, simultaneous images from a driver-side A-frame mounted camera and a passenger-side A-frame mounted camera may be used to provide a complete view of the driver, including features that may be hidden from a single camera, and posture may be determined based on features from both images. This example may involve fusing the two images into a single image, or analyzing each separately and combining the analyses, depending on the configuration of the ECU 30.

[0062] 5 is a flowchart of an example method 200 for monitoring a driver in which the cab camera 34 is an eye-tracking camera and the ECU 30 utilizes a CNN 67. The cab camera 34 records images of a driver in the cab of the vehicle 14 (step 202) and determines the driver's gaze direction in the recorded images (step 203). The ECU 30 determines whether the gaze is outside a predefined warning driver region for an amount of time that exceeds a predefined time threshold (step 204).

[0063] Referring now to FIG. 6 , an exemplary vehicle cab 70 is shown schematically with an exemplary predefined warning driver area 72, which includes the windshield 74 and the instrument cluster display 46, but excludes the center console display 46B and other areas that may be indicative of distracted driving and / or mobile device use, such as the driver's knee area 80, most of the driver and passenger windows 84A-B, etc.

[0064] In the example of Figure 6, camera monitor system units 86A-B mounted on A-frame pillars 92A-B are provided within alert driver area 72. Each camera monitor system unit 86A-B includes a respective electronic display 88A-B for providing an external vehicle video feed and may optionally include a respective camera 90A-B (which may optionally be used as cab camera 34 of Figure 2, if desired). In one example, camera monitor system units 86A-B are part of Stoneridge's MIRROREYE system.

[0065] Continuing with reference to FIG. 6 , FIG. 8 schematically illustrates a top view and a side view of a vehicle 14 including the embodiment of FIG. 6 , in which cameras 90A, 90B and at least one other camera 402, 73 are utilized by the ECU 30 to detect the driver's posture using the posture detector 31. Each camera defines a corresponding field of view 410, 420, 430 within the cab 70, and the corresponding fields of view overlap in an overlap region 440. In one example, the at least one other camera 73 is positioned generally forward of the driver, near the location of a traditional rearview mirror. In the example of FIGS. 6 and 8 , the camera 73 is positioned within the Class VIII view display 71. As used herein, generally forward of the driver refers to a camera whose field of view includes the driver while the camera is facing rearward. In another embodiment, the at least one other camera 402, 73 may include multiple cameras generally forward of the driver. A driver head position 442 is within the overlap region 440 while the driver is seated in the cab 70. In some examples, one or more of the cameras 90A, 90B, 402 include an eye-tracking system such as that described above with respect to the cab camera 34 (FIG. 2), which may be utilized by the ECU to supplement or enhance attitude detection.

[0066] In the example of FIG. 6 , each camera 90A, 90B, 402 is included within a corresponding mirror-replacement display of a camera mirror system, which includes a display configured to display at least fields of view II, IV, V, VI, and VIII. For use as a mirror-replacement display, each display includes a clean, clear line of sight of the driver, including the driver's face and other parts of the driver (e.g., hands, arms, shoulders, head, and torso), which can be used to determine the driver's overall posture while the driver is seated in the cab 70. Furthermore, because of their function, a well-trained driver will always face directly toward one or more monitors containing cameras 90A, 90B, 402, and will maintain contact with the steering / control system regardless of which direction they are facing, just as with a traditional mirror. In an alternative example, such as one in which a camera-replacement system is not used, the cameras 73, 90A, 90B, 402 may be positioned on the vehicle frame that would include a mirror-replacement display in a vehicle using a mirror-replacement system, such that a direct line of sight to the vehicle operator exists even if a physical display is omitted from that location on the vehicle. In the illustrated example, the locations include:

[0067] The expected face orientation and hand / arm positions help a neural network such as a CNN67 determine whether the driver is attentive or distracted based on the driver's posture, and whether the driver's attention is towards or away from the zone where the warning is occurring when the warning is issued. Using a camera and a CNN67 to detect distraction is called a distracted driver system.

[0068] In an example of using eye tracking to supplement or enhance a determined posture analysis, eye tracking can identify the direction a driver is looking when posture indicates otherwise. As an example, a posture that includes a hand in front of the mouth may indicate either a sneeze or a yawn. By combining posture determination with eye tracking, the system can determine if the driver's eyes are drooping (indicating a yawn due to fatigue) or if the driver's eyes are rapidly closing and reopening (indicating a sneeze).

[0069] In some examples, the distracted driving system can be integrated into an existing mirror replacement system because the mirror replacement monitors can be relatively easily retrofitted to incorporate cameras 90A, 90B, 402. Additionally, each mirror replacement monitor is connected to the vehicle ECU 30, and the incorporated cameras 73, 90A, 90B, 402 can piggyback on that connection and communicate with the ECU 30 without requiring a dedicated connection. In alternative examples, the cameras 90A, 90B, 402, 73 are independent of the mirror replacement system and can be freely positioned around the cab.

[0070] This, combined with the location advantages associated with the use of mirror replacement systems identified above, means that existing mirror replacement systems are highly suitable for retrofitting.

[0071] In addition to the cameras 90A, 90B, 402 incorporated into the mirror replacement monitor, one or more additional cameras 404 may be included within the cab 70 to provide additional fields of view for the driver or other vehicle operator. The additional cameras 404 may be focused on areas of the vehicle where distractions are more likely to occur, such as the center console where the driver may hold a mobile device (e.g., the center console display 47). Alternatively, the additional cameras 404 may be focused on parts of the driver's body in addition to the driver's face (e.g., the driver's arms or torso) that are utilized to determine the driver's overall posture. In a further alternative, the additional cameras may include a camera focused on areas of the vehicle where distractions may be present and a camera focused on parts of the driver's body that can assist in determining overall posture. The additional camera 404 may have a time-stamped image feed that can be correlated with the image feed from the cameras 90A, 90B, 402 in the mirror replacement monitor, and multiple simultaneous images can provide additional resolution to the determination of an attentive or inattentive driver by the CNN 67 running on the ECU 30 or remotely. In examples where the distracted driving detection system is local to the vehicle, the ECU may be configured to accept high-speed data input and may include hardware-accelerated machine learning to improve the interpretation of video by the CNN 67.

[0072] Continuing to refer to FIG. 6 and again to FIG. 5, if the driver's gaze is within the warning driver region 72 or is outside the warning driver region 72 for a time shorter than a predefined time threshold (“No” in step 204), the ECU 30 resumes monitoring the driver's gaze direction and / or posture.

[0073] Conversely, if in step 204 ECU 30 determines that the driver's gaze is outside of the warning driver region 72 for an amount of time greater than a predefined time threshold ("Yes" in step 204), or if ECU 30 determines that the driver's posture does not match the expected posture for attentive driving, ECU 30 determines that the gaze or posture corresponds to a potential distracted driving event (step 206) and provides to CNN 67 a particular image of the driver when the driver's gaze direction is outside of the warning driver region 72 or the posture when the posture does not match the expected posture for attentive driving (step 208).

[0074] The CNN 67 is trained using images depicting drivers using a mobile device, and the ECU 30 uses the CNN 67 to process the images and determine whether the driver is using a mobile device in a particular image (step 210). The training image set may include images of the driver texting, holding a cell phone up to their face while talking, or holding their hands near their face in a position suggestive of cell phone use (even if the cell phone is not visible). The use of the CNN 67 helps reduce false positives of distracted driver events. Training enables the CNN 67 to identify the relative positions and postures of vehicle components and body parts associated with distracted driving (e.g., hands positioned on the center console away from the steering wheel, eyes toward the center console with hands away from the steering wheel, a foreign object held next to the operator's mouth, or similar postures).

[0075] If mobile device use is detected (“YES” at step 212), the ECU 30 performs one or more predefined actions based on the potential distracted driving event, such as sending the image to the fleet manager 22 for review, storing the image for eventual review in a local repository of abnormal driving 66, and / or providing a warning to the driver (step 214). The warning may be provided, for example, as an audio notification via the vehicle speakers 48, an audio notification to a wireless headset worn by the driver (e.g., using the Bluetooth module 50), and / or a visual notification on the electronic display 46 in the cab 70. The particular image may then be used as additional training data for the CNN 67 as part of a supervised machine learning process (step 216).

[0076] Alternatively, if no mobile device is detected (“NO” at step 212 ), one or more (eg, all) of the predefined actions are omitted and ECU 30 proceeds to step 216 .

[0077] Similar to how ECU 30 can adjust the abnormal driving detection threshold, ECU 30 can also adjust the threshold used to determine when a driver's gaze indicates distracted driving and / or mobile device use. For example, in certain environments, such as inclement weather conditions (e.g., rain, snow, icy roads) and / or in areas with heavy traffic or heavy pedestrian traffic, warning driver region 72 may be narrowed from a default region and / or the time threshold used in step 204 may be shortened from a default value to enforce a more stringent level of driver attention. Conversely, in light traffic and / or good weather conditions (i.e., not icy, not snowy, not slippery), warning driver region 72 may be widened and / or the time threshold used in step 204 may be extended.

[0078] In one example, ECU 30 is configured to select one or both of the warning driver regions 72 based on the predefined time threshold of step 204 and the driver's experience level, which may provide stricter criteria for less experienced drivers and more lenient criteria for more experienced drivers.

[0079] Similarly, the threshold at which a warning is provided to the driver in step 214 may be selected based on the driver's experience level, with it being understood that such a warning may be more appropriate and / or useful to a less experienced driver than a more experienced driver.

[0080] As mentioned above, the ECU may include a driver attention model 68, a driver gaze model 69, and / or a posture detector 31. The use of such models can further refine the driver monitoring system 24, for example, by correlating posture with external objects and determining whether the driver is staring at a given object when they should be looking at other relevant objects in the area around the vehicle. For example, a rapid shift in gaze direction from side to side may indicate a distracted driving event, even if the driver's gaze is focused outside the vehicle (because the driver's attention is not fixed on any one situation long enough to indicate the driver's attention). Furthermore, an attention model showing a prolonged focus on a point near the driver's knees may indicate the driver is using their phone to access the Internet while driving.

[0081] 7 is a flowchart of another exemplary method 300 of monitoring a driver, in which a wireless activity detector 45 is used to monitor wireless signal transmissions based on any of the predefined criteria described above (e.g., signal strength, signal duration, and mobile device identifier) ​​(step 302). If a wireless signal from the driver's mobile device is detected ("Yes" in step 303), an image of the driver within the vehicle cab 70 is recorded and the image is provided to the CNN 67 (step 308). Steps 310-316 are performed similarly to steps 210-216 described above.

[0082] In one example, CNN 67 is omitted and ECU 30 simply transmits the images recorded in step 306 for review or stores the images for eventual review.

[0083] In one example, if the driver's mobile device is paired with the vehicle infotainment system or headset (e.g., via Bluetooth), steps 306-316 are skipped. In one example, wireless activity detector 45 detects whether the mobile device is paired with the headset and / or infotainment system by monitoring transmissions in the Bluetooth frequency band.

[0084] In one example, steps 302-303 are used in method 200 as an additional layer of detection prior to utilizing CNN 67 (e.g., between steps 206 and 208) such that a particular image is provided to CNN 67 only if wireless activity detector 45 detects evidence of a wireless signal from the driver's mobile device, corroborating a potential distracted driving event.

[0085] While exemplary embodiments have been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of the present disclosure. For that reason, the following claims should be studied to determine the scope and content of the present disclosure.

Claims

1. a camera mirror system including a plurality of cameras configured to record a rearward-facing exterior view; a first display screen assembly disposed on a driver's side of the vehicle and configured to display a first rearward-facing view generated by at least one of the plurality of cameras; and a second display screen assembly disposed on a passenger's side of the vehicle and configured to display a second rearward-facing view generated by at least one of the plurality of cameras; at least three cameras configured to record images of a driver within a cab of a vehicle, wherein a first camera of the at least three cameras is positioned within the first display screen assembly on a driver's side of the cab, a second camera of the at least three cameras is positioned within the second display screen assembly on a passenger's side of the cab, and a third camera of the at least three cameras faces toward the rear of the cab and defines a field of view that includes the face of the driver when the driver is looking forward; a controller in communication with the at least three cameras, the controller configured to determine a posture of the driver based on images from the at least three cameras; wherein at least one of the images originates from at least one of the first camera, the second camera, and the third camera; The controller is configured to use a machine learning algorithm to perform an analysis to determine whether the posture is an attentive posture or an inattentive posture, determine the driver's line of sight based at least in part on an analysis of at least three simultaneous images from the at least three cameras, and supplement the attentive / inattentive posture detection with the determined line of sight.

2. 2. The driver monitoring system of claim 1, wherein the first camera defines a field of view that includes at least the driver's face when the driver is looking out the driver's side window, as well as at least one arm of the driver, one hand of the driver, and one shoulder of the driver.

3. 3. The driver monitoring system of claim 2, wherein the second camera defines a field of view that includes the driver's face when the driver is looking out the passenger side window, as well as at least the other of the driver's arms, the driver's hands, and the driver's shoulders.

4. the controller is operable to acquire at least one particular image from a rolling video buffer recorded within a time frame corresponding to an abnormal event; The driver monitoring system of claim 1 , wherein the rolling video buffer includes video feeds originating from the at least three cameras.

5. 10. The driver monitoring system of claim 1, wherein the controller is further configured to analyze the driver's posture determined from the posture detector, thereby determining whether the posture is an attentive posture or an inattentive posture.

6. The driver monitoring system according to claim 1 , wherein the third camera is disposed substantially in front of the driver.

7. 10. The driver monitoring system of claim 1, wherein the first rearview view is a driver's side rearview mirror replacement view and the second rearview view is a passenger's side rearview mirror replacement view.

8. an eye-tracking system including a plurality of cameras, the eye-tracking system configured to record images of a driver in a cab of a vehicle using a first camera and a second camera and to determine a gaze direction of the driver using the recorded images, the first camera being disposed in a first display screen assembly located on a driver's side of the cab and configured to display a first rearview mirror replacement view, and the second camera being disposed in a second display screen assembly located on a passenger's side of the cab and configured to display a second rearview mirror replacement view; a third camera facing the rear of the cab and defining a field of view that includes the driver's face when the driver is looking forward; and a controller in communication with the eye-tracking camera, the controller configured to detect a potential distracted driving event based on the driver's gaze direction deviating from a predefined warning driver region for an amount of time that exceeds a predefined time threshold; Equipped with The driver monitoring system is configured, when the controller detects the potential distracted driving event, to determine the driver's posture based on images from the first camera, the second camera, and the third camera, perform an analysis using a convolutional neural network to determine whether the posture is an attentive posture or an inattentive posture, determine the driver's gaze based at least in part on an analysis of at least three simultaneous images from the first camera, the second camera, and the third camera, and supplement the attentive / inattentive posture detection with the determined gaze.

9. 10. The driver monitoring system of claim 8, further comprising a posture tracking system configured to determine a posture of the driver based in part on a position of a body part, the position of the body part relative to at least one of a vehicle component and a position of another body part.

10. The driver monitoring system of claim 9 , wherein the body part includes at least one of an arm and a torso of the driver.

11. 9. The driver monitoring system of claim 8, wherein the controller is configured to provide at least one image to the convolutional neural network, the convolutional neural network being trained to identify an inattentive driving posture using a first data set including images from the first camera position and a second data set including images from the second camera position.

12. The driver monitoring system of claim 8, wherein the convolutional neural network is configured to analyze recorded images and identify occurrences of the driver's use of a mobile device within the recorded images, and the neural network is configured to cause the driver monitoring system to perform at least one of transmitting the recorded images to a fleet manager and storing the recorded images in a local repository of abnormal driving images.

13. The controller acquiring additional images depicting the driver at random intervals from the eye-tracking camera or another camera; and Sending the additional images to a fleet manager, storing the additional images in a local repository of abnormal driving images, or both. The driver monitoring system of claim 8 configured to:

14. recording images of a driver within a cab of a vehicle using a driver monitoring system including at least three cameras, including a first camera, a second camera, and a third camera, wherein the first camera is disposed within a first display screen assembly located on a driver's side of the cab, the second camera is incorporated into a second display screen assembly located on a passenger's side of the cab, each of the display screen assemblies configured to display a rearview mirror replacement view, the first display screen assembly including the first camera and located on the driver's side of the cab, the second display screen assembly including the second camera and located on the passenger's side of the cab, and the third camera facing rearward of the cab and defining a field of view that includes the driver's face when the driver is looking forward; detecting an abnormal driving event of the vehicle based on input from at least one vehicle sensor; acquiring at least three specific images from the driver monitoring system depicting the driver during an abnormal event, each of the at least three specific images being from a respective one of the at least three cameras; performing at least one of transmitting the particular image to a fleet manager and storing the particular image in a local repository of abnormal driving images; determining a posture of the driver based on images from the first camera, the second camera, and the third camera when the abnormal driving event is detected; analyzing the posture using a machine learning algorithm to determine whether the posture is an attentive posture or an inattentive posture; determining a gaze of the driver based at least in part on an analysis of at least three simultaneous images of the at least three cameras, the gaze detection supplementing attentive / inattentive posture detection; A method for monitoring a driver, including:

15. determining a pose of the driver in the recorded images based at least in part on which of at least the first camera and the second camera includes the driver's face; detecting a potential distracted driving event based on the driver's posture depicted in a particular one of the recorded images being consistent with an inattentive posture for an amount of time that exceeds a predefined time threshold; in response to detecting the distracted driving event, performing at least one of transmitting the particular image to a fleet manager and storing the particular image in a local repository of abnormal driving images; 15. The method of claim 14, further comprising:

Images