Mobile Device Usage Monitoring for Commercial Vehicle Fleet Management

Abstract

The driver monitoring system includes at least two cameras configured to record images of a driver within a cab of a vehicle. A first camera is located on a driver's side of the cab and a second camera is located on a passenger's side of the cab. At least one of the at least two cameras is located at a mirror replacement display or a mirror replacement display location. A controller is in communication with the at least two cameras and configured to determine a driver's posture based on images from the at least two cameras. At least one of the images originates from the first camera and at least one of the second camera.

Description

[Technical field]

[0001] This application relates to driver monitoring, and in particular to recording images depicting a driver during an anomalous driving event.

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This disclosure is a continuation-in-part of U.S. Patent Application No. 16 / 845,228, filed April 10, 2020, which claims priority to U.S. Patent Application No. 17 / 145,891, filed January 11, 2021, which claims priority to U.S. Provisional Application No. 62 / 833,252, filed April 12, 2019, which is incorporated herein by reference in its entirety. [Background technology]

[0003] Commercial vehicle safety costs have risen dramatically, with distracted driving becoming the leading cause of accidents. Cell phone use is believed to be the leading cause of these types of distracted driving accidents. Summary of the Invention

[0004] In one exemplary embodiment, a driver monitoring system includes at least two cameras configured to record images of a driver in a cab of a vehicle, where a first camera is located on a driver's side of the cab and a second camera is located on a passenger side of the cab, and at least one of the at least two cameras is located at a mirror replacement display or a mirror replacement display position; and a controller in communication with the at least two cameras, the controller configured to determine a posture of the driver based on images from the at least two cameras, where at least one of the images is originated from the first camera and at least one of the second camera.

[0005] In another example of the above driver monitoring system, the at least two cameras include a third camera facing toward the rear of the cab and defining a field of view that includes the driver's face when the driver is looking forward.

[0006] In another example of any of the above driver monitoring systems, the first camera is integrated into the driver's side mirror replacement monitor and defines a field of view that includes at least the driver's face when the driver is looking out the driver's side window, as well as at least one of the driver's arms, one of the driver's hands, and one of the driver's shoulders.

[0007] In another example of any of the above driver monitoring systems, the second camera is incorporated into a passenger side mirror replacement monitor and defines a field of view that includes the driver's face when the driver is looking out the passenger side window, as well as at least the other of the driver's arms, the driver's hands, and the driver's shoulders.

[0008] In another example of any of the above driver monitoring systems, the controller is operable to obtain the at least one particular image from a rolling video buffer recorded within a time frame corresponding to the abnormal event, the rolling video buffer including video feeds originating from the at least two cameras.

[0009] In another example of any of the above driver monitoring systems, the controller is further configured to determine a line of sight of the driver based at least in part on analysis of at least two simultaneous images of the at least two cameras.

[0010] In another example of any of the above driver monitoring systems, the controller is configured to supplement the attentive / inatent posture detection with the gaze detection.

[0011] In another example of any of the above driver monitoring systems, the controller is further configured to analyze the driver's posture determined from the posture detector, thereby determining whether the posture is an attentive posture or an inattentive posture.

[0012] In another example of any of the above driver monitoring systems, the analysis to determine whether the posture is an attentive posture or an inattentive posture uses a machine learning algorithm.

[0013] In another example of any of the above driver monitoring systems, the at least two cameras include a third camera positioned generally in front of the driver.

[0014] In one exemplary embodiment, a driver monitoring system includes an eye-tracking system including a plurality of cameras configured to record images of a driver within a vehicle cab and determine a gaze direction of the driver using the recorded images, and a controller in communication with the eye-tracking cameras configured to detect a potential distracted driving event based on the driver's gaze direction deviating from a predefined warning driver region for an amount of time that exceeds a predefined time threshold.

[0015] Another example of the above driver monitoring system further includes a posture tracking system configured to determine a posture of the driver based in part on body part positions, the body part positions relative to at least one of vehicle components and other body part positions.

[0016] In another example of any of the above driver monitoring systems, the body part includes at least one of the driver's hand, arm, torso, and face.

[0017] In another example of any of the above driver monitoring systems, the plurality of cameras includes at least two cameras, and at least one of the at least two cameras is disposed within a mirror replacement monitor.

[0018] In another example of any of the above driver monitoring systems, the controller is configured to provide at least one image to a convolutional neural network, the convolutional neural network being trained to identify an inattentive driving posture using a first data set including images from a first camera position of the at least two cameras and a second data set including images from a second camera position of the at least two cameras.

[0019] In another example of any of the above driver monitoring systems, the neural network is configured to analyze recorded images and identify occurrences of use of a mobile device by the driver in the recorded images, and the neural network is configured to cause the driver monitoring system to at least one of transmit the recorded images to a fleet manager and store the recorded images in a local repository of abnormal driving images.

[0020] In another example of any of the above driver monitoring systems, the controller is configured to obtain additional images depicting the driver from the eye tracking camera or another camera at random intervals, and to transmit the additional images to a fleet manager, store the additional images in the local repository of abnormal driving images, or both.

[0021] An exemplary method of monitoring a driver includes recording images of a driver in a cab of a vehicle using a driver monitoring system including at least two cameras, including a first camera and a second camera, each of the first camera and the second camera being incorporated into a mirror replacement monitor; detecting an abnormal driving event of the vehicle based on input from at least one vehicle sensor; acquiring at least two specific images from the driver monitoring system depicting the driver during the abnormal event, each of the at least two specific images being from a respective one of the at least two cameras; and performing at least one of transmitting the specific images to a fleet manager and storing the specific images in a local repository of abnormal driving images.

[0022] Another example of the above exemplary method of monitoring a driver further includes determining a posture of the driver in the recorded images based at least in part on which of at least the first camera and the second camera includes the driver's face; detecting a potential distracted driving event based on the posture of the driver depicted in a particular image of the recorded images being consistent with an inattentive posture for an amount of time that exceeds a predefined time threshold; and performing at least one of transmitting the particular image to a fleet manager and storing the particular image in a local repository of abnormal driving images in response to detecting the distracted driving event. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a schematic diagram of an example of a fleet management system.

[0024] [Diagram 2] FIG. 2 is a schematic diagram of components of an exemplary driver monitoring system for each vehicle of FIG. 1.

[0025] [Diagram 3]FIG. 3 is a more detailed schematic diagram of the electronic control unit of FIG. 2.

[0026] [Figure 4] 4 is a flow chart of an exemplary method for monitoring a driver.

[0027] [Diagram 5] 4 is a flow chart of an example of another method for monitoring a driver.

[0028] [Figure 6] 1 is a schematic diagram of an example of a vehicle cab.

[0029] [Figure 7] 11 is a flow chart of an example of another method for monitoring a driver.

[0030] [Figure 8] FIG. 1 is a schematic diagram of an exemplary vehicle including a mirror replacement system adapted to facilitate driver monitoring. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The embodiments, examples and alternatives set forth in the claims and in the following description and drawings include any of their various aspects or respective individual features, which may be employed independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible.

[0032] 1 illustrates generally an example fleet management system 10 that includes a fleet 12 of vehicles 14A-N operable to communicate with a fleet manager 22 over a wide area network ("WAN") 16, such as the Internet. The vehicles 14 are operable to record images depicting the drivers of the vehicles 14 and to store or transmit those images, optionally along with associated event data that describes how the vehicles 14 are operating (e.g., acceleration events, steering events, braking events, close-in collisions, etc.).

[0033] In one example, the vehicles 14A-N transmit images and / or event data to the fleet manager 22 by transmitting images to the fleet management server 18, which is accessible by the computing device 20 of the fleet manager 22 overseeing the fleet 12. In one example, the vehicles 14A-N can transmit images and / or event data to the fleet manager 22 by bypassing the fleet management server 18 and transmitting to the computing device 20 of the fleet manager 22. In one example, in addition to or instead of transmitting images to the fleet manager 22, the vehicles 14 store images in a local repository within the vehicle 14. In one example, whether a given image is transmitted over the WAN 16 or stored in the local repository is based on whether the vehicle 14 is currently able to connect to the WAN 16. In the example of FIG. 1, the vehicles 14 are trucks, although it is understood that other commercial vehicles, such as delivery vans, may be used.

[0034] FIG. 2 is a schematic diagram of components of an exemplary driver monitoring system 24 provided in each vehicle 14. In the example of FIG. 2, the driver monitoring system 24 includes an electronic control unit (ECU) 30 operably connected to a telematics module 32, a cab camera 34, an acceleration sensor 36, a steering angle sensor 38, and a braking sensor 40. Although three sensors 34-38 are described, it will be understood that fewer or more sensors may be used. For example, the ECU 30 may be operably connected to an exterior camera 42 operable to record images of the vehicle 14's surrounding environment, an object detection sensor 44 operable to detect objects external to the vehicle 14, a wireless activity detector 45 operable to detect use of a wireless device by the driver, an electronic display 46, a vehicle speaker 48, and / or a Bluetooth module 50.

[0035] In one example, the electronic display 46 and speaker 48 are part of a driver information system ("DIS") that provides information regarding vehicle conditions (e.g., speed, engine RPM, etc.). In this example, the electronic display 46 may be part of a vehicle instrument cluster. As another example, the electronic display 46 may be a center console display that is part of an infotainment system that provides a combination of vehicle and entertainment information (e.g., current radio station, climate control, etc.). In one example, the ECU 30 is integrated into a DIS ECU (not shown) or telematics module 32. In some examples, a posture detector 31 is included within the ECU 30 of the driver monitoring system 24. The posture detector 31 analyzes images from multiple cameras, including the cab camera 24, to determine the overall posture of the driver and triggers one or more responses based on the determined posture. Images from multiple cameras may be analyzed individually and used to validate each other, or images from multiple cameras may be combined into a single image to provide a more complete set of information for the ECU 30 to analyze in determining the driver's posture.

[0036] In the example of Figure 2, ECU 30 is operatively connected to components 31-50 via a vehicle data bus 52, which may be a Controller Area Network ("CAN") bus. Of course, it will be understood that Figure 2 is only an example and that ECU 30 may be connected to certain ones of components 32-50 via other connections other than vehicle data bus 52.

[0037] FIG. 3 is a more detailed schematic diagram of the ECU 30. Referring now to FIG. 3, the ECU 30 includes a processor 60 operatively connected to a memory 62 and a communication interface 64. The processor 60 includes one or more processing circuits, such as a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), or the like. The memory 62 may include one or more types of memory, such as a read only memory (ROM), a random access memory, a cache memory, a flash memory device, an optical storage device, or the like. The memory 62 includes a local repository 66 of abnormal driving images, and may also optionally include a convolutional neural network ("CNN") 67, a driver attention model 68, and / or a driver gaze model 69. In some examples, a posture detector 31 is also included in the memory 62. The CNN 67 is operable, in some examples, to detect whether the driver is utilizing a mobile device in the cab of the vehicle 14. As used herein, a "mobile device" refers to a handheld electronic device, such as a mobile phone, a smart phone, a tablet, a personal media player, or the like. Although illustrated as part of the ECU 30, it is understood that the CNN 67 may instead be stored outside the vehicle 14, such as in the fleet management server 18. The communication interface 64 provides communication between the ECU 30 and other components (e.g., a wired connection to the vehicle data bus 52). In another example, the CNN 67 is operable to detect the driver's overall posture and trigger one or more responses in the ECU 30 based on whether the posture corresponds to attentive or inattentive driving. As used herein, attentive driving refers to posture, gaze, or other operator state that corresponds to the operator recognizing an existing condition and responding appropriately thereto. Inattentive driving refers to posture, gaze, or other operator state that does not correspond to the operator recognizing an existing condition and responding appropriately thereto. As an example, a driver is considered inattentive if they are not eating, using a cell phone, or engaged in other driving-related tasks.

[0038] Referring now to FIG. 2 with continued reference to FIG. 3, the cab camera 34, which may be an eye-tracking camera, is configured to record images of the driver within the cab of the vehicle 14, and each of the sensors 36-44 is configured to detect abnormal driving events of the vehicle based on predefined criteria corresponding to distracted driving. In an alternative example, the video feed from the cab camera 34 may be provided to a posture detector 31. The posture detector 31 is configured to utilize a CNN 67 to track the driver's posture including arm position and orientation, hand position and orientation, torso twist, relative orientation and / or position of hands and face, arm and hand position relative to the steering wheel, and any number of similar posture metrics that are combined to define the posture in which the driver is currently positioned. The determined posture is then correlated with a number of trained postures and identified as either attentive or inattentive. In yet another example, the CNN 67 may combine posture detection with gaze detection of the cab camera 34 to further improve its ability to distinguish between attentive and inattentive postures.

[0039] In some examples, detection of an abnormal driving event by one of the sensors 36-44 causes the ECU 30 to acquire a particular image taken by the cab camera 34, or a posture identified by the posture detector 31, depicting the driver during the abnormal event. The ECU 30 transmits the particular image or the identified posture to the fleet manager 22 using the telematics module 32 and / or stores the particular image or the identified posture in the local repository 66. An abnormal driving event may include a sudden acceleration, a sudden deceleration, a sudden change in steering trajectory, an unexpected long period of constant steering stability, a deviation from an expected road position determined by satellite navigation, or other similar events. Distracted driving may occur for a variety of reasons, including, but not limited to, cell phone use, eating, drowsiness, operating an entertainment system.

[0040] The acceleration sensor 36 is configured to detect abnormal acceleration events that may be indicative of distracted driving, such as abrupt acceleration or deceleration of the vehicle 14. Predefined criteria for the acceleration sensor 36 may include, for example, an acceleration rate that exceeds a predefined acceleration threshold or a deceleration rate that falls below a predefined deceleration threshold.

[0041] The steering angle sensor 38 is configured to detect abnormal steering events, such as a rapid change in steering wheel angle that may indicate a turn. For example, predefined criteria for the steering angle sensor 38 may include a change in steering angle that exceeds a predefined angle threshold within a predefined time period while the vehicle 14 is traveling at a speed that exceeds a predefined speed threshold, which may indicate a turn as a result of distracted driving.

[0042] Brake sensor 40 is configured to detect abnormal braking events, such as rapid braking of vehicle 14, and may be configured, for example, to measure changes in vehicle speed and / or control signals sent to the vehicle braking system.

[0043] The object detection sensor 44 may be, for example, a LIDAR ("Light Detection and Ranging") or RADAR ("Radio Detection and Ranging") sensor. The object detection sensor 44 may be used alone or in combination with the ECU 30 to detect near misses where a collision was narrowly avoided.

[0044] The telematics module 32 includes a wireless transceiver operable to transmit images over the WAN 16. In one example, the telematics module 32 is configured to use a predefined protocol standard, such as one or more 802.11 standards and / or one or more cellular standards (e.g., GSM, CDMA, LTE, etc.).

[0045] The wireless activity detector 45 includes an antenna configured to detect wireless signals and includes associated processing circuitry for determining whether the detected wireless signals are indicative of mobile device use within the cab of the vehicle 14 based on one or more predefined thresholds. Criteria used by the processing circuitry of the wireless activity detector 45 may include any one or combination of signal strength, signal duration, and mobile device identifier. Some examples of mobile device identifiers may include an International Mobile Subscriber Identity ("IMSI"), an Internet Protocol ("IP") address, a Media Access Control ("MAC") address, and if a mobile device identifier associated with a driver is detected, there is an increased likelihood that the signal transmission corresponds to the use of a mobile device by this driver, rather than a pedestrian or the driver of a nearby vehicle.

[0046] In one example, signal duration is used to distinguish between background activity, such as handovers between adjacent cells, where the driver is not actually using the mobile device, and active use of the mobile device (e.g., phone calls, video streaming, etc.), where the signal duration is likely to exceed a predefined signal length threshold.

[0047] In one example, wireless activity detector 45 is configured to limit its monitoring to frequency bands associated with known telecommunications standards, such as GSM band(s), CDMA band(s), LTE band(s), WiMax band(s), WiFi band(s), etc. In one example, wireless activity detector 45 includes multiple antennas, each tuned for a particular frequency band or set of frequency bands, and / or includes one or more antennas configured to sweep multiple such frequency bands.

[0048] In one example, the wireless activity detector 45 is configured to detect based at least in part on signal strength, such that a signal detected from a mobile device in the vehicle cab is more likely to be stronger than a signal from a mobile device in an adjacent vehicle.

[0049] In one example, the cab camera 34 is a video camera operable to provide a rolling buffer of a predefined period (e.g., 30 seconds) that overwrites itself if not backed up, and the ECU 30 is operable to obtain images from frames of the rolling video buffer within a time frame corresponding to an abnormal driving event. This may also provide an opportunity to store video leading up to and before an abnormal driving event to see what happened during and before the abnormal driving event.

[0050] In one example, the ECU 30 is configured to record additional images depicting the driver from the cab camera 34 at random intervals that may occur outside of an abnormal driving event, transmit the additional images to the fleet manager 22, store the additional images in a local repository 66 of abnormal driving images, or both. This random sampling can provide an additional level of compliance for the driver.

[0051] In one example, ECU 30 is configured to adjust predefined criteria used to determine an anomalous driving event based on at least one of traffic density, weather conditions, and object detection in the vicinity of vehicle 14. For example, in adverse weather conditions (e.g., rain, snow, icy roads) and in areas with high traffic or pedestrian traffic, the thresholds used to determine what constitutes anomalous driving may be lowered from default values ​​to more stringent criteria, particularly if vehicle 14 is a large commercial truck.

[0052] The determination of whether adverse weather conditions exist may be based, for example, on a weather forecast received by the ECU 30. The determination of whether the vehicle 14 is in a high traffic or pedestrian area may be based, for example, on traffic reports received by the ECU 30 and / or on object detection from the object detection sensor 44 or the external camera 42.

[0053] The predefined criteria used to detect abnormal driving events may also be selected based on the driver's experience level, which may provide stricter criteria for less experienced drivers and more lenient criteria for more experienced drivers.

[0054] In one example, the ECU 30 is configured to continually identify the driver's posture using posture detector 31 and trigger an inattentive driver response when a posture corresponding to inattentive driving is detected. As an example, the response may include an audio, visual, tactile, or other sensory alert provided to the driver. In another example, the inattentive driver response may be a command to store an image of the driver in an inattentive posture generated from a camera in the vehicle for supervisor review and / or further analysis.

[0055] In one example, the physical implementation described herein facilitates tracking and monitoring of a vehicle operator's posture by the ECU 30. When the ECU 30 detects an abnormal driving event, the ECU 30 reviews images from multiple video feeds to determine the driver's posture. Alternatively, posture is continuously monitored by the posture detector 31. Once posture is determined, the ECU 30 determines whether the driver is distracted based at least in part on posture.

[0056] As an example, if a sudden deceleration event occurs, the ECU 30 may trigger a review and determine the operator's posture. If the operator's posture is forward facing and the orientation of the hands / arms is consistent with both hands gripping the steering wheel, the ECU 30 may determine that the driver is not distracted and that the sudden deceleration is the result of an external event (e.g., an animal attempting to cross the road). In contrast, if the operator's posture includes one hand off the steering wheel or gripping a cell phone and the torso and / or head rotated to face the identified hand, the ECU 30 may determine that the driver was distracted based on the identified posture. Similarly, if the posture includes a food-like object in one or both hands and the position of the object is near the mouth, the operator's posture corresponds to eating and is an inattentive posture. It is understood that the above examples are non-limiting examples of posture determination and that practical implementations may utilize any number of identified postures or relative positions within a posture to determine whether a driver is distracted in a single system.

[0057] In instances where continuous monitoring is performed, attitude detection, such as the example above, is determined continuously or periodically without requiring a prerequisite abnormal operating event.

[0058] In another example, video feeds and other sensor information may be provided via a network connection to a remote controller / processor that makes decisions at an operations center instead of onboard the vehicle, and the decisions are made in the same manner.

[0059] 4 is a flow chart of one exemplary method 100 of monitoring a driver. The ECU 30 monitors one or more vehicle sensors (e.g., sensors 36-44) for abnormal driving events (step 102). If an abnormal driving event is not detected (step 104, "NO"), the ECU 30 continues monitoring for abnormal driving events. If an abnormal driving event is detected (step 104, "YES"), the ECU 30 acquires a particular image from the cab camera 34 depicting the driver during the abnormal event (step 106). The ECU 30 transmits the particular image to the fleet manager and / or stores the image in the local repository 66 of abnormal driving images, and then resumes monitoring the vehicle sensors for abnormal driving events (step 102).

[0060] In some embodiments, the cab camera is an eye-tracking camera configured to record an image of the driver in the cab of the vehicle 14 and determine the driver's gaze direction in the recorded image. Such cameras are commercially available from SmartEye (https: / / smarteye.se / ) and EyeSight (http: / / www.eyesight-tech.com / ). In one example, the cab camera 34 detects gaze by directing infrared or near-infrared light at the user's eye and then measuring the reflection of that infrared light from the driver's eye. The gaze direction can be ascertained based on the angle of reflection. In another example, the cab camera 34 infers the driver's gaze direction by determining a gaze vector from the overall shape of the driver's head and / or the symmetry of the driver's face in the recorded image. Both of these techniques are well known to those skilled in the art and will not be discussed in detail herein. In one example, the cab camera 34 is integrated into a driver information system and / or an instrument cluster.

[0061] In other examples, the cab camera 34 may provide images to a posture detector 31 configured to determine the posture of the driver including which direction the driver is facing, the position of the arms / hands relative to the direction the driver was facing, the position of the arms / hands relative to the steering wheel, and similar directions or positions. Posture tracking may be either rule based using static rules established during configuration (e.g., if there are no hands on the steering wheel, the driver is inattentive) or machine learning based using machine learning algorithms such as a convolutional neural network (CNN) that learns distracted and non-distracted postures from a training set. In some examples, posture detection uses images from at least two different cameras to provide a complete view of the vehicle operator. As an example, simultaneous images from a driver-side A-frame mounted camera and a passenger-side A-frame mounted camera may be used to provide a complete view of the driver including features that may be hidden from a single camera and posture may be determined based on features from both images. This example may also be fused into a single image, or each may be analyzed separately and the analyses combined, depending on the configuration of the ECU 30.

[0062] 5 is a flow chart of an example method 200 for monitoring a driver in which the cab camera 34 is an eye-tracking camera and the ECU 30 utilizes a CNN 67. The cab camera 34 records an image of the driver in the cab of the vehicle 14 (step 202) and determines the driver's gaze direction in the recorded image (step 203). The ECU 30 determines whether the gaze is outside a predefined warning driver region for an amount of time that exceeds a predefined time threshold (step 204).

[0063] Referring now to FIG. 6 , an example vehicle cab 70 is shown diagrammatically with an example predefined warning driver area 72 that includes the windshield 74 and the instrument cluster display 46, but excludes the center console display 46B, and other areas that may be indicative of distracted driving and / or mobile device use, such as the driver's knee area 80, most of the driver and passenger windows 84A-B, etc.

[0064] In the example of Figure 6, camera monitor system units 86A-B mounted on A-frame pillars 92A-B are provided within alert driver area 72. Each camera monitor system unit 86A-B includes a respective electronic display 88A-B for providing an external vehicle video feed and may optionally include a respective camera 90A-B (which may optionally be used as cab camera 34 of Figure 2, if desired). In one example, camera monitor system units 86A-B are part of Stoneridge's MIRROREYE system.

[0065] With continuing reference to FIG. 6, FIG. 8 shows a schematic top and side view of a vehicle 14 including the embodiment of FIG. 6, where the cameras 90A, 90B and at least one other camera 402, 73 are utilized by the ECU 30 to detect the driver's posture using the posture detector 31. Each camera defines a corresponding field of view 410, 420, 430 in the cab 70, and the corresponding fields of view overlap in an overlap region 440. In one example, the at least one other camera 73 is positioned generally in front of the driver, near the location of a conventional rearview mirror. In the example of FIG. 6 and FIG. 8, the camera 73 is positioned in a [Class VIII] view display 71. As used herein, generally in front of the driver refers to a camera that includes the driver in its field of view while the camera is facing backward. In another embodiment, the at least one other camera 402, 73 may include multiple cameras generally in front of the driver. The driver head position 442 is within the overlap region 440 while the driver is seated in the cab 70. In some examples, one or more of the cameras 90A, 90B, 402 include an eye-tracking system, such as that described above with respect to the cab camera 34 (FIG. 2), which may be utilized by the ECU to supplement or enhance attitude detection.

[0066] In the example of FIG. 6, each camera 90A, 90B, 402 is included within a corresponding mirror replacement display of a camera mirror system that includes a display configured to display at least field of view classes II, IV, V, VI, and VIII. For use as a mirror replacement display, each display includes a clean, clear line of sight to the driver, including the driver's face and other parts of the driver (e.g., hands, arms, shoulders, head, and torso) that can be used to determine the driver's overall posture while the driver is seated in the cab 70. Furthermore, for their function, a well-trained driver will always face directly toward one or more monitors that include cameras 90A, 90B, 402, as in the case of a conventional mirror, and will keep their hands in contact with the steering / control system regardless of which direction they are facing. In an alternative example where a camera replacement system is not used, the cameras 73, 90A, 90B, 402 may be located on the vehicle frame that would include a mirror replacement display in a vehicle using a mirror replacement system, such that a direct line of sight to the vehicle operator exists even if a physical display is omitted from that location on the vehicle. In the illustrated example, the locations include:

[0067] The expected face orientation and hand / arm positions help a neural network such as CNN67 to determine whether the driver is attentive or distracted based on the driver's posture, and whether the driver's attention is toward or away from the zone in which the warning is occurring when the warning occurs. Using a camera and CNN67 to detect distraction is called a distracted driver system.

[0068] In an example of using eye tracking to supplement or enhance a determined posture analysis, eye tracking can identify the direction the driver is looking when posture indicates otherwise. As an example, a posture that includes a hand in front of the mouth may indicate either a sneeze or a yawn. Combining posture determination with eye tracking, the system can determine if the driver's eyes are lowered (indicating a yawn due to fatigue) or if the driver's eyes are rapidly closing and reopening (indicating a sneeze).

[0069] In some examples, the mirror replacement monitors can be relatively easily retrofitted to incorporate the cameras 90A, 90B, 402 so that the distracted driving system can be integrated into an existing mirror replacement system. Additionally, each mirror replacement monitor is connected to the vehicle ECU 30, and the incorporated cameras 73, 90A, 90B, 402 can piggyback on that connection and communicate with the ECU 30 without the need for a dedicated connection. In alternative examples, the cameras 90A, 90B, 402, 73 are independent of the mirror replacement system and can be freely positioned around the cab.

[0070] This, combined with the location advantages associated with the use of the mirror replacement system identified above, means that existing mirror replacement systems are highly suitable for retrofit.

[0071] In addition to the cameras 90A, 90B, 402 incorporated into the mirror replacement monitor, one or more additional cameras 404 may be included in the cab 70 to provide additional views of the driver or other vehicle operator. The additional cameras 404 may be focused on areas of the vehicle where distractions are more likely to occur, such as the center console where the driver may hold a mobile device (e.g., center console display 47). Alternatively, the additional cameras 404 may be focused on parts of the driver's body that are utilized to determine the driver's overall posture in addition to the driver's face (e.g., the driver's arms or torso). In a further alternative, the additional cameras may include cameras focused on areas of the vehicle where distractions may be present and cameras focused on parts of the driver's body that can assist in determining the overall posture. The additional camera 404 may have a time-stamped image feed that can be correlated with the image feed from the cameras 90A, 90B, 402 in the mirror replacement monitor, and multiple simultaneous images can provide additional resolution to the determination of an attentive or inattentive driver by the CNN 67 running on the ECU 30 or remotely. In examples where the distracted driving detection system is local to the vehicle, the ECU may be configured to accept high speed data inputs and may include hardware accelerated machine learning to improve the interpretation of video by the CNN 67.

[0072] With continued reference to FIG. 6 and again to FIG. 5, if the driver's gaze is within the warning driver region 72 or is outside the warning driver region 72 for a time shorter than a predefined time threshold ("NO" at step 204), the ECU 30 resumes monitoring the driver's gaze direction and / or posture.

[0073] Conversely, if in step 204 the ECU 30 determines that the driver's gaze is outside the alert driver region 72 for an amount of time greater than a predefined time threshold ("YES" at step 204) or that the driver's posture does not match the expected posture for attentive driving, the ECU 30 determines that the gaze or posture corresponds to a potential distracted driving event (step 206) and provides to the CNN 67 a particular image of the driver when the driver's gaze direction is outside the alert driver region 72 or the posture when the driver's posture does not match the expected posture for attentive driving (step 208).

[0074] The CNN 67 is trained with images depicting a driver using a mobile device, and the ECU 30 utilizes the CNN 67 to process the images and determine whether the driver is using a mobile device in a particular image (step 210). The training image set may include images of the driver sending a text message, holding a cell phone up to their face while talking, holding their hands near their face in a position suggestive of cell phone use (even if the cell phone is not visible), etc. The use of the CNN 67 helps reduce false positives of distracted driver events. The training enables the CNN 67 to identify the relative positions and postures of vehicle components and body parts associated with distracted driving (e.g., hands away from the steering wheel and positioned on the center console, eyes toward the center console and hands away from the steering wheel, a foreign object being held next to the operator's mouth, or similar postures).

[0075] If mobile device use is detected (step 212, "YES"), the ECU 30 performs one or more predefined actions based on the potential distracted driving event, such as sending the images to the fleet manager 22 for review, storing the images for eventual review in a local repository of abnormal driving 66, and / or providing a warning to the driver (step 214). The warning may be provided, for example, as an audio notification via the vehicle speaker 48, an audio notification to a wireless headset worn by the driver (e.g., using the Bluetooth module 50), and / or a visual notification on the electronic display 46 in the cab 70. The particular image may then be used as additional training data for the CNN 67 as part of a supervised machine learning process (step 216).

[0076] Alternatively, if no mobile device is detected (“NO” at step 212 ), one or more (eg, all) of the predefined actions are omitted and ECU 30 proceeds to step 216 .

[0077] Similar to how the ECU 30 can adjust the abnormal driving detection thresholds, the ECU 30 can also adjust the thresholds used to determine when the driver's gaze indicates distracted driving and / or mobile device use. For example, in certain environments, such as inclement weather conditions (e.g., rain, snow, icy roads) and / or heavy traffic or pedestrian areas, the warning driver region 72 may be narrowed from a default region and / or the time thresholds used in step 204 may be shortened from a default value to enforce a more stringent level of driver attention. Conversely, in light traffic and / or good weather conditions (i.e., not icy, not snowy, not slippery), the warning driver region 72 may be widened and / or the time thresholds in step 204 may be extended.

[0078] In one example, ECU 30 is configured to select one or both of the warning driver regions 72 based on the predefined time thresholds of step 204 and the driver's experience level. This may provide stricter criteria for less experienced drivers and more lenient criteria for more experienced drivers.

[0079] Similarly, the threshold at which a warning is provided to the driver in step 214 may be selected based on the driver's experience level, it being understood that such a warning may be more appropriate and / or useful to a less experienced driver than a more experienced driver.

[0080] As mentioned above, the ECU may include a driver attention model 68, a driver gaze model 69, and / or a posture detector 31. The use of such models can layer further improvements to the driver monitoring system 24, for example, by correlating posture with external objects and determining whether the driver is staring at a given object when the driver should be looking at other relevant objects in the area around the vehicle. For example, a rapid shift in gaze direction from side to side may indicate a distracted driving event, even if the driver's gaze is focused outside the vehicle (because the driver's attention is not fixed on any situation for a sufficient period of time to indicate the driver's attention). Additionally, an attention model showing a prolonged focus on a point near the driver's knees may indicate that the driver is using a phone to access the Internet while driving.

[0081] 7 is a flow chart of another example method 300 of monitoring a driver, in which a wireless activity detector 45 is used to monitor wireless signal transmissions based on any of the predefined criteria discussed above (e.g., signal strength, signal duration, and mobile device identifier) ​​(step 302). If a wireless signal from the driver's mobile device is detected ("YES" in step 303), an image of the driver within the vehicle cab 70 is recorded and the image is provided to the CNN 67 (step 308). Steps 310-316 are performed similarly to steps 210-216 described above.

[0082] In one example, CNN 67 is omitted and ECU 30 simply transmits the images recorded in step 306 for review or stores the images for eventual review.

[0083] In one example, if the driver's mobile device is paired with the vehicle infotainment system or headset (e.g., via Bluetooth), steps 306-316 are skipped. In one example, wireless activity detector 45 detects whether the mobile device is paired with a headset and / or infotainment system by monitoring transmissions in the Bluetooth frequency band.

[0084] In one example, steps 302-303 are used in method 200 as an additional layer of detection prior to utilizing CNN 67 (e.g., between steps 206 and 208) such that a particular image is provided to CNN 67 only if wireless activity detector 45 corroborates a potential distracted driving event by detecting evidence of a wireless signal from the driver's mobile device.

[0085] Although exemplary embodiments have been disclosed, a person of ordinary skill in this art would recognize that certain modifications would come within the scope of the disclosure. For that reason, the following claims should be studied to determine the scope and content of the disclosure.

Claims

1. A camera mirror system comprising a plurality of cameras configured to record a rearward facing external view, a first display screen assembly positioned on a driver's side of the vehicle and configured to display a first rearward facing view generated by at least one of the plurality of cameras, and a second display screen assembly positioned on a passenger's side of the vehicle and configured to display a second rearward facing view generated by at least one of the plurality of cameras; at least two cameras configured to record images of a driver within a cab of a vehicle, a first camera of the at least two cameras being disposed within the first display screen assembly on a driver's side of the cab and a second camera of the at least two cameras being disposed within the second display screen assembly on a passenger's side of the cab; a controller in communication with the at least two cameras, the controller configured to determine a posture of the driver based on images from the at least two cameras; and at least one of the images originates from at least one of the first camera and the second camera.

2. the at least two cameras include a third camera; The driver monitoring system of claim 1 , wherein the third camera faces toward the rear of the cab and defines a field of view that includes the driver's face when the driver is looking forward.

3. 2. The driver monitoring system of claim 1, wherein the first camera defines a field of view that includes at least the driver's face when the driver is looking out the driver's side window, as well as at least one arm of the driver, one hand of the driver, and one shoulder of the driver.

4. 4. The driver monitoring system of claim 3, wherein the second camera defines a field of view that includes the driver's face when the driver is looking out the passenger side window, as well as at least the other of the driver's arms, the driver's hands, and the driver's shoulders.

5. the controller is operable to obtain at least one particular image from the rolling video buffer recorded within a time frame corresponding to the abnormal event; The driver monitoring system of claim 1 , wherein the rolling video buffer includes video feeds originating from the at least two cameras.

6. The driver monitoring system of claim 1 , wherein the controller is further configured to determine a line of sight of the driver based at least in part on an analysis of at least two simultaneous images of the at least two cameras.

7. The driver monitoring system of claim 6 , wherein the controller is configured to supplement attentive / inattentive posture detection with gaze detection.

8. A driver monitoring system as described in claim 7, wherein the analysis to determine whether the posture is an attentive posture or an inattentive posture uses a machine learning algorithm.

9. 2. The driver monitoring system of claim 1, wherein the controller is further configured to analyze the driver's posture determined from a posture detector, thereby determining whether the posture is an attentive posture or an inattentive posture.

10. The driver monitoring system of claim 1 , wherein the at least two cameras include a third camera positioned generally in front of the driver.

11. A driver monitoring system as described in claim 1, wherein the first rearward looking view is a driver's side rearview mirror replacement view and the second rearward looking view is a passenger side rearview mirror replacement view.

12. an eye-tracking system including a plurality of cameras, the eye-tracking system configured to record images of a driver in a cab of a vehicle using a first camera and a second camera and to determine a gaze direction of the driver using the recorded images, the first camera being disposed in a first display screen assembly located on a driver's side of the cab and configured to display a first rearview mirror replacement view, and the second camera being disposed in a second display screen assembly located on a passenger side of the cab and configured to display a second rearview mirror replacement view; a controller in communication with the eye-tracking camera, the controller configured to detect a potential distracted driving event based on the driver's eye gaze direction deviating from a predefined warning driver region for an amount of time that exceeds a predefined time threshold; A driver monitoring system comprising:

13. 13. The driver monitoring system of claim 12, further comprising a posture tracking system configured to determine a posture of the driver based in part on a position of a body part, the position of the body part relative to at least one of a vehicle component and a position of another body part.

14. The driver monitoring system of claim 13 , wherein the body part includes at least one of an arm and a torso of the driver.

15. 13. The driver monitoring system of claim 12, wherein the controller is configured to provide at least one image to a convolutional neural network, the convolutional neural network being trained to identify an inattentive driving posture using a first data set including images from a first camera position of the at least two cameras and a second data set including images from a second camera position of the at least two cameras.

16. 13. The driver monitoring system of claim 12, wherein a neural network is configured to analyze recorded images and identify occurrences of use of a mobile device by the driver in the recorded images, and wherein the neural network is configured to cause the driver monitoring system to at least one of transmit the recorded images to a fleet manager and store the recorded images in a local repository of abnormal driving images.

17. The controller: acquiring additional images depicting the driver at random intervals from the eye-tracking camera or another camera; and Sending the additional images to a fleet manager, storing the additional images in a local repository of abnormal driving images, or both. The driver monitoring system of claim 12 configured to:

18. recording an image of a driver within a cab of a vehicle using a driver monitoring system including at least two cameras including a first camera and a second camera, each of the first camera and the second camera being incorporated into a corresponding display screen assembly, each of the display screen assemblies being configured to display a rearview mirror replacement view, a first display screen assembly of the corresponding display screen assemblies including the first camera and located on a driver's side of the cab, and a second display screen assembly of the corresponding display screen assemblies including the second camera and located on a passenger's side of the cab; detecting an abnormal driving event of the vehicle based on input from at least one vehicle sensor; acquiring at least two specific images from the driver monitoring system depicting the driver during an abnormal event, each of the at least two specific images being from a respective one of the at least two cameras; transmitting the particular image to a fleet manager and storing the particular image in a local repository of abnormal driving images; A method for monitoring a driver, comprising:

19. determining a pose of the driver in the recorded images based at least in part on which of at least the first camera and the second camera includes a face of the driver; detecting a potential distracted driving event based on the driver's posture depicted in a particular image of the recorded images being consistent with an inattentive posture for an amount of time that exceeds a predefined time threshold; in response to detecting the distracted driving event, at least one of transmitting the particular image to a fleet manager and storing the particular image in a local repository of abnormal driving images; 20. The method of claim 18, further comprising:

Images