Display camera calibration for camera mirror systems

Abstract

The camera mirror system for a vehicle includes, among other things, a first and a second exterior camera configured to provide a captured image of the exterior of the vehicle corresponding to a legally prescribed view. The system further includes a first and a second display in the vehicle cab configured to depict the captured image from the first and the second exterior cameras, respectively. The system further includes a first and a second interior camera configured to capture a driver image in the vehicle cab. The first and the second interior cameras each have a first and a second interior field of view that overlap each other in a vehicle driver area in the vehicle cab, the first display being within the second interior field of view and the second display being within the first interior field of view. The system further includes a controller in communication with the first and the second displays and the first and the second interior cameras. The controller is configured to monitor driver activity in the vehicle driver area using the first and the second interior cameras. The controller is configured to initiate a calibration procedure in response to a calibration event. The calibration procedure includes first and second displays showing the first and second patterns, respectively, and the first and second interior cameras detecting the second and first patterns, respectively, and calibrating the first and second cameras with respect to the vehicle driver area to establish distances between the first and second interior cameras, respectively, and the second and first patterns, respectively.

Description

[Technical field]

[0001] The present disclosure relates to a camera mirror system (CMS) for use in commercial trucks that incorporates an in-vehicle camera to identify driver gaze and / or posture to recognize driver distraction events, for example.

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 63 / 129927, filed December 23, 2020. [Background technology]

[0003] There is growing interest among vehicle manufacturers and fleet operators in detecting events related to distracted driving. One approach to detecting such events involves training a camera on the driver. Algorithms are used to identify driver behaviors that indicate a driver is distracted, such as the driver looking away from the road, eating, or feeling drowsy. The reliability of these algorithms and their ability to detect events may depend on accurately determining the position of the camera relative to the driver. Summary of the Invention

[0004] In one exemplary embodiment, a camera mirror system for a vehicle includes first and second exterior cameras configured to provide captured images of the vehicle exterior that correspond to a legally prescribed view, among other things. The system further includes first and second displays in a vehicle cab configured to depict the captured images from the first and second exterior cameras, respectively. The system further includes first and second interior cameras configured to capture driver images in the vehicle cab. The first and second interior cameras each have first and second interior fields of view that overlap one another in a vehicle driver area in the vehicle cab, the first display being within the second interior field of view and the second display being within the first interior field of view. The system further includes a controller in communication with the first and second displays and the first and second interior cameras. The controller is configured to monitor driver activity in the vehicle driver area using the first and second interior cameras. The controller is configured to initiate a calibration procedure in response to a calibration event. The calibration procedure includes first and second displays showing first and second patterns, respectively, and the first and second interior cameras detecting the second and first patterns, respectively, and calibrating the first and second cameras with respect to a vehicle driver area to establish distances between the first and second interior cameras, respectively, and the second and first patterns, respectively.

[0005] In a further embodiment of any of the above, the legally defined views include a Class II view and a Class IV view.

[0006] In a further embodiment of any of the above, the first and second displays are disposed on A-pillars on the left and right sides, respectively, of the vehicle cabin.

[0007] In a further embodiment of any of the above, the first and second in-vehicle cameras are provided on the first and second displays, respectively, and configured to be movable with the first and second displays, respectively, the first and second in-vehicle cameras having mutually adjustable operating positions.

[0008] In a further embodiment of any of the above, the controller is configured to determine a distance from each of the first and second in-vehicle cameras to a driver characteristic to evaluate the driver activity.

[0009] In a further embodiment of any of the above, the driver activities include at least one of driver gaze and driver posture.

[0010] In a further embodiment of any of the above, the calibration event corresponds to at least one of the first and second in-vehicle cameras deviating from a nominal operating position.

[0011] In a further embodiment of any of the above, the calibration event corresponds to at least one of the first and second displays deviating from the nominal operating position.

[0012] In a further embodiment of any of the above, the calibration event is configured to occur within a calibration window.

[0013] In a further embodiment of any of the above, the calibration window is during a vehicle start-up procedure.

[0014] In a further embodiment of any of the above, the calibration window is less than a threshold vehicle speed.

[0015] In a further embodiment of any of the above, at least one of the first and second patterns includes a plurality of lines.

[0016] In a further embodiment of any of the above, at least one of the first and second patterns is a black and white checkerboard.

[0017] In another exemplary embodiment, a method of calibrating an in-cabin camera used for driver observation within a vehicle cab, the method configured to be used with a camera mirror system having first and second exterior cameras configured to capture images of the exterior of the vehicle corresponding to a legally prescribed view, and first and second displays in the vehicle cab configured to depict images captured from the first and second cameras, respectively, the method includes, inter alia, the steps of providing an in-cabin camera movable between operating positions; detecting a calibration event including the camera being moved between the operating positions; displaying a pattern on at least one of the first and second displays after detection of the calibration event; detecting the pattern with the mounted in-cabin camera; and calibrating the in-cabin camera with the pattern to establish a distance between the in-cabin camera and the pattern.

[0018] In a further embodiment of any of the above, the in-car camera is one of a first and a second in-car camera, the first in-car camera having the second display within its field of view and the second in-car camera having the first display within its field of view.

[0019] In a further embodiment of any of the above, the first and second in-car cameras are provided on the first and second displays, respectively, and are configured to be movable with the first and second displays, respectively.

[0020] In any other embodiment above, a further distance for a driver characteristic can be determined based on the distance, the driver characteristic being related to a driver activity.

[0021] In a further embodiment of any of the above, the driver activities include at least one of driver gaze and driver posture.

[0022] In a further embodiment of any of the above, the calibration event occurs during at least one of a vehicle start-up procedure and below a threshold vehicle speed.

[0023] In a further embodiment of any of the above, the displaying step includes displaying the pattern on the first and second displays, the detecting step includes detecting each pattern by a respective other of the first and second in-vehicle cameras mounted relative to a respective other of at least one of the first and second displays, and the calibrating step includes calibrating the respective other of the first and second in-vehicle cameras with the respective pattern to establish a distance between the respective other of the first and second in-vehicle cameras and the respective pattern.

[0024] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0025] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera mirror system (CMS) used to provide at least Class II and Class IV views.

[0026] [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera mirror system providing Class II, Class IV, Class V and Class VI views.

[0027] [Diagram 2] FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an in-cabin camera.

[0028] [Diagram 3]FIG. 2 is a schematic diagram of components of an exemplary driver monitoring system for the vehicle of FIG. 1.

[0029] [Figure 4] FIG. 4 is a more detailed schematic diagram of the electronic control unit of FIG. 3.

[0030] [Diagram 5] 4 is a flow chart of an exemplary method for monitoring a driver.

[0031] [Figure 6] 4 is a flow chart of an exemplary method for calibrating an in-car camera for use in a driving monitoring system.

[0032] [Figure 7] FIG. 1 is a perspective view showing a schematic calibration procedure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in relation to one embodiment are applicable to all embodiments, except where such features are incompatible.

[0034] A schematic diagram of a commercial vehicle 10 is shown in Figures 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. Although commercial trucks are contemplated in this disclosure, the invention may be applied to other types of vehicles. The vehicle 10 incorporates a camera mirror system (CMS) 15 (Figure 2) with driver and passenger side camera arms 16a, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely.

[0035] Each camera arm 16a, 16b includes a base that is fixed to, for example, the vehicle cab 12. A pivot arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20a, 20b is disposed within the camera arm. Each of the exterior cameras 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1B), which are legally defined views in the commercial trucking industry. EX1 , FOV EX2 If desired, multiple cameras may be used in each camera arm 16a, 16b to provide these views. Each arm may also provide a housing that encloses electronics configured to provide various features of the CMS 15.

[0036] First and second video displays 18a, 18b are positioned within the vehicle cab 12 on or near the A-pillars on the driver's and passenger's sides, respectively, to display Class II and Class IV views on each side of the vehicle 10, thereby providing rearward views along the vehicle 10 captured by exterior cameras 20a, 20b.

[0037] If Class V and Class VI view video is also required, a camera housing 16c and camera 20c may be located at or near the front of the vehicle 10 to provide these views (FIG. 1B). A third display 18c located within the cab vehicle 12 near the top center of the windshield may be used to display the Class V and Class VI views toward the front of the vehicle 10 to the driver.

[0038] The displays 18a, 18b, 18c face a driver area 24 within the cab 22 where a driver is seated in a driver's seat 26. A driver monitoring system (DMS) 29 is desirably incorporated into the CMS 15 to evaluate driver activity in the driver area 24 and identify distracted driver behavior. An example of a DMS 29 is shown in FIG. 2. First and second in-vehicle cameras 28a, 28b are positioned within the vehicle cab 22 near the first and second displays 18a, 18b and configured to capture images of the driver within the vehicle cab 22. The first and second in-vehicle cameras 28a, 28b each have a first and second in-vehicle field of view (FOV) that at least partially overlap one another within the driver area 24. IN1 ,FOV IN2 The first display 18a provides a second in-vehicle field of view (FOV). IN2 The second display 18b is positioned to be located within the first in-vehicle field of view FOV IN1 Driver activity may be determined by monitoring the driver's gaze and / or posture while operating the vehicle using the first and second in-vehicle cameras 28a, 28b.

[0039] 3 and with continued reference to FIG 2, first and second interior cameras 28a, 28b, which may be eye-tracking cameras, are configured to record images of the driver within the cab 22. One such eye-tracking system is disclosed in USSN 16 / 845,228, filed April 10, 2020, which is incorporated by reference herein in its entirety.

[0040] The sensors 36-44, 54-56 associated with the DMS 29 are configured to detect abnormal driving events of the vehicle based on predefined criteria corresponding to distracted driving, for example, by determining the driver's gaze direction in the recorded images. The ECU or controller 30 is in communication with the first and second interior cameras 28a, 28b and is configured to detect a potential distracted driving event based on the driver's gaze direction shown in a particular image in the recorded images being outside a predefined warning driver region for a time period exceeding a predefined time threshold. The controller 30 is operatively connected to the components 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 54, 56 via a vehicle data bus 52, which may be a controller area network ("CAN") bus. The controller 30 is configured to perform one or both of the following predefined actions based on the potential distracted driving event: sending the particular image to a fleet manager and storing the particular image in a local repository of abnormal driving images. The controller 30 may be one controller or multiple controllers, as appropriate.

[0041] 4 is a more detailed schematic diagram of the controller 30. The controller 30 includes a processor 60 operatively connected to a memory 62 and a communication interface 64. The processor 60 includes one or more processing circuits, such as a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), or the like. The memory 62 may include one or more types of memory, such as a read only memory (ROM), a random access memory, a cache memory, a flash memory device, an optical storage device, or the like. The memory 62 includes a local repository 66 of abnormal driving images and may also optionally include a convolutional neural network ("CNN") 67, a driver attention model 68, and / or a driver gaze model 69. In some examples, a posture detector 70 is also included in the memory 62. The CNN 67 is operable, in some examples, to detect whether the driver is utilizing a mobile device in the cab of the vehicle 10. As used herein, "mobile device" refers to a handheld electronic device, such as a mobile phone, a smart phone, a tablet, a personal media player, or the like. While illustrated as part of the controller 30, it is understood that the CNN 67 may instead be stored external to the vehicle 10, such as on a fleet management server. The communications interface 64 provides communications between the controller 30 and other components (e.g., a wired connection to the vehicle data bus 52). In another example, the CNN 67 is operable to detect the driver's overall posture and trigger one or more responses in the controller 30 based on whether the posture corresponds to attentive or inattentive driving.

[0042] In some embodiments, the first and second interior cameras 28a, 28b are provided by eye-tracking cameras configured to record images of the driver in the cab of the vehicle 10 and determine the driver's gaze direction in the recorded images. Such cameras are commercially available from SmartEye (https: / / smarteye.se / ) and EyeSight (http: / / www.eyesight-tech.com / ). In one example, the first and second interior cameras 28a, 28b detect gaze by directing infrared or near-infrared light at the user's eye and then measuring the reflection of that infrared light from the driver's eye. The gaze direction can be ascertained based on the angle of reflection. In another example, the first and second interior cameras 28a, 28b infer the driver's gaze direction by determining a gaze vector from the general shape of the driver's head and / or the symmetry of the driver's face in the recorded images. Both of these techniques are well known to those skilled in the art and will not be discussed in detail herein. In one example, the first and second interior cameras 28a, 28b are integrated into the driver information system and / or the instrument cluster 46.

[0043] The first and second in-vehicle cameras 28a, 28b can also be used to detect the driver's body position, e.g., leaning forward or to the left or right, or driving posture related to the position of the driver's limbs. Determining posture can provide additional information related to the driver's activity and where the driver's attention is focused.

[0044] The posture detector 70 is configured to utilize the CNN 67 to track the driver's posture, including arm position and orientation, hand position and orientation, torso twist, relative orientation and / or position of hands and face, arm and hand position relative to the steering system, and any number of similar posture metrics that combine to define the posture the driver is currently in. The determined posture is then correlated with a number of trained postures and identified as either attentive or inattentive. In yet another example, the CNN 67 can combine posture detection with gaze detection of the first and second in-vehicle cameras 28a, 28b to further improve its ability to distinguish between attentive and inattentive postures.

[0045] In one example, the controller 30 is configured to continually identify the driver's posture using posture detector 70 and trigger an inattentive driver response when a posture corresponding to inattentive driving is detected. As an example, the response may include an audio, visual, tactile, or other sensory alert provided to the driver. In another example, the inattentive driver response may be a command to store an image of the driver in an inattentive posture generated from a camera in the vehicle for supervisor review and / or further analysis.

[0046] In one example, the physical implementation described herein facilitates tracking and monitoring of a vehicle operator's posture by the controller 30. When the controller 30 detects an abnormal driving event, the controller 30 reviews images from multiple video feeds to determine the driver's posture. Alternatively, posture is continuously monitored by the posture detector 70. Once posture is determined, the controller 30 determines whether the driver is distracted based at least in part on posture.

[0047] 5 is a flow chart of one exemplary method 100 of monitoring a driver. The controller 30 monitors one or more vehicle sensors (e.g., sensors 36-44, 54-56) for abnormal driving events (step 102). If an abnormal driving event is not detected ("No" at step 104), the controller 30 continues monitoring for abnormal driving events. If an abnormal driving event is detected ("Yes" at step 104), the controller 30 acquires a particular image from the cab camera 34 depicting the driver during the abnormal event (step 106). The controller 30 transmits the particular image to the fleet manager and / or stores the image in the local repository 66 of abnormal driving images, and then resumes monitoring the vehicle sensors for abnormal driving events (step 108).

[0048] Referring to FIG. 7, the first and second in-car cameras 28a, 28b are provided on the first and second displays 18a, 18b, respectively, which are mounted, for example, on a display housing 80. The housing 80 is secured to a vehicle structure, such as an A-pillar 82, via a mounting bracket 86 having an adjustable connection 88. The adjustable connection 88 may be a ball and socket or other rotating joint, allowing the driver to orient the display in a desired direction. The adjustment may be used, for example, to reduce glare from the display screen. It should also be understood that the first and second in-car cameras 28a, 28b may be mounted separately from the first and second displays 18a, 18b, such as on the A-pillar 82 or on the instrument panel. Although the first and second in-car cameras 28a, 28b are shown on or near the first and second displays 18a, 18b, one or both of the in-car cameras may be located elsewhere, such as on or near the display 18c. In any case, at least one of the first and second interior cameras 28a, 28b has its field of view FOV IN1 , FOV IN2 are adjustable relative to the driver area 24.

[0049] During vehicle assembly, the first and second interior cameras 28a, 28b may undergo a calibration procedure to ensure that they encompass the desired area within the vehicle cab 12 for monitoring driver activity. The controller 30 relies on the capture of accurate and coordinated images of the driver area 24 by the first and second interior cameras 28a, 28b (e.g., driver feature coordinate data and / or image stitching). Thus, if either of the first and second interior cameras 28a, 28b is disturbed or otherwise adjusted after this initial calibration, the controller 30 may no longer be able to accurately interpret the driver activity. Thus, if the first and second interior cameras 28a, 28b are moved by directly adjusting the display or the interior cameras, they must be recalibrated. A warning (e.g., audible or visual) may be provided to the driver until the interior cameras are recalibrated.

[0050] A controller, for example, controller 30, is in communication with the first and second displays 18a, 18b and the first and second in-vehicle cameras 28a, 28b. The controller 30 is configured to monitor driver activity in the vehicle driver area 24 using the first and second in-vehicle cameras 28a, 28b having mutually adjustable operating positions. The controller 30 is configured to determine distances from each of the first and second in-vehicle cameras 28a, 28b to driver features (e.g., eyes, head, etc.) to evaluate the driver activity. The driver activity includes at least one of driver gaze and driver posture, as previously described. According to the present disclosure, the controller 30 is configured to initiate a calibration procedure in response to a calibration event, as shown at 200 in FIG. 6.

[0051] Referring to FIG. 6, during operation, the calibration procedure 200 includes detecting a calibration event (step 202) that includes at least one of the first and second in-car cameras 28a, 28b being moved between operating positions. The calibration event is configured to occur within a calibration window, for example, during the vehicle's start-up procedure and / or while the vehicle is below a threshold vehicle speed. For example, the calibration event corresponds to at least one of the first and second in-car cameras 28a, 28b moving out of a reference operating position and / or at least one of the first and second displays 18a, 18b moving out of a reference operating position. For example, the displays may be adjustable to allow the driver to position the displays to minimize glare. For in-car cameras attached to displays, moving the display may move the cameras, requiring recalibration of the cameras. Thus, if only one camera is moved, perhaps only one camera is recalibrated using the displays visible to that camera.

[0052] The pattern (at 84 in FIG. 7) is displayed on at least one of the first and second displays after calibration event detection 202 (step 204). In one example, the pattern 84 is presented on both displays to establish the distance between each of the first and second in-car cameras 28a, 28b and the second and first patterns 84, respectively. In one example, at least one of the first and second patterns includes multiple lines (e.g., "T", "X", etc.) that are easier to detect. Although the patterns 84 are shown to be the same as each other, the patterns may be different. At least one of the first and second patterns, in one example, is a black and white checkerboard pattern that provides good contrast.

[0053] The pattern 84 is detected by the other of the first and second interior cameras 28a, 28b (step 206), which are mounted on the opposite side of the cab 22. The other of the first and second interior cameras 28a, 28b is calibrated with the pattern to establish a distance between the other of the first and second interior cameras 28a, 28b and the pattern 84 (step 208).

[0054] Also, while particular component arrangements are disclosed in the illustrated embodiments, it should be understood that other arrangements would benefit from the present disclosure. Although a particular sequence of steps has been shown, described and claimed, it should be understood that, unless otherwise indicated, the steps may be performed in any order, separated or combined and still benefit from the present invention.

[0055] Although the different examples have specific components shown in the figures, embodiments of the invention are not limited to those specific combinations. Some components or features of one example may be used in combination with features or components of another example.

[0056] Although exemplary embodiments have been disclosed, a person of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. First and second exterior cameras configured to provide captured images of an exterior of the vehicle corresponding to first and second views; first and second displays in a vehicle cab configured to depict the captured images from the first and second exterior cameras, respectively; first and second in-vehicle cameras configured to capture driver images within the vehicle cab, the first and second in-vehicle cameras having first and second in-vehicle fields of view that overlap one another in a vehicle driver area within the vehicle cab, the first display being within the second in-vehicle field of view and the second display being within the first in-vehicle field of view; a controller in communication with the first and second displays and the first and second interior cameras, the controller configured to monitor driver activity in the vehicle driver area using the first and second interior cameras, the controller configured to initiate a calibration procedure in response to a calibration event, the calibration procedure including calibrating the first and second interior cameras with respect to the vehicle driver area such that the first and second displays show first and second patterns, respectively, and the first and second interior cameras detect the second and first patterns, respectively, to establish distances between the first and second interior cameras and the second and first patterns, respectively; A camera mirror system for a vehicle.

2. The system of claim 1 , wherein the first and second views include a class II view and a class IV view.

3. 3. The system of claim 2, wherein the first and second displays are disposed on A-pillars on left and right sides, respectively, of the vehicle cabin.

4. 2. The system of claim 1, wherein the first and second interior cameras are provided on the first and second displays, respectively, and configured to be movable with the first and second displays, respectively, and the first and second interior cameras have adjustable operating positions relative to one another.

5. The system of claim 1 , wherein the controller is configured to determine a distance from each of the first and second in-vehicle cameras to a driver characteristic to evaluate the driver activity.

6. The system of claim 5 , wherein the driver activity includes at least one of driver gaze and driver posture.

7. The system of claim 1 , wherein the calibration event corresponds to at least one of the first and second interior cameras deviating from a nominal operating position.

8. The system of claim 7 , wherein the calibration event corresponds to at least one of the first and second displays deviating from the nominal operating position.

9. The calibration event is configured to occur within a calibration window; the calibration window is during a vehicle start-up procedure; or The system of claim 1 , wherein the calibration window is less than a threshold vehicle speed.

10. The system of claim 1 , wherein at least one of the first and second patterns comprises a plurality of lines.

11. The system of claim 10 , wherein at least one of the first and second patterns is a black and white checkerboard.

12. 1. A method of calibrating at least one of first and second interior cameras used to observe a driver within a vehicle cab, comprising: The method is configured for use with a camera mirror system having first and second exterior cameras configured to capture images of an exterior of the vehicle corresponding to first and second views, and first and second displays in a vehicle cab configured to depict images captured from the first and second exterior cameras, respectively, the method comprising: providing first and second in-vehicle cameras movable between operational positions; detecting a calibration event including at least one of the first and second interior cameras being moved between the operating positions; displaying a pattern on at least one of the first and second displays after detecting the calibration event; detecting a pattern with the first and second in-vehicle cameras mounted thereon; calibrating at least one of the first and second in-vehicle cameras with the pattern to establish a distance between the first and second in-vehicle cameras and the pattern; A method comprising:

13. The method of claim 12, wherein the first in-vehicle camera has the second display within its field of view, and the second in-vehicle camera has the first display within its field of view.

14. The method of claim 13 , wherein the first and second in-vehicle cameras are disposed on the first and second displays, respectively, and configured to be movable with the first and second displays, respectively.

15. The method described in claim 13, wherein it is possible to determine a distance to a driver characteristic based on the distance in order to evaluate driver activity.

16. The method of claim 15 , wherein the driver activity includes at least one of driver gaze and driver posture.

17. The method of claim 12 , wherein the calibration event occurs during at least one of a vehicle start-up procedure and below a threshold vehicle speed.

18. 14. The method of claim 13, wherein the displaying step includes displaying a pattern on the first and second displays, the detecting step includes detecting each of the patterns by the other of the first and second in-vehicle cameras mounted relative to the other of at least one of the first and second displays, and the calibrating step includes calibrating the other of the first and second in-vehicle cameras with the respective patterns to establish a distance between the other of the first and second in-vehicle cameras and the respective patterns.

Images