Camera monitoring system including image-based impaired driving analysis

Abstract

A method for detecting unfocused vehicle operation includes using a camera monitoring system (CMS) controller to identify a degree of centering of a vehicle including the CMS controller within a first lane line and a second lane line. The determination is done by analyzing at least one image generated by the CMS, and in response to identifying that an unattended driving event has occurred according to a predefined characteristic that the vehicle is centered off the lane center.

Description

Technical Field

[0001] The present disclosure relates to a camera monitoring system (CMS) for a vehicle, and more particularly to a CMS module configured to identify and respond to impaired driving using computer-based analysis of images generated by the CMS. Background Art

[0002] Mirror replacement systems and camera systems for supplementing the field of view of mirrors are used in vehicles to enhance the ability of a vehicle operator to see the surrounding environment. A camera monitoring system (CMS) includes one or more camera systems that provide a field of view to the vehicle operator, including a front field of view, a side field of view, and a rear field of view. In some examples, the camera system covers a larger field of view than a conventional mirror, or includes views that cannot be fully obtained via a conventional mirror and can be used as a mirror supplement and / or mirror replacement. In other examples, the CMS can use computer algorithms and processors to generate stitched, manipulated, and / or extrapolated views that can provide a vehicle operator with a large amount of additional information.

[0003] In addition to mirror replacement, images provided by cameras in the CMS can be used to detect aspects of the environment and aspects of the vehicle, and can be used in conjunction with image processing-based detection processes that can achieve various safety, convenience, and operational efficiency benefits. Summary of the Invention

[0004] An exemplary method for detecting inattentive vehicle operation includes: using a camera monitoring system (CMS) controller to identify the degree of centering of a vehicle including the CMS controller within a first lane line and a second lane line by analyzing at least one image generated by the CMS, and identifying that an inattentive driving event has occurred in response to the vehicle centering deviating from the lane center according to predefined characteristics.

[0005] In another example of the above method for detecting inattentive vehicle operation, the predefined characteristics include at least one of instantaneous vehicle centering and vehicle centering varying over time.

[0006] In another example of any of the above methods for detecting inattentive vehicle operation, the predefined characteristic is vehicle centering varying over time, and wherein the deviation is the difference between the curvature of the line defined by the vehicle centering varying over time and the curvature of the lane defined by the first lane line and the second lane line.

[0007] In another example of any of the above methods for detecting inattentive vehicle operation, the curvature of the lane is determined at least in part by determining the geospatial location of the vehicle determined by the CMS controller and comparing the geospatial location with map data.

[0008] In another example of any of the above methods for detecting inattentive vehicle operation, lane curvature is determined using image analysis of at least one image generated by the CMS, at least in part.

[0009] In another example of any of the above methods for detecting inattentive vehicle operation, lane curvature is determined using only image analysis of at least one image generated by the CMS.

[0010] Another example of any of the above methods for detecting inattentive vehicle operation further includes activating an inattentive driving response in response to identifying an inattentive driving event.

[0011] In another example of any of the above methods for detecting inattentive vehicle operation, the inattentive driving response includes any combination of one or more of the following: storing driving parameters for subsequent upload to a remote fleet monitoring system, inadvertently uploading driving parameters to a remote fleet monitoring system, activating an audio alert for the vehicle operator, and activating a visual alert for the vehicle operator.

[0012] In another example of any of the above methods for detecting inattentive vehicle operation, the driving parameters include at least two of speed, steering angle, gear, and engine RPM, and the at least one image generated by the CMS.

[0013] In another example of any of the above methods for detecting inattentive vehicle operation, the at least one image generated by the CMS includes at least one of a rearward image generated by a rearward camera mounted on a trailer and an image generated by a class II / IV mirror.

[0014] In one exemplary embodiment, a camera monitoring system (CMS) for a vehicle includes: a plurality of outward-facing cameras disposed around the vehicle, each camera defining a field of view configured to include at least one lane line, the at least one lane line defining a lane in which the vehicle is operating as the vehicle moves; and a CMS controller configured to receive images from each of the cameras of the plurality of outward-facing cameras and including a non-transitory memory and a processor, the non-transitory memory configured to cause the controller to identify the degree of centering of the vehicle including the CMS controller within a first lane line and a second lane line by analyzing at least one image generated by the CMS, and to identify that an inattentive driving event has occurred in response to the vehicle centering deviating from the lane center according to a predefined characteristic.

[0015] In another example of the above CMS for a vehicle, the predefined characteristic includes at least one of instantaneous vehicle centering and vehicle centering over time.

[0016] In another example of any of the above-described CMSs for a vehicle, the predefined characteristic is a time-varying vehicle centering, and wherein the deviation is the difference between the curvature of a line defined by the time-varying vehicle centering and the curvature of a lane defined by the first lane line and the second lane line.

[0017] In another example of any of the above-described CMSs for a vehicle, the memory further stores instructions for causing the CMS to activate a distracted driving response in response to identifying a distracted driving event.

[0018] In another example of any of the above-described CMSs for a vehicle, the distracted driving response includes any combination of one or more of the following: storing driving parameters for subsequent upload to a remote fleet monitoring system, inadvertently uploading driving parameters to a remote fleet monitoring system, activating an audio alert for a vehicle operator, and activating a visual alert for a vehicle operator.

[0019] These and other features of the present invention can be best understood from the following specification and drawings, which are briefly described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present disclosure can be further understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0021] Figure 1A is a schematic front view of a commercial truck having a camera monitoring system (CMS) for providing at least Class II and Class IV views.

[0022] Figure 1B is a schematic top view of a commercial truck having a camera monitoring system providing Class II, Class IV, Class V, and Class VI views.

[0023] Figure 2 is a schematic top perspective view of a cab including a display and an interior camera.

[0024] Figure 3 Schematically shows views generated by a forward camera within the CMS of Figure 1A and Figure 1B

[0025] Figure 4 Schematically shows views generated by a forward camera within the CMS of Figure 1A and Figure 1B

[0026] Figure 5 Schematically shows the operation of an image-based impaired driving detection module within the CMS.

[0027] Figure 6 shows a high-level example process for using a CMS to identify impaired driving events and trigger a response.

[0028] Figure 7 shows a high-level method for responding to distracted driving events detected using Figure 5 the method.

[0029] Any of the foregoing paragraphs, claims, or examples, examples, and alternatives of the following description and drawings, including any of their aspects or corresponding individual features, may be taken independently or in any combination. Features described in connection with one embodiment apply to all embodiments unless the features are incompatible. Detailed Description

[0030] A schematic diagram of a commercial vehicle 10 is shown in Figure 1A and Figure 1B is shown. Figure 2 is a schematic top perspective view of the passenger compartment of vehicle 10 including a display and an interior camera. Vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that vehicle cab 12 and / or trailer 14 can be of any configuration. Although a commercial truck is envisioned in the present disclosure, the present invention can also be applied to other types of vehicles. Vehicle 10 includes a camera monitoring system (CMS) 15 ( Figure 2 ), which has a driver camera arm 16a and a co-driver side camera arm 16b mounted to the outside of vehicle cab 12. If desired, camera arms 16a, 16b may also include conventional side mirrors integrated therewith, but CMS 15 can be used to completely replace the side mirrors. In additional examples, each side may include multiple camera arms, each arm housing one or more cameras and / or side mirrors.

[0031] Each of camera arms 16a, 16b includes a base fixed to, for example, cab 12. A pivot arm is supported by the base and can be hinged relative to the base. At least one rearward camera 20a, 20b is respectively disposed within the camera arm. External cameras 20a, 20b respectively provide external fields of view FOV EX1 , FOV EX2 , each field of view including at least one of a Class II view and a Class IV view ( Figure 1B) These views are views required by law in the commercial trucking industry. If desired, multiple cameras can also be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. Any reference to "class" views is not intended to be restrictive, but rather as an example of the type of view provided by a particular camera to the display. Each arm 16a, 16b can also provide a housing that encloses the electronics configured to provide CMS 15.

[0032] The first video display 18a and the second video display 18b are arranged on or near the A-pillars 19a, 19b on each of the driver's side and the passenger's side within the vehicle cab 12 to display Class II and Class IV views on each side of the vehicle 10, which provide rearward side views of the vehicle 10 captured by the external cameras 20a, 20b.

[0033] If video of Class V and / or Class VI views is also desired, the camera housing 16c and the forward camera 20c can be arranged at or near the front of the vehicle 10 to provide those views ( Figure 1B ). The forward camera 20c is configured with a field of view that includes Class V and Class VI regions and extends beyond the Class V and Class VI regions towards the horizon. A third display 18c arranged within the cab 12 near the center of the top of the windshield can be used to display Class V and Class VI views towards the front of the vehicle 10 to the driver. The displays 18a, 18b, 18c face the driver area 24 within the cab 22, where the operator sits on the driver's seat 26. The position, size, and field of the view streamed to any particular display can be different from the configuration described in this disclosure and still incorporate the disclosed invention.

[0034] If video of Class VIII views is desired, additional camera housings can be provided on the sides and rear of vehicle 10 to provide additional fields of view that include some or all of the Class VIII regions of vehicle 10. As shown, the Class VIII views include views adjacent to trailer 14 and near the rear of vehicle 10 (including the rear of trailer 14). In one example, a view of the rear proximity of vehicle 10 is generated by a rear-facing camera disposed at the rear of vehicle 10 and can include an adjacent rear proximity and a conventional rear view (e.g., a view extending rearward to the horizon, such as a view that can be generated by a rearview mirror in a vehicle without trailer 14). In such an example, the third display 18c can include one or more frames that display the Class VIII views. Alternatively, additional displays can be added near the first display 18a, second display 18b, and third display 18c or at alternative locations within the cab of vehicle 10, and a display dedicated to providing the Class VIII views can be provided.

[0035] Continuing reference Figures 1A to 2 , Figure 3 schematically shows an exemplary forward view 200, such as can be generated by forward camera 20a or by any similarly positioned camera. Included within view 200 is the road 210 on which vehicle 10 is traveling. Road 210 includes lane lines 212, 214, where the outer lane line 212 is defined by a solid line and the inner lane line 214 is defined by dashed segments. The lanes defined by lane lines 212, 214 are separated by a standardized width, where the width is consistent along the length of road 210. In most cases, the width of road 210 and the lanes on road 210 are standardized according to local regulations and will be consistent on multiple different roads within a given location. Rearward side cameras 20a, 20b generate a rearward view 300 Figure 3 simultaneously with the forward view 200 of Figure 4 , including trailer 12 and road 210, which are shown in Figure 4 , where like numerals indicate like elements. Although

[0036] shows a view from the driver side of vehicle 10, a similar view is generated by the co-pilot side camera 18b.

[0036] In yet another example vehicle, additional cameras can be included within CMS 15 and provide similar images that include all or part of road 210. The views explicitly shown and described are exemplary, and any combination of views that include lane lines 212, 214 can be used in the processes described herein.

[0037] Continuing reference Figures 1A to 4 , Figure 5Schematically shows an example operation of a vehicle 10 equipped with the CMS15 described herein. The CMS15 includes image analysis software that is configured to identify the positioning of lane lines 212, 214 within various images received from cameras 20a, 20b, 20c and other cameras in the CMS15 using edge detection and similar processes. The positioning of lane lines 212, 214 relative to the vehicle 10 can be tracked over time by the CMS15, thereby determining the degree of centering of the vehicle 10 within the corresponding lanes 210a, 210b and identifying the path (driving line 410) followed by the vehicle 10. This process is referred to as image-based centering. Lane centering determines how far the center of the vehicle 10 is from each lane line 212, 214, and this distance can be used in a variety of ways by the CMS15 and other vehicle systems such as driver assistance systems.

[0038] The CMS15 also includes an impaired driving detection and analysis module 17. The impaired driving detection and analysis module 17 is a software module within the controller of the CMS15. The software module 17 is configured to analyze the vehicle path (driving line 410) that varies over time determined by image-based lane centering to identify and respond to indications of impaired driving.

[0039] During normal operation of the vehicle 10, the impaired driving detection and analysis module 17 uses edge detection image analysis on images 200, 300 received from cameras within the CMS15 to identify lane line edges and determine the position (centering) of the vehicle relative to each lane. The detected positioning of each lane line 212, 214 within the image is converted to a three-dimensional position relative to the vehicle 10. The three-dimensional position is then used to determine the degree of centering (e.g., the degree of proximity where the longitudinal center of the vehicle 10 is equal to the distance 412, 414 between the inner edges of the lane lines 212, 214).

[0040] Then, the impaired driving detection and analysis module 17 compares the degree of centering of the vehicle 10 with a predetermined criterion. When the degree of centering does not meet the predetermined criterion, the module 17 detects that an impaired driving event is occurring or has occurred.

[0041] In one example, the criterion can be instantaneous centering. In this example, when the vehicle 10 deviates from the center by more than a predetermined amount (e.g., the inequality between the distances 412, 414 exceeds a predetermined distance), the driver is considered inattentive. As an example, in one example, the predetermined distance is two feet. In a practical example, the deviation distance depends on the lane width, and a calibration table can be used in conjunction with a lookup table that correlates road types with lane widths so that an appropriate distance can be selected. In yet another example, this value determined by looking up the lookup table can be adjusted based on a factor related to the actual lane width determined by the CMS15 using image analysis.

[0042] Mathematically, it is possible to calculate when the deviation from the center exceeds the margin by a predetermined distance, and the CMS markings increase the count of potentially impaired driving. Once the counter reaches a certain value (e.g., after a certain number of collective events within a given time frame), a record of impaired driving is triggered. In one specific implementation, the impaired driving counter decrements over time at a predefined rate to compensate for unexpected events that might increment the possible impaired driving counter erroneously.

[0043] In one specific example, the United States Federal Highway Administration recommends a lane width of 11 to 12 feet (3.35 - 3.66 meters). The trailer width is approximately 8 feet 4 inches (2.54 meters), so in the United States, a deviation greater than 2 feet 4 inches (0.71 meters) can be used as a trigger for a highway. Additionally, the ratio between the lane width, the trailer width, and the deviation can be used as a calibration parameter for roads with different lane widths and / or trailers other than the standard trailer width.

[0044] In an alternative example, when the vehicle centering (driving line 410) over time does not match or approximately match the curvature of the lane lines 212, 214 over time, it is determined that the driver is inattentive. As used herein, the curvature of a lane line refers to the path of the lane line and can include extended straight portions where minimal geometric curvature (if any) occurs. In one example, the curvature of the lane lines 212, 214 is determined using the same image analysis process that the CMS 15 uses to identify the degree of vehicle 10 centering. As used herein, "approximately match" means that the driving line 410 has a profile that includes a minimal deviation from the lane lines 212, 214, as expected of an attentive driver. The same or similar predefined distance and / or adjusted predefined distance methods described above regarding instantaneous centering can be used to determine the deviation from the lane lines 212, 214.

[0045] In yet another alternative example, inattentive driving is detected when the vehicle 10 deviates from the center by more than a predetermined amount or when the vehicle centering over time does not approximately match the curvature of the lane lines.

[0046] In some examples, the CMS 15 also includes a GPS and / or geospatial positioning system 21 configured to identify the geospatial location of the vehicle 10. In these examples, the geospatial location of the vehicle 10 is used to identify the positioning of the vehicle 10 relative to a set of map data, and the set of map data is used to identify the expected profile of the roads 210a, 210b. Then the expected profile of the road is used to verify the accuracy of the image - based lane line profile or as an alternative to the image - based lane line profile. In this way, the positioning system 21 complements and enhances the image - based analysis.

[0047] Continuing to refer to FIGS. 1 to Figure 5 , Figure 6Method 500 for performing the above general procedure is shown. In step 510 of "Detecting Vehicle Centering over Time", CMS15 initially detects vehicle centering and tracks the centering over time. As described above, this centering determines the driving line 410, and in step 520 of "Comparing the Centering with a Predetermined Criterion", the driving line 410 is compared with an instantaneous centering threshold, a lane profile, or both. When the predetermined criterion is met, the impaired driving detection and analysis module 17 activates one or more inattentive driver responses in step 530 of "Activating an Inattentive Response".

[0048] In one example, the inattentive response is an auditory and / or visual alert to the vehicle operator. The alert prompts the driver to pay attention and actively attempt to remedy the inattentive driving. In one example, the alert can include a flashing warning on a display screen accompanied by a jarring audio output.

[0049] In another example, the inattentive response is the activation of a fleet monitoring system that monitors data regarding inattentive operation and stores the monitored data for subsequent upload to a fleet monitoring system or any similar system or database to monitor vehicle operation. In a variant of this example, CMS15 includes remote networking hardware and uploads the monitored data when inattentive driving occurs, and / or instead of locally saving the data for subsequent upload.

[0050] The stored and / or uploaded data includes vehicle parameters (speed, steering angle, gear position, engine RPM, geospatial positioning of the vehicle, etc.) obtained by the CMS from the general vehicle controller via a CAN bus or similar system, image data from a camera within CMS15, and any information derived from either or both of the vehicle parameters and the image data (e.g., percentage deviation from the centerline, road conditions, lane width, etc.). In the case where local data storage is for later upload, the storage can occur only when inattentive driving detection is triggered, or periodically depending on the amount of memory on the vehicle.

[0051] Once uploaded, the fleet management system utilizes the inattentive driving data to continuously monitor the driver statistics of a given driver, monitor the driving of the entire fleet, provide data for potential event reports, or any similar use.

[0052] Continuing to refer to FIGS. 1 to Figure 6 , Figure 7Method 600 for responding to inattentive driver alerts is schematically shown. Initially, when CMS 15 indicates an inattentive driver, in accordance with method 500 described above, CMS 15 activates a driver monitoring subroutine in "Activate Driver Monitoring" step 610 and activates a driver alert system within CMS 15 in "Activate Driver Alert" step 620, substantially simultaneously. As used herein, "substantially simultaneously" means that steps 610 and 620 occur independently of each other and at similar time instances. It should be understood that in an actual embodiment, a single controller may not be able to achieve precise simultaneous operation.

[0053] Once activated, in "Begin Data Storage" step 630, the driver monitoring step 610 determines data related to inattentive driving input to CMS 15 (e.g., speed, steering angle, gear position, engine RPM, geospatial positioning of the vehicle), image data for initial detection, continuous image data received from cameras (e.g., Class IV images on both sides), vehicle operation data from the general vehicle controller during inattentive operation, etc., which are collected by CMS 15 and stored in the non-transitory memory of CMS 15.

[0054] Data storage continues until the latter of after the vehicle operation resumes attentive operation or a predetermined period (e.g., 5 minutes) after a minimum time period. Thereafter, CMS 15 ends the driver monitoring process in "End Driver Monitoring" step 640. After ending data collection, the stored data is uploaded to the fleet monitoring system in "Upload Data to Fleet System" step 650.

[0055] In some examples, in the case where vehicle 10 has a wireless data connection capable of connecting to a central data repository, the data can be uploaded simultaneously with collection and storage until the inattentive driving event stops.

[0056] By implementing the above CMS features, vehicle operation can be monitored, and inattentive operations can be marked and tracked for fleet owners, insurance purposes, and / or driver feedback. Additionally, in some examples, using the monitoring system described herein allows monitoring to be achieved without directly monitoring (e.g., video recording) the vehicle operator.

[0057] Although example embodiments have been disclosed, those of ordinary skill in the art will recognize that certain modifications will fall within the scope of the claims. For this reason, the following claims should be studied to determine their true scope and content.

Claims

1. A method for detecting inattentive vehicle operation, comprising: using a camera monitoring system (CMS) controller to identify the degree of centering of a vehicle including the CMS controller within a first lane line and a second lane line by analyzing at least one image generated by the CMS; and identifying that an inattentive driving event has occurred in response to the vehicle centering deviating from the lane center according to predefined characteristics.

2. The method according to claim 1, wherein The predefined characteristics include at least one of instantaneous vehicle centering and vehicle centering varying over time.

3. The method according to claim 2, wherein, The predefined characteristic is vehicle centering varying over time, and wherein the deviation is the difference between the curvature of a line defined by the vehicle centering varying over time and the curvature of a lane defined by the first lane line and the second lane line.

4. The method according to claim 3, wherein Determining the curvature of the lane is at least partially accomplished by determining the geospatial positioning of the vehicle determined by the CMS controller and comparing the geospatial positioning with map data.

5. The method according to claim 3, wherein, Determining the curvature of the lane is at least partially accomplished using image analysis of the at least one image generated by the CMS.

6. The method according to claim 5, wherein Determining the curvature of the lane is accomplished only using image analysis of the at least one image generated by the CMS.

7. The method according to claim 1, further comprising activating an inattentive driving response in response to identifying the inattentive driving event.

8. The method according to claim 7, wherein The inattentive driving response includes any combination of one or more of the following: storing driving parameters for subsequent upload to a remote fleet monitoring system, inadvertently uploading driving parameters to the remote fleet monitoring system, activating an audio alert for a vehicle operator, and activating a visual alert for the vehicle operator.

9. The method according to claim 8, wherein, The driving parameters include: the at least one image generated by the CMS, and at least two of speed, steering angle, gear, and engine RPM.

10. The method according to claim 1, wherein, The at least one image generated by the CMS includes at least one of a rearward image generated by a rearward camera mounted on a trailer and an image generated by a class II / IV mirror.

11. A camera monitoring system (CMS) for a vehicle, comprising: a plurality of outward-facing cameras disposed around the vehicle, each camera defining a field of view configured to include at least one lane line, the at least one lane line defining a lane in which the vehicle is operating when the vehicle is moving; and a CMS controller configured to receive images from each of the plurality of outward-facing cameras and including a non-transitory memory and a processor, the non-transitory memory configured to cause the controller to identify the degree of centering of a vehicle including the CMS controller within a first lane line and a second lane line by analyzing at least one image generated by the CMS, and to identify that an inattentive driving event has occurred in response to the vehicle centering deviating from the lane center according to predefined characteristics.

12. The CMS according to claim 11, wherein, The predefined characteristics include at least one of instantaneous vehicle centering and vehicle centering varying over time.

13. The CMS according to claim 11, wherein, The predefined characteristic is a vehicle centering that varies over time, and wherein the deviation is the difference between the curvature of a line defined by the vehicle centering that varies over time and the curvature of a lane defined by the first lane line and the second lane line.

14. The CMS according to claim 11, wherein, The memory also stores instructions for causing the CMS to activate a distracted driving response in response to identifying the distracted driving event.

15. The CMS according to claim 14, wherein The distracted driving response includes any combination of one or more of the following: storing driving parameters for subsequent upload to a remote fleet monitoring system, inadvertently uploading driving parameters to the remote fleet monitoring system, activating an audio alert for a vehicle operator, and activating a visual alert for the vehicle operator.

Images