Camera monitoring system including image-based driving impairment analysis

Abstract

A method for detecting distracted driving includes using a camera monitoring system (CMS) controller to identify how centered a vehicle including a CMS controller is within a first lane line and a second lane line by analyzing at least one image generated by the camera monitoring system and identifying that a distracted driving event has occurred in response to the vehicle centering deviating from the lane center according to a predetermined characteristic.

Description

[Technical Field]

[0001] The present disclosure relates to camera surveillance systems (CMS) for vehicles, and in particular to CMS modules configured to identify and respond to impaired driving using computer-based analysis of images generated by the CMS. [Background technology]

[0002] Mirror replacement systems and camera systems that supplement mirror views are utilized in vehicles to enhance the vehicle operator's ability to view the surrounding environment. A camera surveillance system (CMS) includes one or more camera systems that provide the vehicle operator with a field of view that includes forward, side, and rearward views. In some examples, the camera system covers a wider field of view than a traditional mirror or includes views not fully available through a traditional mirror and can be used as a mirror supplement and / or mirror replacement. In other examples, a CMS may use computer algorithms and processors to generate stitched, manipulated, and / or extrapolated views that can provide substantial additional information to the vehicle operator.

[0003] In addition to mirror replacement, images provided via cameras within the CMS can be utilized to detect aspects of the environment and aspects of the vehicle, and can be used in conjunction with image processing-based detection processes that can achieve various safety, convenience, and operational efficiency benefits. Summary of the Invention

[0004] An exemplary method for detecting distracted vehicle operation includes using a camera monitoring system (CMS) controller to identify how centered a vehicle including the CMS controller is within a first lane line and a second lane line by analyzing at least one image generated by the CMS, and identifying that a distracted driving event has occurred in response to the centering of the vehicle deviating from the lane center according to predetermined characteristics.

[0005] In another example of the above method for detecting inadvertent vehicle operation, the predetermined characteristic includes at least one of instantaneous vehicle centering and vehicle centering over time.

[0006] In another example of any of the above methods for detecting inadvertent vehicle operation, the predetermined characteristic is vehicle centering over time, and the deviation is a difference between a curvature of a line defined by the vehicle centering over time and a curvature of a lane defined by the first lane line and the second lane line.

[0007] In another example of any of the above methods for detecting inadvertent vehicle operation, the lane curvature is determined, at least in part, by determining a geospatial position of the vehicle as determined by the CMS controller and comparing the geospatial position to map data.

[0008] In another example of any of the above methods for detecting inadvertent vehicle operation, the lane curvature is determined, at least in part, using image analysis of the at least one image generated by the CMS.

[0009] In another example of any of the above methods for detecting inadvertent vehicle operation, the lane curvature is determined using only image analysis of the at least one image generated by the CMS.

[0010] Another example of any of the above-described methods for detecting distracted vehicle operation further includes initiating a distracted driving response in response to identifying the distracted driving event.

[0011] In another example of any of the above methods for detecting careless vehicle operation, the careless driving response includes any combination of one or more of storing driving parameters for subsequent upload to a remote fleet monitoring system, carelessly uploading driving parameters to the remote fleet monitoring system, activating an audio alert to a vehicle operator, and activating a visual alert to the vehicle operator.

[0012] In another example of any of the above methods for detecting inattentive vehicle operation, the driving parameters include at least two of speed, steering angle, gear, and engine RPM, and the at least one image generated by the CMS.

[0013] In another example of any of the above methods for detecting inattentive vehicle operation, the at least one image generated by the CMS includes at least one of a rear-facing image generated by a trailer-mounted rear-facing camera and an image generated by a Class II / IV mirror.

[0014] In one exemplary embodiment, a camera surveillance system (CMS) for a vehicle includes a plurality of externally facing cameras positioned around the vehicle, each camera defining a field of view configured to include at least one lane line defining a lane in which the vehicle is traveling while the vehicle is moving; and a CMS controller configured to receive images from each camera of the plurality of externally facing cameras and including a non-transitory memory and a processor, the non-transitory memory configured to analyze at least one image generated by the CMS to cause the controller to identify how centered a vehicle including the CMS controller is within a first lane line and a second lane line, and to identify that a distracted driving event has occurred in response to vehicle centering deviating from a lane center according to a predetermined characteristic.

[0015] In another example of the above CMS for a vehicle, the predetermined characteristic includes at least one of instantaneous vehicle centering and vehicle centering over time.

[0016] In another example of any of the above-described CMS for a vehicle, the predetermined characteristic is the vehicle centering over time; The deviation is the difference between the curvature of a line defined by the vehicle centering over time and the curvature of a lane defined by the first lane line and the second lane line.

[0017] In another example of any of the above-described CMS for a vehicle, the memory further stores instructions for causing the CMS to initiate a distracted driving response in response to identifying the distracted driving event.

[0018] In another example of any of the above-described CMS for a vehicle, the distracted driving response includes any combination of one or more of storing driving parameters for subsequent upload to a remote fleet monitoring system, inadvertently uploading driving parameters to the remote fleet monitoring system, activating an audio alert to a vehicle operator, and activating a visual alert to the vehicle operator.

[0019] These and other features of the present invention can be best understood from the following specification and drawings. [Brief explanation of the drawings]

[0020] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0021] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera monitor system (CMS) used to provide at least Class II and Class IV views.

[0022] [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera monitoring system providing Class II, Class IV, Class V, and Class VI views.

[0023] [Figure 2] FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an interior camera.

[0024] [Figure 3] 1C illustrates a schematic view generated by a forward-facing camera in the CMS of FIGS. 1A and 1B.

[0025] [Figure 4] 1C schematically illustrates an example of a posterior-lateral view generated by the CMS of FIGS. 1A and 1B.

[0026] [Figure 5] 1 illustrates the operation of an image-based driving impairment detection module within a CMS.

[0027] [Figure 6] An example of a high-level process for using a CMS to identify an impaired driving event and trigger a response is provided.

[0028] [Figure 7] 6 illustrates a high-level method for responding to a distracted driving event detected using the method of FIG. 5.

[0029] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE INVENTION

[0030] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. FIG. 2 is a schematic top perspective view of the cab of the vehicle 10, including a display and an interior camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be of any configuration. While commercial trucks are contemplated in this disclosure, the present invention is applicable to other types of vehicles. The vehicle 10 incorporates a camera monitoring system (CMS) 15 (FIG. 2) that includes driver and passenger side camera arms 16a, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, each arm housing one or more cameras and / or mirrors.

[0031] Each camera arm 16a, 16b includes a base fixed to, for example, the cab 12. A pivoting arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20a, 20b is disposed within each camera arm. Each exterior camera 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1b), which are legally defined views in the commercial trucking industry. EX1 , FOV EX2 16a, 16b. If desired, multiple cameras may be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in the European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. References to "class" views are not intended to be limiting, but rather as an illustration of the type of view provided to the display by a particular camera. Each arm 16a, 16b may also provide a housing enclosing electronics configured to provide various features of the CMS 15.

[0032] First and second video displays 18a, 18b are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19a, 19b and display Class II and Class IV views on each side of the vehicle 10, which provide rear-facing views along the vehicle 10 captured by exterior cameras 20a, 20b.

[0033] If Class V and / or Class VI view footage is also required, a camera housing 16c and forward-facing camera 20c may be positioned at or near the front of vehicle 10 to provide these views ( FIG. 1B ). Forward-facing camera 20c is configured to include the Class V and Class VI areas and extend its field of view beyond the Class V and Class VI areas toward the horizon. A third display 18c, positioned within cab 12 near the top center of the windshield, can be used to display Class V and Class VI views forward of vehicle 10 to the driver. Displays 18a, 18b, and 18c face a driver area 24 within cab 22, where the driver is seated in driver's seat 26. The location, size, and field of view(s) streamed to a particular display may vary from the configurations described herein and still encompass the invention of this disclosure.

[0034] If a Class VIII view is required, additional camera housings can be positioned on the sides and rear of the vehicle 10 to provide additional fields of view that include some or all of the vehicle's Class VIII zone. As shown, the Class VIII view includes a view surrounding the immediate vicinity of the trailer 14 and a rear close-up view of the vehicle 10 that includes the area behind the trailer 14. In one example, the rear close-up view of the vehicle 10 is generated by a rear-facing camera positioned at the rear of the vehicle 10 and may include both an immediate rear close-up view and a traditional rear view (e.g., a view extending rearward to the horizon provided by a vehicle's rearview mirror without the trailer 14). In such an example, the third display 18c may include one or more frames displaying the Class VIII view. Alternatively, additional displays may be added near the first, second, and third displays 18a, 18b, and 18c, or at other locations within the vehicle's 10 cab, to provide a dedicated display providing the Class VIII view.

[0035] 1A-2, FIG. 3 schematically illustrates an exemplary forward-facing view 200, such as may be generated by forward-facing camera 20a or any similarly positioned camera. View 200 includes road 210 along which vehicle 10 is traveling. Road 210 includes lane lines 212, 214, with outer lane line 212 defined by a solid line and inner lane line 214 defined by a dashed line segment. The lanes defined by lane lines 212, 214 are spaced apart by a standardized width, which is constant along the length of road 210. In most cases, the width of road 210 and the lanes thereon are standardized by local regulations and are consistent across multiple different roads within a given region. Concurrently with forward-facing view 200 of FIG. 3, rear-facing side cameras 20a, 20b generate a rear-facing view 300, including trailer 12 and road 210, which is shown in FIG. 4, where like numerals indicate like elements. FIG. 4 shows a view from the driver's side of the vehicle 10, although a similar view is produced by the passenger side camera 18b.

[0036] In yet another embodiment of the vehicle, additional cameras may be included within CMS 15 to provide similar images that include all or part of road 210. The views explicitly shown and described are exemplary, and any combination of views that include lane lines 212, 214 may be used in the processes described herein.

[0037] 1A-4, FIG. 5 schematically illustrates an example of the operation of a vehicle 10 equipped with a CMS 15 described herein. The CMS 15 includes image analysis software configured to use edge detection and similar processes to identify the location of lane lines 212, 214 in various images received from cameras 20a, 20b, 20c, and other cameras within the CMS 15. The positions of lane lines 212, 214 relative to the vehicle 10 can be tracked over time by the CMS 15 to determine how centered the vehicle 10 is within the corresponding lane 210a, 210b and identify the path (driving line 410) the vehicle 10 will follow. This process is referred to as image-based centering. Lane centering determines how far the center of the vehicle 10 is from each lane line 212, 214, and this distance can be used in multiple ways by the CMS 15 and other vehicle systems, such as driver assistance systems.

[0038] CMS 15 further includes impaired driving detection and analysis module 17. Impaired driving detection and analysis module 17 is a software module within the controller of CMS 15. Software module 17 is configured to analyze the vehicle path over time (driving line 410) determined by image-based lane centering, identify indications of impaired driving, and respond.

[0039] During typical operation of the vehicle 10, the driving impairment detection and analysis module 17 uses edge detection image analysis on images 200, 300 received from the cameras in the CMS 15 to identify the edges of lane lines and determine the vehicle's position relative to each lane (centering). The detected position of each lane line 212, 214 in the image is converted to a three-dimensional position relative to the vehicle 10. The three-dimensional position is then used to determine how centered the vehicle is (e.g., how equal the distances 412, 414 between the longitudinal center of the vehicle 10 and the inside edges of the lane lines 212, 214 are).

[0040] The driving impairment detection and analysis module 17 then compares how centered the vehicle 10 is to predetermined criteria. If the centering does not meet the predetermined criteria, the module 17 detects that a driving impairment event is occurring or has occurred.

[0041] In one example, the criterion may be instantaneous centering. In this example, if the vehicle 10 is off-center by more than a predetermined amount (e.g., the difference between distances 412 and 414 exceeds a predetermined distance), the driver is deemed inattentive. For example, the predetermined distance in one example is 2 feet. In a practical example, the deviation distance depends on the lane width, and a calibration table with a lookup table correlating road type with lane width may be used so that an appropriate distance can be selected. In yet another example, this value determined by referencing the lookup table may be adjusted based on a factor related to the actual lane width determined by the CMS 15 using image analysis.

[0042] Whether the deviation from the center exceeds the edge by a predetermined distance can be mathematically calculated, and the CMS mark increments a potential driving impairment counter. When the counter reaches a certain value (e.g., after a certain number of cluster events occur within a predetermined time frame), recording of a driving impairment is triggered. In one particular implementation, the driving impairment counter is decremented over time at a predefined rate to compensate for contingencies that may erroneously increment the potential driving impairment counter.

[0043] In one specific example, a lane width of 11-12 feet (3.35-3.66 meters) is recommended by the U.S. Federal Highway Administration. Because trailer width is approximately 8 feet 4 inches (2.54 meters), a deviation of more than 2 feet 4 inches (0.71 meters) may be used as a trigger for U.S. highways. Furthermore, the ratio between lane width, trailer width, and deviation may be used as a calibration parameter for roads with different lane widths and / or trailers outside the standard trailer width.

[0044] In another example, a driver is determined to be inattentive if the vehicle's centering (driving line 410) over time does not match or approximately match the curvature of the lane lines 212, 214 over time. As used herein, lane line curvature refers to the path of the lane lines, which may include stretches of straight lines, if any, where minimal geometric curvature occurs. The curvature of the lane lines 212, 214, in one example, is determined using the same image analysis process by which the CMS 15 identifies the centering of the vehicle 10. As used herein, "approximately matched" refers to a driving line 410 having a contour that includes minimal deviation from the lane lines 212, 214, as would be expected from an attentive driver. Deviation from the lane lines 212, 214 may be determined using the same or similar predetermined distance and / or adjusted predetermined distance methods as described above with respect to instantaneous centering.

[0045] In yet another alternative, distracted driving is detected when the vehicle 10 is off-center by more than a predetermined amount, or when the vehicle centering over time does not approximately match the curvature of the lane lines.

[0046] In some examples, the CMS 15 further includes a GPS and / or geospatial positioning system 21 configured to identify the geospatial location of the vehicle 10. In these examples, the geospatial location of the vehicle 10 is used to identify the position of the vehicle 10 relative to a set of map data, which is used to identify predicted contours of the roads 210 a, 210 b. The predicted contours of the roads are then used to verify the accuracy of the image-based lane line contours or as a substitute for the image-based lane line contours. In this manner, the positioning system 21 complements and enhances the image-based analysis.

[0047] With continued reference to Figures 1 through 5, Figure 6 illustrates a method 500 for carrying out the general procedure described above. CMS 15 first detects vehicle centering and tracks the centering over time in a "Detect vehicle centering over time" step 510. As described above, this centering determines a driving line 410, which is compared to an instantaneous centering threshold, lane contours, or both in a "Compare centering to predetermined criteria" step 520. If the predetermined criteria are met, driving impairment detection and analysis module 17 activates one or more inattentive driver responses in an "Activate inattentive response" step 530.

[0048] The inattention response, in one example, is an audible and / or visual alert to the vehicle operator. The alert calls the driver's attention in a proactive attempt to correct the inattentive driving. In one example, the alert may include a flashing warning on a display screen accompanied by a harsh audio output.

[0049] In another example, the inattention response is activation of a fleet monitoring system that monitors data related to distracted driving and stores the monitored data for subsequent upload to a fleet monitoring system or any similar system or database for monitoring vehicle operation. In one variation of this example, CMS 15 includes remote network hardware to upload monitored data when distracted driving occurs, and alternatively, stores the data locally for subsequent upload.

[0050] The data stored and / or uploaded may include vehicle parameters (speed, steering angle, gear, engine RPM, vehicle geospatial location, etc.) from the general vehicle controller(s) obtained by the CMS via a CAN bus or similar system, image data from cameras within the CMS 15, and any information derived from either or both of the vehicle parameters and image data (e.g., percent deviation from centerline, road conditions, lane width, etc.). In the case of local data storage for later uploading, storage may occur only when distracted driving detection is triggered or may be stored periodically depending on the amount of memory installed in the vehicle.

[0051] Once uploaded, the fleet management system can utilize the distracted driving data to continuously monitor the driving statistics of a given driver, monitor the driving of the entire fleet, provide data for potential accident reporting, or any similar use.

[0052] 1 through 6, FIG. 7 generally illustrates a method 600 for responding to a distracted driving warning. Initially, when the CMS 15 indicates an inattentive driver, in accordance with method 500 above, the CMS 15 activates a driver monitoring subroutine at an "Activate Driver Monitoring" step 610 and substantially simultaneously activates a driver alert system within the CMS 15 at an "Activate Driver Alert" step 620. As used herein, "substantially simultaneously" refers to steps 610 and 620 occurring independently of one another and at similar times. It is understood that in a practical embodiment, a single controller may not be able to achieve precise simultaneous operation.

[0053] Once initiated, driver monitoring step 610 determines that data coming into CMS 15 related to distracted driving (e.g., speed, steering angle, gear, engine RPM, vehicle geospatial location), image data used to make the initial detection, continuous image data received from cameras (e.g., Class IV images from both sides), distracted vehicle driving data from the general vehicle controller, etc. are being collected by CMS 15 and stored in non-transitory memory of CMS 15 in "begin data storage" step 630.

[0054] Data storage continues until a predetermined period (e.g., 5 minutes) after vehicle operation resumes attentive driving, or a minimum period, whichever is later. CMS 15 then terminates the driver monitoring process in an "End Driver Monitoring" step 640. Following the end of data collection, the stored data is uploaded to the fleet monitoring system in an "Upload Data to Fleet System" step 650.

[0055] In some examples, if the vehicle 10 has a wireless data connection that can connect with a central data repository, data may be collected, stored, and uploaded simultaneously until the distracted driving event has ceased.

[0056] By implementing the above-described CMS functionality, vehicle operation can be monitored and distracted driving can be flagged and tracked for fleet owner, insurance purposes, and / or driver feedback. Further, in some examples, use of the monitoring systems described herein can achieve monitoring without direct vehicle operator oversight (e.g., video recording).

[0057] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. 1. A method for detecting distracted vehicle operation, comprising: using a camera surveillance system (CMS) controller to identify how centered a vehicle including said CMS controller is within a first lane line and a second lane line by analyzing at least one image generated by a camera surveillance system; identifying that a distracted driving event has occurred in response to the centering of the vehicle deviating from a lane center according to a predetermined characteristic; A method comprising:

2. The method of claim 1 , wherein the predetermined characteristic comprises at least one of instantaneous vehicle centering and vehicle centering over time.

3. the predetermined characteristic is the vehicle centering over time; 3. The method of claim 2, wherein the deviation is a difference between a curvature of a line defined by the vehicle centering over time and a curvature of a lane defined by the first lane line and the second lane line.

4. 4. The method of claim 3, wherein the lane curvature is determined, at least in part, by determining a geospatial position of the vehicle as determined by the CMS controller and comparing the geospatial position to map data.

5. The method of claim 3 , wherein the lane curvature is determined, at least in part, using image analysis of the at least one image produced by the camera surveillance system.

6. The method of claim 5 , wherein the lane curvature is determined using only image analysis of the at least one image produced by the camera surveillance system.

7. The method of claim 1 , further comprising initiating a distracted driving response in response to identifying the distracted driving event.

8. 10. The method of claim 7, wherein the distracted driving response comprises any combination of one or more of storing driving parameters for subsequent upload to a remote fleet monitoring system, inattentively uploading driving parameters to the remote fleet monitoring system, activating an audio alert to a vehicle operator, and activating a visual alert to the vehicle operator.

9. The method of claim 8 , wherein the driving parameters include at least two of speed, steering angle, gear, and engine RPM, and the at least one image produced by the camera surveillance system.

10. 2. The method of claim 1, wherein the at least one image generated by the camera surveillance system includes at least one of a rear-facing image generated by a trailer-mounted rear-facing camera and an image generated by a Class II / IV mirror.

11. 1. A camera surveillance system (CMS) for a vehicle, comprising: a plurality of exterior-facing cameras positioned around the vehicle, each camera defining a field of view configured to include at least one lane line defining a lane in which the vehicle is operating while the vehicle is moving; a CMS controller configured to receive images from each of the plurality of external-facing cameras, the CMS controller including a non-transitory memory and a processor; the non-transitory memory configured to cause the controller to: identify, by analyzing at least one image generated by the camera surveillance system, how centered a vehicle including the CMS controller is within a first lane line and a second lane line; and, in response to the centering of the vehicle deviating from a lane center according to a predetermined characteristic, identify that a distracted driving event has occurred.

12. 12. The camera surveillance system of claim 11, wherein the predetermined characteristics include at least one of instantaneous vehicle centering and vehicle centering over time.

13. the predetermined characteristic is vehicle centering over time; 12. The camera surveillance system of claim 11, wherein the deviation is the difference between the curvature of a line defined by the vehicle centering over time and the curvature of a lane defined by the first lane line and the second lane line.

14. The camera surveillance system of claim 11 , wherein the memory further stores instructions for causing the camera surveillance system to activate a distracted driving response in response to identifying the distracted driving event.

15. 15. The camera surveillance system of claim 14, wherein the distracted driving response includes any combination of one or more of storing driving parameters for subsequent upload to a remote fleet monitoring system, uploading distracted driving parameters to the remote fleet monitoring system, activating an audio alert to a vehicle operator, and activating a visual alert to the vehicle operator.

Images