Camera surveillance system including image-based driving error analysis function

Abstract

A method for detecting distracted driving includes using a camera monitoring system (CMS) controller to identify how centered a vehicle including a CMS controller is within a first lane line and a second lane line by analyzing at least one image generated by the camera monitoring system and identifying that a distracted driving event has occurred in response to the vehicle centering deviating from the lane center according to a predetermined characteristic.

Description

Technical Field

[0001] The present disclosure relates to a camera monitoring system (CMS) for a vehicle, and more particularly to a CMS module configured to identify and respond to driving impairments using computer-based analysis of images generated by the CMS.

Background Art

[0002] Mirror replacement systems, and camera systems that complement the mirror view, are utilized in vehicles to enhance the ability of the vehicle operator to view the surrounding environment. A camera monitoring system (CMS) includes one or more camera systems that provide a field of view including forward, side, and rear views to the vehicle operator. In some examples, the camera system covers a wider field of view than a conventional mirror, or includes views that are not fully obtainable via a conventional mirror and can be used as mirror supplementation and / or mirror replacement. In other examples, the CMS may use computer algorithms and processors to generate stitched, manipulated, and / or extrapolated views that can provide substantial additional information to the vehicle operator.

[0003] In addition to mirror replacement, images provided via cameras within the CMS can be utilized to detect aspects of the environment and aspects of the vehicle, and can be used with image processing-based detection processes that can achieve various safety, convenience, and operational efficiency advantages.

Summary of the Invention

[0004] An exemplary method for detecting inattentive vehicle operation includes the steps of: using a camera monitoring system (CMS) controller to analyze at least one image generated by the CMS to determine the extent to which a vehicle including the CMS controller is centered within a first lane line and a second lane line; and determining that an inattentive driving event has occurred in response to a deviation of the vehicle's centering from the lane center according to predetermined characteristics.

[0005] In another example of the above method for detecting inattentive vehicle operation, the predetermined characteristics include at least one of instantaneous vehicle centering and vehicle centering over time.

[0006] In another example of any of the above methods for detecting inattentive vehicle operation, the predetermined characteristic is the vehicle centering over time, and the deviation is the difference between the curvature of the line defined by the vehicle centering over time and the curvature of the lane defined by the first lane line and the second lane line.

[0007] In another example of any of the above methods for detecting inattentive vehicle operation, the curvature of the lane is determined, at least in part, by determining the geospatial position of the vehicle, which is determined by the CMS controller, and comparing the geospatial position with map data.

[0008] In another example of any of the above methods for detecting inattentive vehicle operation, the curvature of the lane is determined, at least in part, using image analysis of the at least one image generated by the CMS.

[0009] In another example of any of the above methods for detecting inattentive vehicle operation, the curvature of the lane is determined solely by image analysis of the at least one image generated by the CMS.

[0010] Another example of any of the above-described methods for detecting inattentive vehicle operation further includes activating an inattentive driving response in response to identifying the inattentive driving event.

[0011] In another example of any of the methods described above for detecting inattentive vehicle operation, the inattentive driving response includes any combination of one or more of the following: storing driving parameters for upload to a subsequent remote fleet monitoring system; inattentively uploading the driving parameters to the remote fleet monitoring system; triggering an audible alert to the vehicle operator; and triggering a visual alert to the vehicle operator.

[0012] In another example of any of the above methods for detecting inattentive vehicle operation, the driving parameters include at least two of speed, steering angle, gear, and engine RPM, and the at least one image generated by the CMS.

[0013] In another example of any of the above methods for detecting inattentive vehicle operation, the at least one image generated by the CMS includes at least one of a rear-facing image generated by a rear-facing camera mounted on the trailer and an image generated by a Class II / IV mirror.

[0014] In one exemplary embodiment, a camera monitoring system (CMS) for a vehicle includes a plurality of outward-facing cameras positioned around the vehicle, each camera defining a field of view configured to include at least one lane line defining the lane in which the vehicle is traveling while it is moving, and a CMS controller configured to receive images from each of the plurality of outward-facing cameras, and including non-temporary memory and a processor, wherein the non-temporary memory is configured to analyze at least one image generated by the CMS to cause the controller to identify the extent to which the vehicle including the CMS controller is centered within a first lane line and a second lane line, and to identify that an inattentive driving event has occurred in response to a deviation of the vehicle centering from the lane center according to predetermined characteristics.

[0015] In another example of the CMS for the vehicle described above, the predetermined characteristics include at least one of instantaneous vehicle centering and vehicle centering over time.

[0016] In any other example of the CMS for vehicles described above, the predetermined characteristic is the vehicle centering over time, The deviation is the difference between the curvature of the line defined by the vehicle centering over time and the curvature of the lane defined by the first lane line and the second lane line.

[0017] In any other example of the vehicle CMS described above, the memory further stores instructions for the CMS to activate a distracted driving response in response to identifying the distracted driving event.

[0018] In another example of any of the above-described vehicle CMSs, the distracted driving response includes one or more arbitrary combinations of memorizing driving parameters for upload to a subsequent remote fleet monitoring system, inadvertently uploading the driving parameters to the remote fleet monitoring system, activating an audio alert to a vehicle operator, and activating a visual alert to the vehicle operator.

[0019] These and other features of the present invention can be best understood from the following specification and drawings.

Brief Description of the Drawings

[0020] This disclosure can be further understood by referring to the following detailed description in conjunction with the accompanying drawings.

[0021] [Figure 1A] It is a schematic front view of a commercial truck equipped with a camera monitoring system (CMS) used to provide at least Class II and Class IV views.

[0022] [Figure 1B] It is a schematic top view of a commercial truck equipped with a camera monitoring system that provides views of Class II, Class IV, Class V, and Class VI.

[0023] [Figure 2] It is a schematic upper perspective view of a vehicle cab including a display and an in-vehicle camera.

[0024] [Figure 3] It schematically shows the views generated by the forward camera in the CMS of FIGS. 1A and 1B.

[0025] [Figure 4] It schematically shows an example of a rear side view generated by the CMS of FIGS. 1A and 1B.

[0026] [Figure 5] This provides a schematic overview of the operation of the image-based operational fault detection module within the CMS.

[0027] [Figure 6] This example demonstrates a high-level process for identifying operational failure events and triggering responses using a CMS (Critical Management System).

[0028] [Figure 7] Figure 5 shows a high-level method for responding to inattentive driving events detected using the method described therein.

[0029] The embodiments, examples, and substitutes, claims, or the following descriptions and drawings in the paragraphs above may be adopted independently or in any combination, including any of their various aspects or their respective individual features. Features described in relation to one embodiment are applicable to all embodiments unless such features are incompatible. [Modes for carrying out the invention]

[0030] A schematic diagram of a commercial vehicle 10 is shown in Figures 1A and 1B. Figure 2 is a schematic upper perspective view of the cab of the vehicle 10, including a display and an interior camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be in any configuration. Although a commercial truck is intended in this disclosure, the present invention can be applied to other types of vehicles. The vehicle 10 incorporates a camera surveillance system (CMS) 15 (Figure 2), which includes camera arms 16a and 16b on the driver's and passenger's sides mounted on the outside of the vehicle cab 12. If necessary, the camera arms 16a and 16b may also include conventional mirrors integrated with them, but the CMS 15 can also completely replace the mirrors. In additional examples, multiple camera arms may be included on each side, each arm housing one or more cameras and / or mirrors.

[0031] Each camera arm 16a, 16b includes a base fixed to, for example, the driver's cab 12. A swivel arm is supported by a base and may be articulated to it. At least one rear-facing camera 20a, 20b is positioned within each camera arm. Each of the external cameras 20a, 20b has an external field of view (FOV) that includes at least one of the Class II view and Class IV view (Figure 1b), which are legally defined views in the commercial truck industry. EX1 FOV EX2 This provides the following. If necessary, multiple cameras may be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in the European R46 Act, and the United States and other countries also have similar driver visibility requirements for commercial trucks. The reference to “class” views is not intended to be limiting, but rather to be an example of the types of views that may be provided to the display by a particular camera. Each arm 16a, 16b may also provide a housing that encloses electronics configured to provide various features of the CMS 15.

[0032] The first and second video displays 18a and 18b are located on or near the A-pillars 19a and 19b in the driver's seat 12, on the driver's side and passenger's side, respectively, and display Class II and Class IV views on each side of the vehicle 10, which provide a rearward view along the vehicle 10 captured by the external cameras 20a and 20b.

[0033] If images of Class V and / or Class VI views are also required, a camera housing 16c and a forward-facing camera 20c may be positioned in front of or near the front of the vehicle 10 to provide these views (Figure 1B). The forward-facing camera 20c is configured to include Class V and Class VI regions and to extend its field of view beyond the Class V and Class VI regions toward the horizon. A third display 18c, positioned in the driver's cab 12 near the upper center of the windshield, can be used to display Class V and Class VI views toward the front of the vehicle 10 to the driver. Displays 18a, 18b, and 18c are oriented toward the driver's area 24 in the driver's cab 22 where the driver is seated in the driver's seat 26. The position, size, and (multiple) fields of view streamed to a particular display may differ from the configurations described herein, but still constitute the invention of this disclosure.

[0034] If a Class VIII view is required, additional camera housings can be positioned on the sides and rear of the vehicle 10 to provide an additional field of view that includes part or all of the Class VIII zone of the vehicle 10. As illustrated, the Class VIII view includes a view surrounding the immediate vicinity of the trailer 14 and a rear-proximity view of the vehicle 10 that includes the area behind the trailer 14. In one example, the rear-proximity view of the vehicle 10 is generated by a rear-facing camera positioned at the rear of the vehicle 10 and may include both a direct rear-proximity view and a conventional rear view (e.g., a view extending to the horizon obtained by the rearview mirror of a vehicle without the trailer 14). In such an example, the third display 18c may include one or more frames displaying the Class VIII view. Alternatively, additional displays can be added near the first, second, and third displays 18a, 18b, 18c, or at another location in the driver's cab of the vehicle 10, to provide a dedicated display for providing the Class VIII view.

[0035] Continuing with reference to Figures 1A and 2, Figure 3 schematically shows an exemplary forward view 200, which may be generated by a forward-facing camera 20a or any similarly positioned camera. The view 200 includes a road 210 on which the vehicle 10 is traveling. The road 210 includes lane lines 212, 214, with the outer lane line 212 defined by a solid line and the inner lane line 214 defined by a dashed line segment. The lanes defined by lane lines 212, 214 are spaced apart from each other with a standardized width, and this width is constant along the length of the road 210. In most cases, the width of the road 210 and the lanes on the road 210 are standardized by local regulations and are consistent across multiple different roads within a given area. Simultaneously with the forward view 200 in Figure 3, the rear-facing side cameras 20a and 20b generate a rear-facing view 300 including the trailer 12 and the road 210, which is shown in Figure 4, where similar figures indicate similar elements. Figure 4 shows the view from the driver's side of vehicle 10, but a similar view is generated by the passenger-side camera 18b.

[0036] In yet another embodiment of the vehicle, an additional camera may be included in the CMS 15 to provide similar images including all or part of the road 210. The views expressly shown and described are illustrative, and any combination of views including lane lines 212, 214 may be used in the processes described herein.

[0037] Referring to Figures 1A to 4, Figure 5 schematically illustrates an example of the operation of a vehicle 10 equipped with the CMS 15 described herein. The CMS 15 includes image analysis software configured to identify the positions of lane lines 212, 214 in various images received from cameras 20a, 20b, 20c and other cameras within the CMS 15, using edge detection and similar processes. The positions of the lane lines 212, 214 relative to the vehicle 10 can be tracked over time by the CMS 15, thereby determining how centered the vehicle 10 is within the corresponding lanes 210a, 210b and identifying the path (driving line 410) that the vehicle 10 follows. This process is called image-based centering. Lane centering determines how far the center of the vehicle 10 is from each lane line 212, 214, and this distance can be used in multiple ways by the CMS 15 and other vehicle systems such as driver assistance systems.

[0038] The CMS15 further includes a driver failure detection and analysis module 17. The driver failure detection and analysis module 17 is a software module within the controller of the CMS15. The software module 17 is configured to analyze the vehicle path over time (driving line 410) determined by image-based lane centering, identify signs of driver failure, and respond.

[0039] During normal operation of vehicle 10, the driving fault detection and analysis module 17 uses edge detection image analysis on images 200,300 received from the camera in the CMS 15 to identify the edges of the lane lines and determine the vehicle's position relative to each lane (centering). The detected positions of each lane line 212,214 in the image are converted into a three-dimensional position relative to vehicle 10. The three-dimensional position is then used to determine the degree of centering (for example, how equal the distance 412,414 between the longitudinal center of vehicle 10 and the inner edges of the lane lines 212,214 is).

[0040] Next, the driving malfunction detection and analysis module 17 compares the degree to which the vehicle 10 is centered relative to a predetermined standard. If the centering does not meet the predetermined standard, the module 17 detects whether or not a driving malfunction event has occurred.

[0041] In one example, the criterion could be instantaneous centering. In this example, if vehicle 10 is significantly off-center by a predetermined amount (for example, if the difference between distances 412 and 414 exceeds a predetermined distance), the driver is considered inattentive. For example, the predetermined distance in this example is 2 feet. In a practical example, the deviation distance depends on the lane width, and a calibration table with a lookup table correlating road type and lane width can be used so that an appropriate distance can be selected. In yet another example, this value, determined by referring to the lookup table, can be adjusted based on a coefficient related to the actual lane width determined by CMS 15 using image analysis.

[0042] Whether the deviation from the center exceeds a predetermined distance beyond the edge can be calculated mathematically, and the CMS mark increments a potential operational fault counter. When the counter reaches a certain value (for example, after a certain number of cluster events have occurred within a given time frame), the operational fault is recorded. In certain implementations, the operational fault counter is decremented over time at a predefined rate to compensate for accidental events that might incorrectly increment the potential operational fault counter.

[0043] In certain cases, a lane width of 11–12 feet (3.35–3.66 meters) is recommended by the U.S. Federal Highway Administration. Since a trailer width is approximately 8 feet 4 inches (2.54 meters), a deviation exceeding 2 feet 4 inches (0.71 meters) may be used as a trigger for U.S. highways. Furthermore, the ratio between lane width, trailer width, and deviation can be used as a calibration parameter for roads with different lane widths and / or trailers outside the standard trailer width.

[0044] In another example, if the time-series centering of the vehicle (driving line 410) does not match, or does not closely match, the time-series curvature of the lane lines 212,214, the driver is judged to be inattentive. As used herein, the curvature of the lane lines refers to the path of the lane lines and may include, if any, extending straight sections that result in the smallest geometric curvature. The curvature of the lane lines 212,214 is determined, in one example, using the same image analysis process that the CMS 15 uses to identify the centering of the vehicle 10. As used herein, “closely match” means that the driving line 410 has a contour that includes the smallest deviation from the lane lines 212,214, as expected of an attentive driver. The deviation from the lane lines 212,214 can be determined using the same or similar predetermined distance and / or adjusted predetermined distance method as described above with respect to instantaneous centering.

[0045] In yet another alternative example, inattentive driving is detected when vehicle 10 is off-center by a larger amount than a predetermined amount, or when the vehicle's centering over time does not substantially match the curvature of the lane line.

[0046] In some examples, the CMS15 further includes a GPS and / or geospatial positioning system 21 configured to identify the geospatial location of the vehicle 10. In these examples, the geospatial location of the vehicle 10 is used to identify the location of the vehicle 10 relative to a set of map data, and the set of map data is used to identify the predicted contours of roads 210a, 210b. The predicted contours of the roads are then used to verify the accuracy of the image-based lane line contours, or as an alternative to the image-based lane line contours. In this way, the positioning system 21 complements and enhances the image-based analysis.

[0047] Referring to Figures 1 through 5, Figure 6 shows a method 500 for performing the general procedure described above. The CMS 15 first detects the vehicle's centering and tracks the centering over time in step 510, “detect vehicle centering over time”. As described above, this centering determines the driving line 410, which is compared to an instantaneous centering threshold, a lane contour, or both in step 520, “compare centering to predetermined criteria”. If the predetermined criteria are met, the driver error detection and analysis module 17 activates one or more inattentive driver responses in step 530, “activate inattentive responses”.

[0048] Inattentive responses are, in one example, auditory and / or visual alerts to the vehicle operator. The alert draws the driver's attention as an active attempt to correct inattentive driving. In one example, the alert may include a flashing warning on a display screen accompanied by an audible audio output.

[0049] In another example, the inattention response is the activation of a fleet monitoring system that monitors data related to inattentional driving and stores the monitored data for subsequent upload to a fleet monitoring system or any similar system or database for monitoring vehicle operations. In one variation of this example, the CMS15 includes remote network hardware that uploads the monitored data when inattentional driving occurs, and / or, instead stores the data locally for subsequent uploads.

[0050] The data stored and / or uploaded includes vehicle parameters (speed, steering angle, gear, engine RPM, vehicle geospatial position, etc.) from (multiple) general-purpose vehicle controllers acquired by the CMS via a CAN bus or similar system, image data from cameras within the CMS15, and any information derived from either or both of the vehicle parameters and / or image data (e.g., deviation from the centerline, road conditions, lane width, etc.). For local data storage for later upload, storage may occur only when inattentive driving detection is triggered, or it may be stored periodically depending on the amount of memory installed in the vehicle.

[0051] Once uploaded, the fleet management system uses the careless driving data to continuously monitor the driving statistics of a given driver, monitor the driving of the entire fleet, provide data for potential accident reporting, or use it for any similar purposes.

[0052] Referring to Figures 1 through 6, Figure 7 schematically illustrates method 600 for responding to an inattentive driving warning. First, when the CMS 15 indicates an inattentive driver, in accordance with method 500 above, the CMS 15 activates the driver monitoring subroutine in step 610, “activate driver monitoring,” and substantially simultaneously activates the driver alert system within the CMS 15 in step 620, “activate driver alert.” As used herein, “substantially” means that steps 610 and 620 occur independently of each other but at similar points in time. It is understood that in actual embodiments, a single controller may not be able to achieve exact simultaneous operation.

[0053] Once activated, the driver monitoring step 610 determines that the CMS 15 is collecting data related to inattentive driving, such as data coming into the CMS 15 (e.g., speed, steering angle, gear, engine RPM, vehicle geospatial position), image data used for initial detection, continuous image data received from cameras (e.g., Class IV images from both sides), and vehicle driving data from the general-purpose vehicle controller during inattentive driving, and stores this data in the non-temporary memory of the CMS 15 in the "start data storage" step 630.

[0054] Data storage continues for a predetermined period (e.g., 5 minutes) after the vehicle driver resumes careful driving, or until the later of the minimum period. The CMS15 then terminates the driver monitoring process in step 640, "terminate driver monitoring." Following the termination of data collection, the stored data is uploaded to the fleet monitoring system in step 650, "upload data to fleet system."

[0055] In some cases, if vehicle 10 has a wireless data connection that can connect to a central data repository, data can be uploaded simultaneously with collection and storage until the inattentive driving event is stopped.

[0056] By implementing the CMS functions described above, vehicle operation can be monitored for fleet owners, insurance purposes, and / or driver feedback, allowing for flagging and tracking of inattentive driving. Furthermore, in some cases, monitoring can be achieved without direct monitoring of the vehicle operator (e.g., video recording) by using the monitoring systems described herein.

[0057] While exemplary embodiments have been disclosed, those skilled in the art will recognize that certain modifications fall within the scope of the claims. Therefore, the following claims should be considered in order to determine their true scope and content.

Claims

1. A method for detecting careless driving, A step of identifying the vehicle centering of a vehicle including a CMS (Camera Monitoring System) controller by analyzing at least one image generated by a camera monitoring system using the CMS controller, wherein the vehicle centering corresponds to the extent to which the vehicle is centered within a first lane line and a second lane line. The steps include: incrementing a driving error counter in response to the vehicle centering deviating from the lane center according to predetermined characteristics; To compensate for accidental events, the process involves decrementing the operational failure counter over time at a predefined rate, The steps include: activating an inattentive driving response based on the fact that the aforementioned driving failure counter reaches a specific value within a predetermined period; Methods that include...

2. The method according to claim 1, wherein the predetermined characteristic includes at least one of the instantaneous value of the vehicle centering and the vehicle centering over time.

3. The predetermined characteristic is the vehicle centering over time, The method according to claim 2, wherein the deviation is the difference between the curvature of the line defined by the vehicle centering over time and the curvature of the lane defined by the first lane line and the second lane line.

4. The method according to claim 3, wherein the curvature of the lane is determined at least in part by determining the geospatial position of the vehicle determined by the CMS controller and comparing the geospatial position with map data.

5. The method according to claim 3, wherein the curvature of the lane is determined at least in part by image analysis of the at least one image generated by the camera monitoring system.

6. The method according to claim 5, wherein the curvature of the lane is determined using only image analysis of the at least one image generated by the camera monitoring system.

7. The method according to claim 1, wherein the inattentive driving response includes one or any combination of the following: storing driving parameters for upload to a subsequent remote fleet monitoring system; uploading the driving parameters to the remote fleet monitoring system; triggering an audible alert to the vehicle operator; and triggering a visual alert to the vehicle operator.

8. The method according to claim 7, wherein the driving parameters include at least two of speed, steering angle, gear, and engine RPM, and at least one image generated by the camera monitoring system.

9. The method according to claim 1, wherein the at least one image generated by the camera monitoring system includes at least one of a rear-facing image generated by a rear-facing camera mounted on the trailer and an image generated by a Class II / IV mirror.

10. A camera monitoring system (CMS) for vehicles, A plurality of outward-facing cameras arranged around the vehicle, each camera having a field of view configured to include at least one lane line defining the lane in which the vehicle is operating while it is moving, A CMS controller, which includes non-temporary memory and a processor, is configured to receive images from each of the multiple outward-facing cameras. The non-temporary memory is provided, The vehicle centering of the vehicle including the CMS controller is identified by analyzing at least one image generated by the camera monitoring system, wherein the vehicle centering corresponds to the degree to which the vehicle is centered within the first lane line and the second lane line. In response to the vehicle centering deviating from the lane center according to predetermined characteristics, the driving error counter is incremented. To compensate for accidental events, the operational failure counter is decremented over time at a predefined rate, and The inattentive driving response is activated based on the fact that the aforementioned driving failure counter reaches a specific value within a predetermined period. A camera surveillance system configured to have the controller perform the aforementioned action.

11. The camera monitoring system according to claim 10, wherein the predetermined characteristics include at least one of the instantaneous value of the vehicle centering and the vehicle centering over time.

12. The aforementioned predetermined characteristic is vehicle centering over time. The camera monitoring system according to claim 10, wherein the deviation is the difference between the curvature of the line defined by the vehicle centering over time and the curvature of the lane defined by the first lane line and the second lane line.

13. The camera monitoring system according to claim 10, wherein the inattentive driving response includes one or any combination of the following: storing driving parameters for subsequent upload to a remote fleet monitoring system; uploading the driving parameters to the remote fleet monitoring system; triggering an audible alert to the vehicle operator; and triggering a visual alert to the vehicle operator.

Images