Camera mirror system display for commercial vehicle including system for identifying road marking
Patent Information
- Application Number
- JP2022161489
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-10-19
- Filing Date
- 2022-10-06
- Publication Date
- 2025-10-14
- Estimated Expiration
- 2042-10-06
AI Technical Summary
Existing camera mirror systems for commercial vehicles struggle to distinguish between trailer edges and visually similar road features like lane lines, parking lines, or curbs, leading to inaccurate trailer angle detection.
A method involving image processing techniques to identify road features by converting RGB images to grayscale, applying edge detection, using a Hough transform to extract lines, and comparing line characteristics with known road sign features to differentiate between trailer edges and road signs, thereby enhancing trailer angle detection accuracy.
This approach minimizes errors in trailer angle detection by reducing reliance on computationally expensive polynomial fitting, accurately identifying road features and maintaining the trailer's position within the camera view.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical Field]
[0001] The present disclosure relates to a camera mirror system (CMS) for use in commercial trucks and an automatic view panning system therefor. [Background technology]
[0002] Mirror replacement systems, and camera systems that supplement the mirror view, are utilized in commercial vehicles to enhance the vehicle operator's ability to view their surrounding environment. Camera mirror systems (CMS) utilize one or more cameras to provide the vehicle operator with an enhanced field of view. In some instances, mirror replacement systems cover a wider view than traditional mirrors or include views not fully available through traditional mirrors.
[0003] One feature included in some camera mirror systems is an auto-pan function that automatically or semi-automatically pans the camera view to maintain a view of the trailing edge of the trailer. In some implementations, identification of the trailer edge is used to monitor trailer parameters and characteristics, which are then used to assist in identifying the location of the trailer edge.
[0004] One of the difficulties faced by existing systems is distinguishing between the edge of a trailer and visually similar lines in the field of view represented by lane lines, parking lines, curbs, or similar linear road features or markings. Summary of the Invention
[0005] In one exemplary embodiment, a method for identifying road features in an image includes receiving an image at a controller; using the controller to identify an area of interest in the image and convert the area of interest from red-green-blue (RGB) to a single color; using the controller to detect a set of edges in the area of interest and identify at least one line in the set of edges; comparing the identified at least one line with at least one line of a set of known and expected road sign features, and identifying a set of at least one first line of the at least one line as corresponding to a road feature in response to the at least one first line matching the set of known and expected road sign features.
[0006] In another example of the above method for identifying road features in an image, identifying the set of known and expected road marking features includes an expected width of at least one parking line and lane line.
[0007] Another example of any of the above methods for identifying road features in an image further includes identifying two lines in the at least one first set of lines as corresponding to lane lines if the two lines are substantially parallel and consistently spaced between 4.5 inches (11.43 cm) and 6.5 inches (16.51 cm).
[0008] In another example of any of the above methods for identifying road features in an image, the line spacing is an average of the shortest distance between the two substantially parallel lines.
[0009] In another example of any of the above methods for identifying road features in an image, identifying the set of lines within the set of edges includes feature extraction by Hough transform.
[0010] In another example of any of the above methods for identifying road features in an image, identifying the set of lines within the set of edges further comprises filtering background noise from an output of the feature extraction.
[0011] In another example of any of the above methods for identifying road features in an image, the set of known and expected road marking features includes known and expected features of parking lines, lane lines, and curbs.
[0012] Another example of any of the above methods for identifying road features in an image further includes, in response to all lines in the set of at least one second line not matching the set of known and expected road sign features, identifying the set of at least one second line in the at least one line as corresponding to a trailer feature.
[0013] Another example of any of the above methods for identifying road features in an image further includes digitally removing edges that do not correspond to lines in the at least one second set of lines to create a filtered edge image, and providing the filtered edge image to a trailer feature detection module of a camera mirror system (CMS).
[0014] Another example of any of the above methods for identifying road features in an image further includes using the trailer feature detection module to identify a bottom line of a trailer and determining a trailer angle at least in part using the position of the bottom line of the trailer within the region of interest.
[0015] Another example of any of the above methods for identifying road features in an image further includes panning the CMS view based at least in part on the determined trailer angle so that the trailer end is maintained by the CMS view.
[0016] In another example of any of the above methods for identifying road features in an image, using the controller to identify at least one line in the set of edges is performed without using polynomial data fitting.
[0017] In another example of any of the above methods for identifying road features in an image, converting the region of interest from red-green-blue (RGB) to a single color using the controller includes converting the region of interest to grayscale or extracting a green channel from the region of interest.
[0018] Another example of any of the above methods for identifying road features in an image further includes distinguishing between at least two corresponding road features by identifying a color of at least the first line and comparing the identified color to an expected color of the corresponding road feature. In one exemplary embodiment, a camera mirror system for a vehicle includes at least one camera defining a field of view that includes a view of a ground surface; and a camera mirror system (CMS) controller including a processor and a memory, the memory storing instructions for causing the controller to perform an image-based detection method for identifying road features in an image, the image-based detection method including receiving an image from the at least one camera at the controller; identifying an area of interest in the image and converting the area of interest from red-green-blue (RGB) to gray using the controller; detecting a set of edges in the area of interest and identifying at least one line within the set of edges using the controller; comparing the identified at least one line with at least one line of a set of known and expected road sign features; and identifying a first set of at least one line of the at least one line as corresponding to a road feature in response to the at least one first line matching the set of known and expected road sign features.
[0019] In another example of the above vehicle measurement system, the at least one camera defines a class II and a class IV view.
[0020] In another example of any of the above measurement systems for a vehicle, comparing the at least one line to a set of known and expected road sign features includes identifying the at least one second set of lines in the at least one line as corresponding to a trailer feature in response to all lines in the set of at least one second line not matching the set of known and expected road sign features.
[0021] In another example of any of the above measurement systems for a vehicle, the controller further includes a trailer feature detection module; The trailer feature detection module is configured to determine a trailer angle based at least in part on a position of the at least one second line in the image.
[0022] In another example of any of the above measurement systems for a vehicle, the controller is further configured to pan the Class II view presented to the vehicle driver based at least in part on the determined trailer angle so that the Class II view includes at least a portion of the trailer end.
[0023] These and other features of the present invention can be best understood from the following specification and drawings.
[0024] The present disclosure may be better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings. [Brief explanation of the drawings]
[0025] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera mirror system (CMS) used to provide at least Class II and Class IV views. [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera mirror system providing Class II, Class IV, Class V, and Class VI views. [Figure 2] FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an interior camera. [Figure 3] 1 is a flowchart of a method for identifying road signs. [Figure 4] Figure 4A shows a partial CMS view including a road with lane lines, and Figure 4B shows the same partial CMS view after identifying edges in the image. DETAILED DESCRIPTION OF THE INVENTION
[0026] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible.
[0027] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. While commercial trucks are contemplated in this disclosure, the invention may be applied to other types of vehicles. The vehicle 10 incorporates a camera mirror system (CMS) 15 (FIG. 2) with driver and passenger side camera arms 16a, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, or the CMS 15 may be used to completely replace the mirrors. In additional examples, multiple camera arms may be included on each side, with each arm housing one or more cameras and / or mirrors.
[0028] Each camera arm 16a, 16b includes a base fixed to, for example, the cab 12. A pivoting arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20a, 20b is disposed within each camera arm. Each of the exterior cameras 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II and a Class IV view (FIG. 1B), which are legally defined views in the commercial trucking industry. EX1 , FOV EX2 If desired, multiple cameras may be used on each camera arm 16 a, 16 b to provide these views. Each arm 16 a, 16 b may also provide a housing that encloses electronics configured to provide various features of CMS 15.
[0029] First and second video displays 18a, 18b are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19a, 19b and display Class II and Class IV views on each side of the vehicle 10, which provide rearward views along the vehicle 10 captured by exterior cameras 20a, 20b.
[0030] If Class V and Class VI view footage is also desired, a camera housing 16c and camera 20c may be located at or near the front of the vehicle 10 to provide these views (FIG. 1B). A third display 18c located within the cab 12 near the top center of the windshield may be used to display Class V and Class VI views toward the front of the vehicle 10 to the driver.
[0031] If Class VIII view video is required, camera housings can be positioned on the sides and rear of vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of vehicle 10. In such an example, third display 18c can include one or more frames displaying the Class VIII view. Alternatively, additional displays can be added near first, second, and third displays 18a, 18b, 18c to provide a dedicated display providing the Class VIII view. Displays 18a, 18b, 18c face toward a driver's area 24 within cab 22, where a driver sits in driver's seat 26.
[0032] In some implementations, the CMS may include an automatic panning function that pans the Class II view within the Class IV view to maintain the trailer's end position within the Class II view, thereby providing enhanced visibility to the vehicle driver. To provide this functionality, the CMS includes a set of image feature detection algorithms that use image-based analysis of the CMS view to detect trailer features, including the trailer angle. The trailer angle is detected at least in part based on the bottom line of the trailer detected within the Class IV view. Certain road markings, such as parking lines, lane lines, and curbs, define long lines within the CMS view that can be difficult to distinguish from the trailer's edge line using conventional systems, such as polynomial data fitting, across the image. Therefore, lane lines, parking lines, curbs, and other similar elements positioned within the view can cause significant angle detection errors if the trailer angle detection erroneously identifies the road marking or feature as the trailer's edge.
[0033] FIG. 3 illustrates a method implemented by a CMS to distinguish between trailer edge lines and road signs by identifying lines in an image without using computationally expensive polynomial data fitting. First, the method identifies a region of interest and crops the remaining portion from the image by "cropping the region of interest" in step 310. The region of interest may be identified using any conventional methodology to reduce the image feed to a portion containing potential edge lines. The region of interest is selected to minimize the analyzed area while preserving desired features in the region of interest. In one example, the region is chosen so that the trailer edge is always present in the region of interest for all possible trailer angles. The region is made as small as possible to eliminate noise from the image and reduce potential errors caused by noise.
[0034] Once cropping is complete, the red-green-blue (RGB) images captured by the CMS camera are converted to grayscale via "Convert RGB to Gray" in step 320. Converting the image(s) to gray increases the contrast at each potential edge, further reducing the processing required to distinguish edges within the region of interest. In another example, the conversion to gray step can be omitted, and the edge detection processing described below can be performed on a single color (e.g., green) extracted from the image.
[0035] Once the cropping and conversion to gray is complete, an edge detection algorithm is run on the gray image by "Perform Edge Detection" in step 350. Edge detection algorithms include various mathematical methods aimed at identifying points in a digital image where the image intensity changes abruptly or where there are discontinuities. The points where the image intensity changes abruptly are organized into a set of curved segments called edges.
[0036] 4A and 4B illustrate an exemplary transformation of a gray image region 410 (FIG. 4A) into an edge-detected image 420 (FIG. 4B). The gray image region 410 includes a road surface 412 that includes road markings 414, such as lane lines or parking lines. The road markings are white stripes, but may also have yellow stripes or other colors depending on the geographic region and their location on the road surface. In addition to the road markings 414, the road surface may include numerous potholes, rocks, rough patches, etc., which may also result in the presence of one or more edges in the image. After edge detection is performed, the edge image 420 includes numerous edges 422, 424, 426, including edges 424, 426 of the road markings 414 as well as edges 422 resulting from various road roughnesses.
[0037] To separate edges 424, 426 corresponding to road sign lines and trailer edges from noisy edge detections, the CMS uses a Hough transform to find lines within the edges detected by "Find Lines in Edges" in step 340. The Hough transform is a feature extraction technique used in image analysis to find imperfect instances of objects within a given class of shapes through a voting procedure. This voting procedure is performed in parameter space, from which object candidates are obtained as local maxima in a so-called accumulator space explicitly constructed by an algorithm for computing the Hough transform. While the use of the known Hough transform process to identify lines in edge image 420 is described here, it will be understood that other methodologies for detecting lines in edge images, including neural network-based line identification, may be used in alternative embodiments with minimal modifications to the overall system.
[0038] After identifying each line in the edge image 420, the CMS uses a road sign identification module in the process to determine whether the identified lines 424, 426 correspond to road signs, per step 350, "Identify Road Signs." The road sign identification module stores known and expected characteristics of road signs and compares the stored known and expected characteristics with the identified lines. If the known and expected characteristics of a road feature match the line detected by the CMS, the CMS determines that the line corresponds to a road feature. In one example, a road sign (e.g., a lane line and a parking line) results in two edge lines that follow at least approximately the same trajectory, which can be said to be approximately parallel. In another example, a road sign is characterized by the presence of two lines separated by a standard distance or a standard distance range. By way of example, lane lines may be expected to be between 5 inches (12.17 cm) and 6 inches (15.24 cm) wide. In such an example, the identification module looks for two adjacent lines that are between 4.5 inches (11.43 cm) and 6.5 inches (16.51 cm) apart, and identifies the set of edge lines within that range as corresponding to lane lines, with an additional 0.5 inches included for each edge of the range to account for variations that may occur due to imperfect printing of lane lines, parking lines overlapping existing lane lines, and similar real-world variations.
[0039] Similar ranges may be identified and used for parking lines at 3.5 inches (8.89 cm) to 4.5 inches (11.43 cm), curbs at 5.5 inches (13.97 cm) to 6.5 inches (16.51 cm), or other road markings. In another example, the distance between edge lines 424, 426 may be measured in image pixels instead of distance. In one such example, the range for parking lines and lane lines may be 16 to 25 pixels.
[0040] In another example, color data for each identified road feature can be used to distinguish between types of features (e.g., yellow lines indicating separation between lane lines of opposing traffic directions can be distinguished from white lines indicating separation between lane lines of parallel traffic directions).
[0041] If the identification module determines that the edge lines 424, 426 correspond to a road sign, the lines 424, 426 are filtered along with the noise edge 422 by "Filter Noise" in step 360. If a line remains after filtering the noise, the process presumes that the line corresponds to a trailer line and provides the location of the trailer line(s) to a trailer feature estimation module in the CMS. The trailer feature estimation module determines whether the remaining line(s) are the end of a trailer, the bottom edge of a trailer, or other trailer features, and calculates the line angle and the world coordinates of the line in the image plane by "Determine Trailer Angle" in step 370. The line angle is used to determine the trailer angle according to known trailer angle determination methods.
[0042] By using the above discrimination method, the trailer angle determination system avoids reliance on cubic polynomial data fitting over the entire area and enables detection of road features that are perpendicular or nearly perpendicular to the trailer edge, thereby minimizing the occurrence of false trailer angle detection.
[0043] While exemplary embodiments have been disclosed, one of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.
Claims
1. 1. A method for identifying road features in an image, comprising: receiving an image at a controller; using the controller to identify regions of interest in the image and convert the regions of interest from red-green-blue (RGB) to monochrome; using the controller to detect a set of edges within the region of interest and identify at least one line within the set of edges; comparing the at least one line with a set of known and expected road sign features and identifying a set of at least one first line among the at least one line as corresponding to a road feature in response to the at least one first line matching at least one of the set of known and expected road sign features; identifying a set of at least one second line in the at least one line as corresponding to a trailer feature in response to all lines in the set of at least one second line not matching the set of known and expected road sign features; A method comprising:
2. The method of claim 1 , wherein identifying the set of known and expected road sign features includes expected widths of at least one of parking lines and lane lines.
3. 3. The method of claim 2, further comprising identifying two lines in the at least one first set of lines as corresponding to lane lines responsive to the two lines being substantially parallel and consistently spaced between 4.5 inches (11.43 cm) and 6.5 inches (16.51 cm) apart.
4. The method of claim 3 , wherein the line spacing is the average of the shortest distance between the two substantially parallel lines.
5. The method of claim 1 , wherein identifying the set of lines within the set of edges comprises feature extraction by Hough transform.
6. The method of claim 5 , wherein identifying the set of lines within the set of edges further comprises filtering background noise from an output of the feature extraction.
7. The method of claim 1 , wherein the set of known and expected road marking features includes known and expected features for parking lines, lane lines, and curbs.
8. 8. The method of claim 7, further comprising digitally removing edges that do not correspond to lines in the at least one second set of lines to create a filtered edge image, and providing the filtered edge image to a trailer feature detection module of a camera mirror system (CMS).
9. 9. The method of claim 8, further comprising using the trailer feature detection module to identify a bottom line of a trailer and determining a trailer angle at least in part using a location of the bottom line of the trailer within the region of interest.
10. 10. The method of claim 9, further comprising panning the CMS view based at least in part on the determined trailer angle so that the trailer tail is maintained by the CMS view.
11. The method of claim 1 , wherein using the controller to identify at least one line in the set of edges is performed without using polynomial data fitting.
12. 2. The method of claim 1, wherein using the controller to convert the region of interest from red-green-blue (RGB) to monochrome includes converting the region of interest to grayscale or extracting a green channel from the region of interest.
13. 2. The method of claim 1, further comprising distinguishing between at least two corresponding road features by identifying a color of at least the first line and comparing the identified color to an expected color of the corresponding road feature.
14. at least one camera defining a field of view that includes a view of the ground; a camera mirror system (CMS) controller including a processor and a memory, the memory storing instructions for causing the controller to perform an image-based detection method for identifying road features in an image; Equipped with The image-based detection method comprises: receiving images from the at least one camera at the controller; using the controller to identify regions of interest in the image and convert the regions of interest from red-green-blue (RGB) to monochrome; using the controller to detect a set of edges within the region of interest and identify at least one line within the set of edges; comparing the at least one line with a set of known and expected road sign features and identifying a set of at least one first line among the at least one line as corresponding to road features in response to the at least one first line matching the set of known and expected road sign features; Includes and comparing the at least one line with a set of known and expected road sign features includes identifying a set of at least one second line in the at least one line as corresponding to a trailer feature in response to all lines in the set of at least one second line not matching the set of known and expected road sign features.
15. 15. The camera mirror system of claim 14, wherein the at least one camera defines a Class II and a Class IV view.
16. the controller further includes a trailer feature detection module; The camera mirror system of claim 14 , wherein the trailer feature detection module is configured to determine a trailer angle based at least in part on a position of the at least one second line in the image.
17. 17. The camera mirror system of claim 16, wherein the controller is further configured to pan the Class II view presented to the vehicle driver based at least in part on the determined trailer angle so that the Class II view includes at least a portion of a trailer end.
18. The camera mirror system of claim 14, wherein converting the target area from red-green-blue (RGB) to a single color using the controller includes converting the target area to grayscale or extracting a green channel from the target area.
19. A method for identifying features in an image, comprising: receiving an image at a controller; using the controller to identify regions of interest in the image and convert the regions of interest from red-green-blue (RGB) to monochrome; using the controller to detect a set of edges within the region of interest and identify at least one line within the set of edges; comparing the at least one line with a set of road sign features, and identifying a set of at least one first line among the at least one line as corresponding to a road feature in response to the at least one first line matching at least one of the set of road sign features; identifying a set of at least one second line in the at least one line as corresponding to a trailer feature in response to all lines in the set of at least one second line not matching the set of road sign features; A method comprising: