Method for displaying an image incorporating a human-machine interface for a commercial vehicle camera system, and vehicle system

Abstract

A method of operating a vehicle camera system includes receiving a first image from at least one video camera, identifying a first object in the first image, determining a distance between a vehicle component and the identified object, and modifying the first image by incorporating a human-machine interface (HMI) within the first image, the human-machine interface including a display configured to communicate the distance between the object and the vehicle component, and displaying the modified image to a vehicle operator.

Description

Technical Field

[0001] The present disclosure generally relates to commercial vehicle systems, and more particularly to human machine interfaces for displaying distance information.

[0002] (Cross - Reference to Related Applications) This application claims priority to U.S. Patent Application No. 63 / 041,176, filed on Jun. 19, 2020.

Background Art

[0003] Commercial vehicles such as tractor - trailers include a driving assistance system that assists the driver in moving the vehicle into a dock area, reverse operations, overtaking an object, and other similar vehicle operations. Existing vehicle systems incorporate proximity sensors and other similar systems to provide warnings when a commercial vehicle is within a threshold distance of an object. For example, the warning may include a repetitive beeping sound or a flashing light when an object is within the threshold distance.

[0004] Proximity - based warning systems have limited types of warnings they can provide and cannot effectively convey information more detailed than a binary detection of whether an object is within a threshold distance.

Summary of the Invention

[0005] An exemplary method for operating a vehicle camera system includes generating a first image using at least one video camera, identifying a first object in the first image, determining a distance between a vehicle component and the identified object, and modifying the first image by incorporating a human machine interface (HMI) within the first image, wherein the human machine interface includes a display configured to convey the distance between the object and the vehicle component, and displaying the modified image to a vehicle operator.

[0006] In another example of a method for operating the vehicle camera system described above, the step of determining the distance between the object and the vehicle component is at least partially based on image analysis of the first image.

[0007] In another example of any of the methods for operating the vehicle camera system described above, the step of determining the distance between the object and the vehicle component is at least partially based on physical sensor measurements.

[0008] In another example of any of the methods for operating the vehicle camera system described above, the physical sensor measurements include at least one of radar sensor measurements, LIDAR sensor measurements, infrared sensor measurements, time-of-flight sensors, and ultrasonic sensor measurements.

[0009] In another example of any of the methods for operating the vehicle camera system described above, the physical sensor measurements include radar sensor measurements and ultrasonic measurements.

[0010] In another example of any of the methods for operating the vehicle camera system described above, the step of modifying the first image by incorporating a human-machine interface (HMI) within the first image and the step of displaying the modified mirror replacement image to the vehicle operator are performed in response to the determined distance being less than a pre-defined threshold distance.

[0011] In another example of any of the methods for operating the vehicle camera system described above, the threshold distance is about 30 m.

[0012] In another example of any of the methods for operating the vehicle camera system described above, the threshold distance is an activation distance for at least one automated driving assistance function.

[0013] In another example of any of the methods for operating the vehicle camera system described above, the vehicle component is an ego portion.

[0014] In another example of any of the methods for operating the vehicle camera system described above, the ego portion is a trailer.

[0015] In another example of any of the methods for operating the vehicle camera system described above, the human machine interface includes at least one of a numerical indicator, a multicolor overlay, and a bar graph.

[0016] In another example of any of the methods for operating the vehicle camera system described above, the human machine interface includes a combination of at least two of the numerical indicator, the multicolor overlay, and the bar graph.

[0017] In another example of any of the methods for operating the vehicle camera system described above, the human machine interface includes an object indicator that identifies an object detected in the displayed image.

[0018] In another example of any of the methods for operating the vehicle camera system described above, the step of generating the first image includes generating a mirror replacement image by combining a plurality of images generated from different vehicle cameras.

[0019] In another example of any of the methods for operating the vehicle camera system described above, the method further includes the step of overlaying at least one distance line on the first image, and the at least one distance line is calibrated at a predefined distance using the distance displayed on the human machine interface.

[0020] In one exemplary embodiment, a vehicle system includes at least one outward-facing camera, a controller including an input connected to an output of the at least one outward-facing camera, and an in-vehicle display connected to the controller. The controller is configured to cause the controller to identify a first object in a first image received by the controller, determine a distance between the vehicle component and the identified object, modify the first image by incorporating a human machine interface (HMI) within the first image, and output the modified image to the display. The human machine interface includes a display configured to convey the distance between the object and the vehicle component.

[0021] In another example of the vehicle system described above, the controller is connected to a proximity sensor configured to determine a distance between the proximity sensor and a detected object.

[0022] In another example of any of the vehicle systems described above, the proximity sensor includes at least one of a radar sensor, a LIDAR sensor, an infrared sensor, a time-of-flight sensor, and an ultrasonic sensor.

[0023] In another example of any of the vehicle systems described above, the proximity sensor includes a radar sensor and an ultrasonic sensor.

[0024] In another example of any of the vehicle systems described above, the controller includes an object detection module based at least in part on an image.

[0025] In another example of any of the vehicle systems described above, the object detection module based at least in part on an image includes an object detection function based on an auxiliary sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0026]

Figure 1

[0027]

Figure 2

[0028]

Figure 3

[0029]

Figure 4

[0030]

Figure 5

[0031]

Figure 6

DETAILED DESCRIPTION OF THE INVENTION

[0032] A schematic view of a commercial truck 10 is shown in FIG. 1. The truck 10 includes a vehicle cab 12 that pulls a trailer 14. An exemplary trailer 14 is an ego part that can move independently of the cab 12. The driver and passenger side camera housings 16 are attached to the vehicle cab 12. In some examples, the camera housings 16 may also include conventional mirrors integrated with them. The first and second displays 18 are respectively arranged on the driver side and the passenger side within the vehicle cab 12, and display Class II and Class IV views on each side of the vehicle 10. Fewer or more displays than those shown may be used, including additional classes of displays, and the displays may be arranged in positions different from those shown. In another example, additional cameras 11 and 13 may be used, and stitching may be utilized when combining images to form a surround view image. The illustrated camera positions are merely exemplary, and the actual implementation may include multiple additional cameras of specific view types required in a given system.

[0033] FIG. 2 shows, in a very schematic manner, an example of a camera mirror system 20. In one example, first and second rearward cameras 22, 24 are arranged within the camera housing 16. The first and second cameras 22, 24 provide first and second fields of view 22 FOV1, FOV2 corresponding to, for example, Class IV and Class II views. The first and second fields of view FOV1, FOV2 overlap each other, providing an overlap region 25. However, it should be understood that the cameras may be arranged in locations different from those shown, and the fields of view provided may be related to completely other classes or other views.

[0034] The ECU or controller 26 communicates with the first and second cameras 22, 24. Various sensors 28 such as radar sensor 38, LIDAR sensor 40, infrared sensor 42, time-of-flight sensor, and / or ultrasonic sensor 44 may communicate with the controller 26. The sensors 28 and / or the first and second cameras 22, 24 are used to detect objects within the images captured by the first and second cameras 22, 24. Alternatively, or in addition to the sensors 28, an image-based object detection algorithm 29 may be included in the controller 26. The image-based object detection algorithm 29 may be a rule-based detection algorithm or an algorithm based on a neural network, and analyzes the images provided from the cameras 22, 24 to identify the presence of one or more objects within the (multiple) images.

[0035] For example, any number of suitable object detection schemes may be used, such as those that rely on a neural network and a three-dimensional geometry model to determine the position of an object in space, such as detection by ego motion. In the case of object detection using a neural network, the first camera 22 and the second camera 24 provide at least one of the sensors used to detect the object. In another example, any object detection system may be used to detect an object within the image plane, which includes not only image-based detection such as neural network analysis, but also detecting an image within the 3D space using a 3D space detection system such as radar, LIDAR, sensors, etc.

[0036] The controller 26 outputs a video signal to the display 18. The display 18 is arranged at a position visible to the vehicle operator inside the vehicle cab. In an example with a plurality of cameras as in the example of FIG. 2, the video signal is a combination of images from the first and second cameras 22, 24, and a stitching algorithm 30 is used to combine the images. In this example, the screen 32 of the display 18 provides a complete field of view 36 consisting of at least first and second adjusted fields of view 46, 48 from the first and second cameras 22, 24 joined by a stitching interface 34 (or also called stitching). The stitching algorithm 30 can utilize any known stitching system to create a single image visible to the vehicle operator.

[0037] Continuing to refer to FIGS. 1 and 2, FIG. 3 shows a mirror replacement display 200 that can be used in the display 18 etc. of FIGS. 1 and 2. The display 200 shows an image of a side view of the ego portion 210 of a commercial vehicle. Although shown as one image in the example of FIG. 2, it is understood that alternative mirror replacement images can be composed of stitched images as described above with respect to FIG. 2. It is understood that the ego portion 210 in a practical implementation may include any attached component that can move independently of the vehicle, but an exemplary ego portion 210 is a commercial shipping trailer. Also visible in the image 200 is an object 220. The object is at a distance 222 from the vehicle component 210.

[0038] The controller 26 shown in FIG. 2 detects the object 220 and determines the distance 222 between the ego portion 210 and the detected object 220. In some examples, the distance is determined using only image-based analysis via the controller 26. In another example, a sensor 212 such as a radar sensor, a LIDAR sensor, an infrared sensor, or an ultrasonic sensor is disposed on the ego portion 210 or the host vehicle. In such an example, the controller 26 receives the sensor output and uses a known distance determination method corresponding to the type of the sensor to determine the distance 222 to the object 220. In yet another example, the distance may be determined by a combination of image-based analysis and sensor-based analysis according to a known system for determining the distance. Further, one or more driving assistance systems 27 are also incorporated in the controller 26. By way of example, the driving assistance system 27 may include a dock assistance system, a trailer reverse assistance system, or other driving assistance systems.

[0039] To further assist the vehicle operator in steering, the controller 26 includes a human machine interface (HMI) module 25 configured to include a human machine interface that modifies the image displayed to the vehicle operator to identify the distance 220. The display example 200 in FIG. 3 implements the human machine interface by superimposing a line between the object 220 and the vehicle component 210. Next to the line, there is a numerical indicator defining the determined distance between the object 220 and the vehicle component 210. The arrow for the distance 222 may further include a color-coding indicating how close the vehicle component 210 and the object 220 are, and the arrow changes from green indicating a safe distance to red indicating a dangerous distance.

[0040] Continuing to refer to FIG. 3, FIG. 4 shows a display 200 including an alternative human machine interface 230. The alternative human machine interface 230 utilizes a color gradient overlaid on the object 220, a bar graph 234 indicating proximity to the object, and a numerical indicator 236 indicating distance. Each example provides an indication of the approximate distance between the object 220 and the vehicle component 210. As used herein, "approximate" refers to the accuracy of the distance determination 220.

[0041] Continuing to refer to FIGS. 3 and 4, FIG. 6 shows another alternative human machine interface 530 (HMI 530). The human machine interface 530 includes a numerical distance indicator 510 indicating the numerical distance and the unit of the numerical distance between the rear of the trailer 210 and the detected object 220. Below the numerical indicator 510, a distance indicator 520 that expands downward and outward is arranged, and the fact that the distance indicator 510 expands further downward and outward corresponds to the vehicle 210 being close to the detected object 220. The distance indicator 510 is separated into a plurality of sections 522, 524, 526. The sections are displayed as individual colors (e.g., the first section 522 is green, the second section 524 is orange, and the third section 526 is red). In an alternative embodiment, the sections can be distinguished by shading with light, medium, and dark shadings. Further, in an alternative example, more than three sections can be included in a similar manner.

[0042] The human machine interface 530 of FIG. 6 also includes a distance line 504. The distance line 504 is a static line overlaid on the image, and each line 504 corresponds to a specific calibrated distance from the rear of the trailer 210. The calibration line 504 overlaps with the sections of the distance indicator 520 and defines each of the separate regions 522, 524, 526 of the distance indicator 520.

[0043] The calibration of the distance line 504 is determined via a calibration process that uses the proximity and sensing system defined above. To calibrate the distance line 504, the vehicle operator maneuvers the vehicle until the vehicle is a predefined distance (e.g., 40 m) away from a detected object having an angle of 90 degrees with the ground, as indicated by the numerical indicator 510. When within the predetermined distance, the vehicle operator manually sets the corresponding distance line 504. The calibration line can be manually set using a dial, directional arrow buttons, or other conventional inputs to manually shift the horizontal distance line 504 vertically until the distance line is positioned at the bottom of the object. Next, the vehicle operator reverses the vehicle towards the object until the next predefined distance is reached and repeats the calibration process for each predefined distance.

[0044] In some examples, the distance line 504 can be maintained as part of the human-machine interface described herein. In another example, the distance line 504 may be generated or controlled by an individual vision system, and the human-machine interface system is used to perform the calibration of the distance line 504.

[0045] Continuing to refer to FIGS. 1 through 4, FIG. 5 schematically shows an example of a process 300 executed by the controller 26, whereby an image 200 including the human-machine interface 230 is created. Although described in the context of a mirror replacement system, it is understood that this process can be utilized by an image that complements the mirror instead of replacing the mirror, or by any vehicle system that includes a view independent of any mirror system.

[0046] First, the controller 26 receives video feeds from one or more cameras 22, 24 and generates an image for the display video feed in the image generation step 310. In an example where multiple images are stitched together, the stitching is performed by the controller 26 according to a known stitching or combination process.

[0047] Once generated, an object detection process is performed on the generated image 200, and any object 220 within the image 200 is identified in step 320 of "identifying the object(s) in the image". Although shown as a single object in this example, one of ordinary skill in the art will understand that multiple objects can be identified by the controller 26 within a single image.

[0048] Once the object(s) are identified, the controller 26 determines the distance between the object 220 and the vehicle component 210 in step 330 of "determining the distance". In a system that uses image-based analysis or partially image-based analysis, in addition to identifying the presence of the object(s), the ECU determines the edge 214 of the vehicle component 210 and determines the distance from the edge 214 of the vehicle component 210 to the object 220. In a system that uses radar, LIDAR, infrared, or ultrasonic sensors, the distance is determined via the sensors and the distance data is provided to the controller 26. In one particular example, a combination of a long-range radar sensor and a short-range ultrasonic sensor is utilized. In this example, the long-range radar sensor provides accuracy on the scale of feet (0.3 meters), and the ultrasonic sensor provides short-range accuracy on the scale of inches (25.4 millimeters). During operation, the long-range radar sensor is utilized until the detected object is within the range of the ultrasonic sensor and the system switches to ultrasonic radar system measurements.

[0049] Once the distance is determined, in the "Modify Image" step 340, the image is modified by overlaying the human-machine interface 230 on the image, creating a new image. The human-machine interface includes specific information identifying the distance between the vehicle component 210 and the object 220. The specific distance can be conveyed by any combination of color gradients, bar graphs, line distances, and numerical indicators. As an example, the specific information is conveyed using a numerical countdown display in one example, and the countdown indicates the distance between the vehicle component 210 and the object. In another example, the specific information is conveyed using a colored overlay, where each color indicates a specific distance (e.g., green indicates 5 meters, yellow indicates 3 meters, and red indicates 1 meter). In yet another example, the specific information is conveyed using a shaded area of a geometric shape that expands / contracts, and the size of the geometric shape on the screen directly corresponds to the distance. In yet another example, in parallel with or independent of these examples, other systems for conveying the specific distance may be utilized. Thereafter, the modified image is displayed to the vehicle operator in the "Display Modified Image" step 350.

[0050] Referring to all of FIGS. 1 through 5, in some examples, it is understood that continuous operation of the human-machine interface may not be desirable. In such cases, the controller 26 is configured to omit the overlay of the human-machine interface until a trigger condition is met. In some examples, the trigger condition can be the activation of one or more driving assistance systems incorporated in the controller 26. In other examples, the trigger condition can be the detection of an object within a predefined distance from the vehicle. In one example, the predefined distance is 30 meters. In yet another example, the trigger condition can be a combination of the vehicle entering a driving assistance mode and detecting an object within the predefined distance.

[0051] The above-described system shown in FIGS. 1 through 6 provides a human machine interface directly integrated into an image provided to a vehicle operator and includes a specific identification of the distance between the vehicle and a detected object, thereby providing the operator with more reliable and accurate assistance in proceeding past or otherwise interacting with an object that is not directly present in the driver's field of view. Further, in a system where the human machine interface is directly integrated into a mirror replacement system, the information is presented to the driver as a single image in a single location, improving clarity and usability.

[0052] It is further understood that any of the above concepts may be used alone or in combination with any or all of the other above concepts. While one embodiment of the invention has been disclosed, those of ordinary skill in the art will recognize that certain changes fall within the scope of the invention. Therefore, the following claims should be considered to determine the true scope and content of the invention.

Claims

1. A method of operating a vehicle camera system, comprising: Receiving a first image from at least one video camera attached to a side portion of a vehicle cab; Identifying an object in the first image; Determining a distance between a vehicle component and the identified object, at least partially based on image analysis of the first image, wherein the vehicle component is an ego portion capable of moving independently from the vehicle; Generating a modified mirror replacement image by modifying the first image by incorporating a human machine interface (HMI) within the first image, the human machine interface including a display configured to show the object and a part of the vehicle component and to convey a numerical distance between the object and the vehicle component; Displaying the modified image to a vehicle operator and including the method.

2. The method according to claim 1, wherein the at least one video camera includes two video cameras, the two video cameras are arranged backward in a camera housing attached to a side portion of the vehicle cab, and the first image is a combination of each image from the two video cameras.

3. The method according to claim 1, wherein the step of determining the distance between the object and the vehicle component is at least partially based on at least one physical sensor measurement from a physical sensor on the vehicle component.

4. The method according to claim 3, wherein the at least one physical sensor measurement includes at least one of a radar sensor measurement, a LIDAR sensor measurement, an infrared sensor measurement, a time-of-flight sensor, and an ultrasonic sensor measurement.

5. The method according to claim 3, wherein the at least one physical sensor measurement includes a radar sensor measurement and an ultrasonic measurement.

6. The step of generating a modified mirror replacement image by modifying the first image by incorporating a human machine interface (HMI) within the first image, and the step of displaying the modified mirror replacement image to the vehicle operator are performed in response to the determined distance being less than a predefined threshold distance. The method according to claim 1.

7. The method according to claim 6, wherein the threshold distance is about 30 m.

8. The method according to claim 6, wherein the threshold distance is an activation distance for at least one automated driving assistance function.

9. The method according to claim 5, wherein the radar sensor measurements provide accuracy on a feet scale and the ultrasonic measurements provide accuracy on an inches scale.

10. The method according to claim 1, wherein the vehicle component is a trailer.

11. The method according to claim 1, wherein the human-machine interface includes at least one of a numerical indicator, a multi-color overlay, and a bar graph.

12. The method according to claim 11, wherein the human-machine interface includes a combination of at least two of the numerical indicator, the multi-color overlay, and the bar graph.

13. The method according to claim 11, wherein the human-machine interface includes an object indicator that identifies an object identified in the displayed image.

14. The method according to claim 1, wherein changing the first image includes generating a mirror replacement image by combining a plurality of images generated from different vehicle cameras.

15. further comprising the step of overlaying at least one distance line on the first image, wherein the at least one distance line is calibrated at a predefined distance using the distance displayed on the human-machine interface. The method according to claim 1.

16. In a vehicle system comprising at least one outward-facing camera mounted on a side of a vehicle cab, a controller including an input connected to an output of the at least one outward-facing camera, and an inward-facing display connected to the controller and mounted within the vehicle cab. The controller is configured to identify an object in a first image received by the controller from the at least one outward-facing camera, determine a distance between a vehicle component and the identified object based at least in part on image analysis of the first image, where the vehicle component is an ego part capable of moving independently from the vehicle, generate a mirror replacement image modified by incorporating a human machine interface (HMI) into the first image, and output the modified mirror replacement image to the display, and includes a memory storing instructions configured to cause the controller to perform the above operations. The vehicle system, wherein the human machine interface includes a display configured to communicate a numerical distance between the object and the vehicle component.

17. The vehicle system according to claim 16, wherein the controller is connected to a proximity sensor, and the proximity sensor is configured to determine a distance between the proximity sensor and a detected object.

18. The vehicle system according to claim 17, wherein the proximity sensor includes at least one of a radar sensor, a LIDAR sensor, an infrared sensor, a time-of-flight sensor, and an ultrasonic sensor.

19. The vehicle system according to claim 18, wherein the proximity sensor includes a radar sensor and an ultrasonic sensor.

20. The vehicle system according to claim 16, wherein the controller includes an object detection module based at least in part on an image.

21. The vehicle system according to claim 20, wherein the object detection module based at least in part on an image includes an object detection function based on an auxiliary sensor.

22. The vehicle system according to claim 16, wherein the vehicle component is a trailer.

Images