Systems and methods for assisting a driver in operating a vehicle

Abstract

To provide a drive assist system which facilitates positioning of a vehicle to a specific object during the retraction maneuvering of the vehicle.SOLUTION: Systems and methods for assisting drivers in maneuvering a vehicle are disclosed. A plurality of image sensors about a vehicle capture image data. At least one processor is programmed to display one or more types of overhead views on a vehicle display based on the image data. For example, an environment-fixed overhead composite image can be displayed on the vehicle display. In the environment-fixed overhead composite image, an environment about the vehicle is displayed and spatially fixed as the vehicle is maneuvered, and a graphical image representing the vehicle moves within the environment-fixed overhead composite image as the vehicle is maneuvered. This type of overhead view can be shown when, for example, distance between the vehicle and a target object is below a threshold.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a system and method for assisting a driver in operating a vehicle by using an environment-fixed overhead composite image.

Background Art

[0002] Many automobiles are equipped with a rear camera. This rear camera can assist the driver, for example, when the driver shifts the vehicle into reverse gear or when trying to park. Specifically, an image of the environment behind the vehicle can be displayed on the vehicle display. Some automobiles have multiple cameras at various locations around the vehicle. These cameras can be used, among other things, to create an overhead view of the environment around the vehicle. This overhead view can also be displayed on the vehicle display to assist the driver in operating the vehicle.

Summary of the Invention

[0003] In one embodiment, a system for assisting a driver in operating a vehicle is provided. The system includes a plurality of image sensors configured to be attached at various locations around the vehicle, at least some of the image sensors being cameras, and the image sensors being configured to output image data. The system also includes a vehicle display and at least one processor. The at least one processor is configured to (i) receive image data from the cameras, and (ii) use a first image sensor to determine a target object outside the vehicle and the first image sensor a trailerdetermine the distance thereto, (iii) generate an environment-fixed overhead composite image representing a first overhead view of the environment around the vehicle on a vehicle display, such that when the vehicle is maneuvered, the environment displayed on the vehicle display is spatially fixed and a graphic image representing the vehicle is configured to move within the environment-fixed overhead composite image when the vehicle is maneuvered, (iv) generate a vehicle-fixed overhead composite image representing a second overhead view of the environment around the vehicle on the vehicle display, such that when the vehicle is maneuvered, the environment displayed on the vehicle display is spatially movable and a graphic image representing the vehicle is fixed within the vehicle-fixed overhead composite image when the vehicle is maneuvered, and (v) be programmed to switch the vehicle display between the display of the vehicle-fixed overhead composite image and the display of the environment-fixed overhead composite image based on the distance.

[0004] In another embodiment, a system for assisting a driver in maneuvering a vehicle includes a plurality of image sensors configured to be attached at various locations around the vehicle, at least some of the image sensors being cameras, the image sensors being configured to output image data. The system also includes a vehicle display and at least one processor. The at least one processor is configured to (i) receive image data from the cameras, (ii) utilize a first image sensor to determine a distance between the first image sensor and a target object external to the vehicle, and (iii) in response to the distance being less than a threshold, utilize the image data to generate an environment-fixed overhead composite image representing an overhead view of the environment around the vehicle on the vehicle display, such that when the vehicle is maneuvered, the environment displayed within the environment-fixed overhead composite image is spatially fixed and a graphic image representing the vehicle is configured to move within the environment-fixed overhead composite image when the vehicle is maneuvered.

[0005] In one embodiment, a method for assisting a driver in maneuvering a vehicle is provided. The method includes receiving image data from a plurality of image sensors attached around the vehicle, at least some of the image sensors being cameras. The method includes using one of the image sensors and a target object external to the vehicle a trailerDetermining a distance to [object], and in response to the distance being below a threshold, also generating, based on the image data, an environmental fixed overhead composite image representing an overhead view of the environment around the vehicle on a vehicle display, wherein when the vehicle is maneuvered, the environment is spatially fixed within the environmental fixed overhead composite image, and a graphic image representing the vehicle is configured to move within the environmental fixed overhead composite image when the vehicle is maneuvered.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0007] Here, embodiments of the present disclosure are described. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments can take various alternative forms. The figures are not necessarily to scale. Some features may be exaggerated or minimized to show details of a particular component. Thus, the specific structural and functional details disclosed herein should not be construed as limiting, but rather should be interpreted as a representative basis for teaching those skilled in the art to use the embodiments in various ways. As will be understood by those skilled in the art, the various features illustrated and described with reference to any one of the figures can be combined with features illustrated in one or more other figures to generate embodiments that are not explicitly illustrated or described. Combinations of the illustrated features provide representative embodiments for typical applications. However, various combinations and modifications of the features consistent with the teachings of the present disclosure may be desirable for a particular application or implementation.

[0008] In some automobiles (such as passenger cars, vans, trucks, SUVs, etc.), it is common to install a rear camera. This rear camera can assist the driver when the driver shifts the vehicle into reverse gear or when trying to park. Specifically, an image of the environment behind the vehicle can be displayed on the vehicle display (e.g., an infotainment screen or a center console display). Some automobiles also commonly have multiple cameras around the vehicle that can collectively create an aerial view (bird's-eye view) of the vehicle on the vehicle display. Images from multiple cameras are combined to generate a 360-degree view of the environment around the vehicle, as if a virtual camera were placed above the vehicle, and are displayed on the vehicle display. An image representing the top view of the vehicle can be overlaid on the view displayed on the vehicle display. In this document, this is called a vehicle-fixed overhead composite image because the image representing the vehicle remains at a fixed position on the vehicle display when the driver operates the vehicle, and the environment around the vehicle detected by the camera moves when the driver operates the vehicle. These camera systems can assist the user in reversing the vehicle or parking the vehicle in a narrow space.

[0009] Even when using these camera systems, for many drivers, some driving operations still remain difficult. For example, backing the vehicle in a straight line with the trailer in order to attach the vehicle to the trailer hitch can still be very difficult for many drivers. Even when using the vehicle-fixed overhead composite image displayed on the vehicle display, it can be difficult to determine the alignment between the vehicle and the trailer due to the moving environment during vehicle operation.

[0010] Accordingly, according to various embodiments described herein, a system for assisting a driver in maneuvering a vehicle includes an environment-fixed overhead composite image displayed on a vehicle display. In this system, cameras (and other optional image sensors) are used to create an environment-fixed overhead composite image in which the environment around the vehicle detected by the cameras remains stationary while the vehicle is being maneuvered by the driver, and an image representing a top view of the vehicle moves as the vehicle moves. By keeping the environment (such as the road) stationary on the vehicle display while the driver is maneuvering the vehicle, the driver can more accurately grasp how the alignment of the vehicle with respect to a specific object (such as a trailer) changes during the maneuvering.

[0011] FIG. 1 shows a system 10 for assisting a driver in maneuvering a vehicle according to one embodiment. Generally, system 10 includes a plurality of image sensors 12, a plurality of vehicle state sensors 14, an image processing controller 16, and a vehicle display 18. Image sensors 12 and vehicle state sensors 14 are electrically connected to and input data to image processing controller 16. Image processing controller 16 processes data from these sensors and causes vehicle display 18 to display views and / or information based on the inputs from sensors 12, 14.

[0012] Image sensors 12 can include one or more different types of sensors and can be configured to detect the environment around the vehicle. Accordingly, image sensors 12 may sometimes be referred to as environment sensors. Image sensors 12 can include one or more cameras 20, one or more light detection and ranging (LIDAR) sensors 22, and / or one or more radio detection and ranging (RADAR) sensors 24. Although not shown, in other embodiments, image sensors 12 can include one or more sound navigation and ranging (SONAR) sensors, or other types of environment sensors configured to detect the environment outside the vehicle. Image sensors 12 can be attached at various locations on the vehicle, such as fenders, fascia, side mirrors, roof, bumper, panel, etc.

[0013] The camera 20 is configured to capture an image based on external light reflected from an object. Data representing the image, called image data, can be transmitted to an image processing controller 16, which will be further described below. The image captured by the camera may be a single-frame image or a video (e.g., multiple frames).

[0014] Each of the LIDAR sensors 22 can include a transmitter and a receiver. The transmitter irradiates a light pulse toward the environment around the vehicle, and the receiver detects the reflection of the irradiated light pulse. Based on the time it takes for the irradiated light pulse to be reflected and return to the receiver, the LIDAR sensor 22 and the associated controller can map a three-dimensional (3D) view of the environment. Using the reflected light wave, the environment can be visualized three-dimensionally or processed into a point cloud. The LIDAR sensor 22 can rotate about an axis and scan the environment by detecting the reflected light pulse. Alternatively, the LIDAR sensor 22 can be fixed and may have a defined field of view (FOV) defined by the pointing direction of the LIDAR sensor 22. An actuator may be provided to rotate the LIDAR sensor 22 to adjust the FOV.

[0015] The RADAR sensor 24 can be based on, for example, a frequency band of 24 GHz or 77 GHz. In an example suitable for the disclosure provided herein for vehicle parking or reverse maneuvers, the RADAR sensor 24 can be a short-range radar (SRR) sensor. Compared with a long-range radar (LRR) or medium-range radar (MRR) sensor, the SRR sensor has a smaller antenna size and less interference. However, in practice, the RADAR sensor 24 may actually incorporate an MRR or LRR sensor. The RADAR sensor 24 is configured to radiate radio waves toward the environment around the vehicle, and the RADAR receiver detects the reflection of the radiated radio waves. Based on the reflected radio waves, the RADAR sensor 24 and the associated controller can map a three-dimensional (3D) view of the environment.

[0016] Regardless of the type of image sensor 12 being utilized, the signal output therefrom and transmitted to the image processing controller 16 can be referred to as image data. The image data represents the environment detected outside the vehicle, detected by respective cameras 20, LIDAR sensors 22, and / or RADAR sensors 24.

[0017] The vehicle state sensor 14 can include a wheel speed sensor 26, a shift position sensor 28, and a steering angle sensor 30. In short, the wheel speed sensor 26 detects the rotational speed (wheel speed) of the driving wheels, the shift position sensor 28 detects the shift position of the transmission, and the steering angle sensor 30 detects the steering angle (e.g., the direction and degree of the steering angle input by the driver) or the actual steering angle (the turning angle of the wheels) corresponding to the steering angle.

[0018] The shift position sensor 28 is configured to detect in which gear or operating state the vehicle is. For example, the shift position sensor 28 can be a PRNDL sensor configured to detect that the vehicle is currently shifted into Park, Reverse, Neutral, Drive, or Low gear. This can be implemented using a mechanical switch, a Hall effect sensor, or other mechanisms known in the art. As will be further described below, the shift position sensor 28 can start the disclosed system 10, and the system 10 is started, for example, when the vehicle is shifted into Reverse, and causes an overhead view (and optionally a rear view from the rear camera) to be displayed on the vehicle display 18.

[0019] The steering angle sensor 30 is configured to detect the current angle of the steering wheel within the vehicle's cabin. The steering angle sensor 30 can be disposed within the steering column. The steering angle sensor 30 can include, for example, an analog sensor that uses a voltage difference to determine information regarding the angle and turning direction, and a digital sensor that uses light from a light emitting diode (LED) to measure the angle of the steering input.

[0020] The image processing controller 16 may be an electronic control unit (ECU) or may include it. The image processing controller 16 may be an electronic control circuit configured to receive data from the image sensor 12 and the vehicle state sensor 14, process the data, and control the vehicle display 18. The structure shown in FIG. 1 within the image processing controller 16 is merely illustrative, and other circuits or configurations may be provided. In the illustrated embodiment, the image processing controller 16 includes an image data processor 32, a display controller 34, and a memory 36. Generally, the image processing controller 16 may include processor hardware (shared, dedicated, or grouped), such as an image data processor 32 and / or a display controller 34 that executes code, and memory hardware (shared, dedicated, or grouped), such as a memory 36 that stores the code executed by the processor hardware. The code is configured to provide the functions of the controllers and systems described herein.

[0021] The image data processor 32 may include one or more devices selected from a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that operates on signals (analog or digital) based on computer-executable instructions resident in the memory 36.

[0022] Memory 36 can include a single memory device or multiple memory devices, including, but not limited to, random access memory (“RAM”), volatile memory, non-volatile memory, static random access memory (“SRAM”), dynamic random access memory (“DRAM”), flash memory, cache memory, or any other device capable of storing information. Non-volatile storage can include one or more persistent data storage devices, such as a hard drive, optical drive, tape drive, non-volatile solid state device, or any other device capable of permanently storing information. The image data processor 32 may be configured to read and execute computer-executable instructions that embody one or more software programs resident in non-volatile storage into the memory. Programs resident in non-volatile storage may include or be part of an operating system or an application, and may be compiled or interpreted from computer programs created using various programming languages and / or technologies, including, but not limited to, Java, C, C++, C#, Objective C, Fortran, Pascal, Java Script, Python, Perl, and PL / SQL, either alone or in combination. The computer-executable instructions of the program, when executed by the image data processor 32, can be configured to create the environment-fixed overhead composite image described herein and cause such an image to be displayed on the vehicle display 18 by the display controller 34.

[0023] Via image processing controller 16, implementations of the subject matter and operations described in this specification can be implemented in digital electronic circuitry, or in computer software embodied on a tangible medium, firmware, or hardware that includes the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs, e.g., as one or more modules of computer program instructions encoded on one or more computer storage media, for execution by, or to control the operation of, a data processing apparatus. A computer storage media can be, or can include, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. A computer storage media can also be, or can include, one or more discrete components or media (e.g., multiple CDs, disks, or other storage devices), or can be included in them. A computer storage media can be tangible and non-transitory.

[0024] Via the image processing controller 16, a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled languages, interpreted languages, declarative languages, and procedural languages, and the computer program can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computer environment. The computer program may or may not correspond to a file in the file system. The program can be stored in the memory 36 in a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program, or multiple cooperating files (e.g., multiple files that store one or more modules, libraries, subprograms, or parts of the code). The computer program can be deployed to be executed on one computer or on computers located at one site or distributed across multiple sites and interconnected by a communication network. In other words, the image processing controller 16 can incorporate not only a single processor 32 and / or a single controller 34 but also a network of processors or controllers that communicate with each other and, for example, share tasks.

[0025] The processes and logical flows described in this specification can be executed by one or more programmable processors (e.g., the image data processor 32) that execute one or more computer programs, process input data (e.g., from the image sensor 12 and / or the vehicle state sensor 14), and perform actions by generating outputs (e.g., causing the display controller 34 to display an environmental fixed overhead composite image on the vehicle display 18). The processes and logical flows can also be executed by special purpose logic circuitry, such as a field programmable gate array (“FPGA”) or an application specific integrated circuit (“ASIC”), and the apparatus can be implemented as such circuitry. Such special purpose circuitry can be referred to as a computer processor even though it is not a general purpose processor.

[0026] Unless otherwise indicated herein, the structures described above as part of the image processing controller 16 can more generally be referred to as a “processor” or a “controller”. For example, a processor programmed to receive image data from the image sensor 12 and cause a corresponding display on the vehicle display 18 can incorporate one or more of the above suitable structures.

[0027] The vehicle display 18 can be installed inside the vehicle cabin, such as within the center console, or as part of the in-vehicle infotainment system. The vehicle display 18 can be a human-machine interface (HMI) realized as a touch screen, a liquid crystal display (LDC), a light-emitting diode (LED) screen, etc. Under the control of the image processing controller 16, more specifically, by the display controller 34 implemented therein, the vehicle display 18 executes various types of displays to visually provide data to the driver. For example, the image captured from the camera 20 can be displayed on the vehicle display 18 as further described herein. The vehicle display 18 can also be utilized by the driver to perform other in-vehicle tasks such as climate control, entertainment control (e.g., volume, radio tuning, etc.), seat movement, etc. The vehicle display 18 can also be a mobile device (e.g., a smartphone) communicably connected to the vehicle and configured to control the operation of the vehicle. For example, remote parking or remote operation of the vehicle can be developed so that the driver or operator can control the movement of the vehicle using such a mobile device, in which case the mobile device can be regarded as the vehicle display 18.

[0028] Figure 2 shows an embodiment of the system 10 utilized to display one or more camera views of the environment around the vehicle during reverse maneuvering on the vehicle display 18. In this embodiment, both a vehicle-fixed overhead composite image 40 on the left side and a rear view image 42 captured by a single rear camera (e.g., one of the cameras 20) on the right side are shown. The images 40, 42 can be arranged and simultaneously displayed on the vehicle display 18. The images 40, 42 can be displayed on the vehicle display 18 in response to the vehicle being shifted into Reverse (e.g., signaled by the shift position sensor 28), the driver pressing the parking assistance button, the image sensor 12 and the vehicle state sensor 14 indicating a high likelihood of the vehicle being parked due to low speed (e.g., less than 3 miles per hour), and / or an object being detected within a threshold distance.

[0029] The vehicle-fixed overhead composite image 40 can be created by stitching together images from various image sensors 12. This view can be referred to as an aerial view, an overhead view, a 360-degree view, a top-down view, etc. It shows a simulated view from above the vehicle as if a virtual camera were placed directly above the vehicle. A graphic image 44 representing a simulated top view of the vehicle can be overlaid on the stitched-together camera images. In the vehicle-fixed overhead composite image 40, the graphic image 44 of the vehicle is always fixed, while the environment around the vehicle moves as the image sensors 12 mounted on the vehicle move with the movement of the vehicle.

[0030] The vehicle-fixed overhead composite image 40 may include guide lines 46 and / or the rear view image 42 may include guide lines 48. These guide lines 46, 48 can be overlaid on the images generated by their respective cameras. The guide lines 46, 48 are useful for a driver to predict where the vehicle will be located when the vehicle is reversing along the current route. The guide lines 46, 48 can be commanded to be displayed on the vehicle display 18 by the display controller 34 based on the input from the steering angle sensor 30. For example, when the steering angle sensor 30 indicates a change in the current steering angle, the guide lines 46, 48 can be bent or redirected accordingly.

[0031] As described above, even when using the vehicle-fixed overhead composite image 40 and the rear view image 42, for some drivers, the task of backing up can be cumbersome and difficult. For example, backing the vehicle into a position for attachment to a trailer may still remain difficult. The rear view image 42 of FIG. 2 shows an example of a trailer 50 with a coupler 52, and also shows the trailer hitch 54 of the reversing vehicle. The coupler 52 may be a ball coupler or pocket known in the art, and the vehicle's trailer hitch 54 may be a ball mount sized and configured to be received within the coupler 52 by a ball and socket connection. Backing the vehicle to couple with the trailer 50 via a ball and socket connection may not be easy because very precise alignment between the coupler 52 and the trailer hitch 54 may be required.

[0032] Therefore, instead of the vehicle-fixed overhead composite image, an environment-fixed overhead composite image may rather be displayed on the vehicle display 18. An example of the vehicle display 18 showing an embodiment of the environment-fixed overhead composite image 60 is shown in FIGS. 3 and 4, where the graphic image 44 representing a top view of the vehicle is movable within a frame, and the environment around the vehicle (e.g., road, trailer, etc.) is fixed as the vehicle moves.

[0033] FIG. 3 shows an embodiment of a system 10 utilized to display one or more camera views of the environment around a vehicle during a backing maneuver on the vehicle display 18. In this embodiment, as shown in FIG. 2, on the left side, an environment-fixed overhead composite image 60 according to one embodiment is shown, and on the right side, a corresponding rear view image 42 captured by a single rear camera (e.g., one of cameras 20) is shown. The images 60, 42 may be arranged and simultaneously displayed on the vehicle display 18. Also, as in FIG. 2, the environment-fixed overhead composite image 60 may include guide lines 62, and the rear view image 42 may include guide lines 48.

[0034] As described herein, within the environment-fixed overhead composite image 60 shown on the vehicle display 18, the graphic image 44 representing the top view of the vehicle is movable, and the environment around the vehicle (such as the road and the trailer 50) is fixed. This is shown in FIG. 4, which is a time progression diagram of the environment-fixed overhead composite image when the driver reverses the vehicle. The boundary line of the environment-fixed overhead composite image 60 indicates the field of view within the environment-fixed overhead composite image 60 shown on the vehicle display 18. In other words, the items shown within the boundary line can be seen within the environment-fixed overhead composite image 60 on the vehicle display 18. As can be seen, when the driver of the vehicle reverses the vehicle towards the trailer 50, the graphic image 44 also moves within the environment-fixed overhead composite image 60, and the trailer 50 remains fixed. This can help the driver better understand the positional relationship when the vehicle approaches the trailer 50.

[0035] The system 10 can not only stitch together images from various cameras 20 to create the environment-fixed overhead composite image 60, but the system 10 is also configured to determine the size and orientation of the display area of the environment-fixed overhead composite image 60. In other words, since the environment is fixed when the vehicle is reversing, the relative size and orientation of the image 60 displayed on the vehicle display 18 should be carefully calculated. For example, if a target object is within the field of view of the vehicle's camera 20, the target object may be fixed at the bottommost part of the environment-fixed overhead composite image 60. In this example, the target object is the detected trailer, such as the trailer 50. As described, various image sensors 12 can detect the presence of a target object (such as the trailer 50), the distance between the vehicle and the target object, and the relative position (such as the angle) between the vehicle and the target object. The image processing controller 16 is configured to determine, based on this data, the fixed area of the visible environment around the vehicle to be displayed within the environment-fixed overhead composite image 60.

[0036] Figure 5 shows a schematic overhead view of a vehicle 66 being operated and a trailer 50. The vehicle 66 has several image sensors 12 including one camera 20 and a pair of RADAR sensors 24. The camera 20 and associated processor or controller are configured to detect the presence of target objects. In one embodiment, the image processing controller 16 comprises image detection software or a machine learning model configured to detect and classify specific objects. The image detection software or machine learning model is a commercially available product such as REKOGNITION by AMAZON, OPENCV, CLARIFAI, TORCH, etc., and can be pre-trained to recognize specific target objects such as trailers within the field of view of the camera 20 intended to be attached to the vehicle 66.

[0037] When a target object is identified, the camera 20 and associated processor or controller are also configured to detect the distance between the camera 20 and the identified target object, e.g., the trailer 50. However, additional sensors such as the RADAR sensors 24 may be more accurate for such measurements. The RADAR sensors 24 and associated processor or controller may be more suitable for detecting the distance to the identified target object and the relative angle to the identified target object, but the RADAR sensors 24 may not be able to detect the type of object (e.g., perform the above image recognition function). Therefore, the system 10 is configured to combine the image data from the camera 20 and the RADAR sensors 24 to determine the distance and relative angle of the target object. For example, as shown in FIG. 5, the first boundary 70 represents a portion of the target object visible by the camera 20, and the second boundary 72 represents a portion of the target object visible by the RADAR sensors 24. The third boundary 74 represents the combined output of the camera 20 and the RADAR sensors 24.

[0038] The camera 20 and the RADAR sensor 24, and optionally data from the LIDAR sensor 22 etc. can be fused to create a combined output within the boundary 74. Sensor fusion can be performed according to any number of known techniques. For example, a Kalman filter or a Bayesian filter can be used to merge data from various image sensors 12 to improve the detection distance to the target object and the accuracy of the orientation of the target object. Sensor fusion can be carried out in accordance with the teachings of U.S. Patent No. 9,390,624, which is hereby incorporated by reference in its entirety. Considering these teachings, the system 10 generally combines the data output by the RADAR sensor 24 with the data output by the (multiple) cameras 20 to determine a more accurate position (e.g., orientation, angle) of the target object (e.g., trailer 50), and the distance from the vehicle 66 to the target object. FIG. 6 shows an example of such calculated parameters. The angle (α) represents the relative offset angle between the vehicle 66 and the target object (e.g., trailer 50). The fused data from the camera 20 and the RADAR sensor 24 can determine at what angle (α) the target object is relative to the vehicle. For example, the camera 20 and the sensor 24 are fixed to the vehicle 66 extending along a known axis 67 fixed relative to the vehicle, the camera 20 and the sensor 24 can detect the relative central axis of the target object, and the system 10 can determine the angle (α) between the known axis 67 and the determined central axis of the target object. Similarly, the distance (D) from the vehicle 66 to the target object can be determined using the fused data from various image sensors 12.

[0039] Based on the determined position (e.g., orientation, angle (α)) and / or the distance (D) from the vehicle 66 to the target object (e.g., trailer 50), the system 10 can determine the size and position of the environment-fixed overhead composite image 60. FIG. 7 provides a visual illustration of the system 10 that creates the environment-fixed overhead composite image 60 from the cameras 20 used to create the vehicle-fixed overhead composite image 40. Specifically, each camera 20 has its own respective field of view (as indicated by the dashed lines 76 emanating from each camera 20). In one embodiment, the system creates the vehicle-fixed overhead composite image 40 as described herein and then adjusts the size, resolution, and orientation of the image to create the environment-fixed overhead composite image 60. When the angle (α) and the distance (D) are determined, the field of view of the environment-fixed overhead composite image 60 can be set and fixed such that the target object (e.g., trailer 50) appears at the lower center of the image as shown on the vehicle display 18. (See, for example, FIGS. 3 and 4). The width (W) and height (H) of the field of view shown in the environment-fixed overhead composite image 60 can be adjusted such that at least a portion of the vehicle 66 (i.e., the graphic image 44 representing the vehicle) is visible within the environment-fixed overhead composite image 60 displayed on the vehicle display 18.

[0040] In one embodiment, the environment-fixed overhead composite image 60 is a resized, rotated, and repositioned form of the vehicle-fixed overhead composite image 40. Specifically, the vehicle-fixed overhead composite image 40 is determined based on the above teachings and then may be rotated or translated as a function of the angle (α) (as represented by the arrow 78) such that the target object appears at the lower part of the image (e.g., the environment-fixed overhead composite image 60) facing straight ahead. The width (W) and height (H) can be adjusted such that at least a portion (or only a portion) of the graphic image 44 is visible to the driver within the environment-fixed overhead composite image 60.

[0041] Once the area of the environment-fixed overhead composite image 60 is determined, it can be set and fixed throughout the operation of the vehicle 66. Therefore, when the vehicle 66 is operated (e.g., reversed), the environment visible within the environment-fixed overhead composite image 60 remains spatially fixed. This is because, even when the vehicle 66 is operated, the camera 20 mounted on the vehicle 66 moves with respect to the environment as the vehicle 66 moves, so that the target object remains in a fixed position on the vehicle display 18, which may involve certain processing of the image data (e.g., resizing and readjusting the vehicle-fixed overhead composite image 40).

[0042] The image processing controller 16 can also switch the vehicle display 18 between the display of the vehicle-fixed overhead composite image 40 and the display of the environment-fixed overhead composite image 60. For example, when one or more image sensors 12 (or the fused output of various image sensors 12) determine that the distance (D) between one of the image sensors (or the vehicle 66) and the target object exceeds a threshold distance, the image processing controller 16 can cause the vehicle-fixed overhead composite image 40 to be displayed on the vehicle display 18. When the determined distance (D) is less than the threshold distance, the image processing controller 16 can cause the environment-fixed overhead composite image 60 to be displayed on the vehicle display 18. The switching between the display of the vehicle-fixed overhead composite image 40 and the display of the environment-fixed overhead composite image 60 may occur while the driver is operating the vehicle 66. In another embodiment, when the vehicle display 18 is displaying either the vehicle-fixed overhead composite image 40 or the environment-fixed overhead composite image 60, that image is maintained on the vehicle display 18 until the shift position of the vehicle 66 is no longer in Reverse or, alternatively, until the system is terminated.

[0043] FIG. 8 shows an algorithm 100 that can be implemented, for example, by the image processing controller 16. In an embodiment, instructions for executing such an algorithm are stored in the memory 36 and can be accessed for execution by at least one processor (e.g., the image data processor 32 and / or the display controller 34, or another processor). The algorithm 100 begins at 102. The algorithm 100 can be initiated in response to, for example, the vehicle being shifted into Reverse as indicated by the shift position sensor 28, or being manually input by the driver as described above.

[0044] At 104, one or more image sensors 12 (e.g., the camera 20) and associated processors (e.g., the image data processor 32) detect that a target object, such as a hitch or trailer, is within the field of view. This can be performed by the above-described image recognition software. At 106, one or more processors analyze the image data from the image sensor 12, for example, by the above-described method, to determine the distance (D) to the detected target object. In one embodiment, one or more RADAR sensors 24 determine the distance to the target object. In one embodiment, the camera 20 determines the distance to the target object, and the determined distance is modified or enhanced based on the distance read from one or more RADAR sensors 24.

[0045] At 108, the image processing controller 16 or its associated processor determines the position of the target object relative to the vehicle. For example, an angle (α) representing the angle of the target object relative to the vehicle can be determined as described above.

[0046] At 110, the image processing controller 16 compares the determined distance (D) with a threshold value. The threshold value may be, for example, 10 feet. This threshold value can be set by the driver of the vehicle (e.g., via the HMI of the vehicle display 18). Alternatively, this threshold value can be set by the manufacturer of the vehicle 66. If the detected distance (D) exceeds the threshold value, the algorithm proceeds to 112 and displays the vehicle-fixed overhead composite image 40 on the vehicle display 18. If the detected distance (D) does not exceed the threshold value, the algorithm proceeds to 114 and displays the environment-fixed overhead composite image 60 on the vehicle display 18.

[0047] Next, the algorithm returns to the start 102 at 116. This enables a continuous comparison of the distance with the threshold value at 110, and the vehicle display 18 can switch between the vehicle-fixed overhead composite image 40 and the environment-fixed overhead composite image 60 during the maneuvering event.

[0048] Exemplary embodiments have been described above, but these embodiments are not intended to describe all possible forms encompassed by the claims. The words used in this specification are words of description rather than limitation, and it will be understood that various changes can be made without departing from the spirit and scope of the disclosure. As previously mentioned, the features of the various embodiments can be combined to form further embodiments of the invention that may not be explicitly described or illustrated. The various embodiments may provide advantages over other embodiments or prior art implementations with respect to one or more desired characteristics or may be described as preferred, but those skilled in the art will recognize that one or more features or characteristics can be compromised to achieve the overall desired system attributes that depend on a particular application or implementation. These attributes include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. Thus, to the extent that any embodiment is described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics, these embodiments are not outside the scope of the disclosure and may be desirable for a particular application.

Description of the Reference Numerals

[0049] 10: System, 12: Image sensor, 14: Vehicle state sensor, 16: Image processing controller, 18: Vehicle display, 20: Camera, 22: LIDAR sensor, 24: RADAR sensor, 26: Wheel speed sensor, 28: Shift position sensor, 30: Steering angle sensor, 32: Image data processor, 34: Display controller, 36: Memory

Claims

1. A system for assisting a driver in operating a vehicle, comprising: a plurality of image sensors configured to be attached to various locations around the vehicle; at least some of the image sensors are cameras, and the image sensors are configured to output image data; a vehicle display; and at least one processor that receives image data from the cameras, uses a first image sensor to determine a distance between the first image sensor and a trailer which is a target object outside the vehicle, generates an environment-fixed overhead composite image representing a first overhead view of the environment around the vehicle on the vehicle display, when the vehicle is being operated, the environment displayed on the vehicle display is spatially fixed, a graphic image representing the vehicle is configured to move within the environment-fixed overhead composite image when the vehicle is being operated, generates a vehicle-fixed overhead composite image representing a second overhead view of the environment around the vehicle on the vehicle display, when the vehicle is being operated, the environment displayed on the vehicle display is spatially movable, a graphic image representing the vehicle is fixed within the vehicle-fixed overhead composite image when the vehicle is being operated, and is programmed to switch the vehicle display between the display of the vehicle-fixed overhead composite image and the display of the environment-fixed overhead composite image based on the distance.

2. The system according to claim 1, wherein at least one of the processors is further programmed to switch the vehicle display from the display of the vehicle-fixed overhead composite image to the display of the environment-fixed overhead composite image in response to the distance falling below a threshold.

3. The system according to claim 1, wherein at least one of the processors is further programmed to generate an environment-fixed overhead composite image or a vehicle-fixed overhead composite image in response to the vehicle reversing.

4. The system according to claim 1, wherein at least one of the processors is further programmed to determine a size of a display field of view of the environment-fixed overhead composite image based on the distance.

5. The system according to claim 4, wherein at least one of the processors is further programmed to determine a position of the target object based on the image data, and further determine a display field of view of the environment-fixed overhead composite image based on the position of the target object.

6. The system according to claim 5, wherein at least one of the processors is further programmed to determine a display field of view of the environment-fixed overhead composite image such that at least a part of the target object is visible at the lower part of the environment-fixed overhead composite image.

7. The system according to claim 5, wherein at least one of the processors is further programmed to determine a display field of view of the environment-fixed overhead composite image based on a relative angle of orientation of the target object with respect to the vehicle.

8. A system for assisting a driver in maneuvering a vehicle, a plurality of image sensors configured to be attached to various locations around the vehicle, at least some of the image sensors being cameras, the image sensors being configured to output image data, a vehicle display, and at least one processor that receives image data from the cameras, uses a first image sensor to determine a distance between the first image sensor and a trailer, which is a target object outside the vehicle, and in response to the distance being less than a threshold, uses the image data to generate an environment-fixed overhead composite image representing an overhead view of the environment around the vehicle on the vehicle display. A system in which the environment displayed within the environment-fixed overhead composite is spatially fixed when the vehicle is being maneuvered, and a graphic image representing the vehicle is configured to move within the environment-fixed overhead composite image when the vehicle is being maneuvered.

9. At least one of the processors is further programmed to, in response to the distance exceeding a threshold, use the image data to generate a vehicle-fixed overhead composite image representing a second overhead view of the environment around the vehicle on the vehicle display, wherein the environment displayed within the vehicle-fixed overhead composite image is spatially movable when the vehicle is being maneuvered, and the graphic image representing the vehicle is fixed within the vehicle-fixed overhead composite image when the vehicle is being maneuvered, according to the system of claim 8.

10. The system according to claim 1, wherein at least one of the processors is further programmed to generate an environment-fixed overhead composite image or a vehicle-fixed overhead composite image in response to the vehicle reversing.

11. At least one of the processors is further programmed to switch the vehicle display from displaying a vehicle-fixed overhead composite image to displaying an environment-fixed overhead composite image in response to the distance changing from a state where it exceeds a threshold to a state where it is less than the threshold, for the system according to claim 9.

12. At least one of the processors is further programmed to determine the size of the display field of view of the environment-fixed overhead composite image based on the distance, for the system according to claim 8.

13. At least one of the processors is further programmed to determine the position of the target object based on the image data and further determine the display field of view of the environment-fixed overhead composite image based on the position of the target object, for the system according to claim 12.

14. At least one of the processors is further programmed to determine the display field of view of the environment-fixed overhead composite image such that at least a part of the target object is visible at the lower part of the environment-fixed overhead composite image, for the system according to claim 13.

15. A method for assisting a driver in operating a vehicle, comprising: receiving image data from a plurality of image sensors attached around the vehicle; determining a distance between one of the image sensors and a trailer which is a target object outside the vehicle; and generating, in response to the distance falling below a threshold, an environment-fixed overhead composite image representing an overhead view of the environment around the vehicle on a vehicle display based on the image data, wherein at least some of the image sensors are cameras; a method in which the environment is spatially fixed within the environment-fixed overhead composite image and the graphic image representing the vehicle is configured to move within the environment-fixed overhead composite image when the vehicle is being operated.

16. further comprising generating, on the vehicle display, a vehicle-fixed overhead composite image representing a second overhead view of the environment around the vehicle; wherein the environment displayed on the vehicle display is spatially movable when the vehicle is being operated, and the graphic image representing the vehicle is fixed within the vehicle-fixed overhead composite image when the vehicle is being operated, for the method according to claim 15.

17. determining a distance between one of the image sensors and a target object outside the vehicle; and The method according to claim 16, further comprising switching the vehicle display from displaying a vehicle-fixed overhead composite image to displaying an environment-fixed overhead composite image in response to the distance falling below a threshold value. **Claim 18** Determining a distance between one of the image sensors and a target object outside the vehicle, and Determining a size of a display field of view of an environment-fixed overhead composite image based on the distance, the method according to claim 15, further comprising. **Claim 19** Determining a position of the target object based on the image data, and Further determining an orientation of a field of view of an environment-fixed overhead composite image based on the position of the target object, the method according to claim 18, further comprising.

Images