Peripheral monitoring device

Abstract

Peripheral monitoring device (100) with: a recording unit (48) configured to obtain a peripheral image acquired by an imaging unit (24, 24a to 24d) provided in a vehicle (10) and capturing an image of a vehicle periphery, and a display image that can overlay the peripheral image and serves as an operating display for a driver when the driver is operating the vehicle; and an image processing unit (50) configured to change the display image from a first predetermined height to a second predetermined height and to indicate a transition of the change.

Description

AREA

[0001] Embodiments of this disclosure relate to a peripheral monitoring device. BACKGROUND

[0002] A peripheral monitoring device is known from the prior art that captures an image of the situation around a vehicle using a vehicle-mounted imaging unit (camera) and displays the captured image on a display device located near a driver's seat, thereby making it easy for a driver to recognize / view the situation around the vehicle. Such a peripheral monitoring device can display a trajectory prediction line on the screen of a display device to easily recognize, for example, a positional relationship between the subject vehicle and a surrounding object or a space.Furthermore, in a towing vehicle having a towing device, the peripheral monitoring device can display a trajectory prediction line in the direction of a towing vehicle / vehicle to be towed to assist in an approach to the towing vehicle while connected to the towing vehicle. Furthermore, with such a display, a trajectory prediction line can be displayed at a position corresponding to the height of the towing device (trailer hitch) actually approaching the towing vehicle. See, for example, JP 4 739 569 B2 (Reference 1).

[0003] In the prior art, the trajectory prediction line, which is a graphic image different from a camera image, is superimposed on a two-dimensional camera image having a depth in a state where height information is applied to it. Therefore, it may be difficult to determine whether the trajectory prediction line is a line that actually has height information just by observing the displayed trajectory prediction line. Furthermore, if the superimposed line is not recognized as the line with height information, it may appear in an unnatural position, which may interfere with the correct recognition or determination of a driver.

[0004] US 2014 / 0 036 064 A1 relates to an imaging system for a vehicle that includes an imaging sensor and a video display device. The imaging system generates an overlay that is electronically superimposed on the displayed images to assist a driver of the vehicle in performing a backing-up maneuver. The overlay has first, second, and third overlay zones, the overlay zones indicating respective ranges of distances from the rear of the vehicle at respective distances. As indicated to the driver viewing the video display screen when performing a backing-up maneuver, the first distance is closer to the rear of the vehicle than the second distance, and the second distance is closer to the rear of the vehicle than the third distance. The first overlay zone may be a first color, the second overlay zone may be a second color, and the third overlay zone may be a third color.

[0005] Therefore, there is a need for a peripheral monitoring device that makes it easy to recognize that a display image / indicator image (that is, an image serving as an operation indicator when a driver performs a driving operation) is a line with height information when the display image / indicator image with the height information is superimposed on a peripheral image / surrounding image. SUMMARY

[0006] A peripheral monitoring device according to one aspect of this disclosure includes: for example, a capturing / acquiring / providing unit configured to acquire a peripheral image / surrounding image captured by an imaging unit provided in a vehicle and capturing / capturing an image of a vehicle's surroundings / periphery of a vehicle, and a display image / indicator image that can be superimposed on the peripheral image / surrounding image of the vehicle and serves as an operation display of a driver when the driver performs a driving operation / driving operation of the vehicle; and an image processing unit configured to change the display image / indicator image from a first predetermined level to a second predetermined level and indicate a transition of the change.According to this configuration, it is easy for the driver to recognize that the display image / indicator image has altitude information and is displayed even in a state of change (shift / movement) from the first predetermined altitude to the second predetermined altitude, because, for example, the transition of the change is represented in a process of displaying the display image / indicator image.

[0007] In the peripheral monitoring device according to the aspect of this disclosure, the first predetermined height may be lower than, for example, the second predetermined height. According to this configuration, for example, since the display image / indicator image is displayed in a floating state, it is easy to detect / register the behavior of the display image / indicator image, and it is easier to recognize that the display image / indicator image has altitude information and is displayed even when levitated (moving) from the first predetermined height to the second predetermined height.

[0008] In the peripheral monitoring device according to the aspect of this disclosure, the first predetermined height may be higher than, for example, the second predetermined height. According to this configuration, for example, since the display image / indicator image is displayed in a descending motion (descending movement), it is easy to grasp the behavior of the display image / indicator image, and it is easier to recognize that the display image / indicator image has altitude information and is also displayed in a state of descending / moving from the first predetermined height to the second predetermined height.

[0009] In the peripheral monitoring device according to the aspect of this disclosure, the first predetermined height may be, for example, a height in contact with a road surface / lane surface on which the vehicle is located. According to this configuration, for example, the display image / indicator image is displayed so as to float above the road surface / lane surface. As a result, it is easy to recognize the degree of levitation (movement) of the display image / indicator image based on the road surface / lane surface (the magnitude of the height information in the display image / indicator image).

[0010] In the peripheral monitoring device according to the aspect of this disclosure, the second predetermined height may be, for example, a height in contact with a road surface / lane surface on which the vehicle is located. According to this configuration, for example, the display image / indicator image is displayed descending / descending / decline toward the road surface / lane surface. As a result, it is easy to recognize the degree of descent / decline (displacement / movement) of the display image / indicator image with respect to the road surface / lane surface (the magnitude of the height information in the display image / indicator image).

[0011] In the peripheral monitoring device according to the embodiment disclosed herein, the first predetermined height or the second predetermined height may be, for example, a height of a reference position set in the vehicle. According to this configuration, for example, a transition of movement of the display image / indicator image is displayed based on a portion of the vehicle. As a result, it is easy to recognize the degree of movement of the display image / indicator image (the magnitude of the height information in the display image / indicator image).

[0012] In the peripheral monitoring device according to the aspect of this disclosure, the image processing unit can, for example, change a direction indicated by the display image / indicator image according to a current steering angle of the vehicle. According to this configuration, for example, it is possible to effectively utilize the display image / indicator image with altitude information when moving the vehicle, since the display image / indicator image is displayed / presented in the direction of the vehicle.

[0013] In the peripheral monitoring device according to the aspect of this disclosure, for example, the image processing unit may display / present a first display image / indicator image at the first predetermined height and a second display image / indicator image at the second predetermined height side by side in a height direction. According to this configuration, it is easy to recognize a movement state based on one of the first display image / indicator image and the second display image / indicator image, and it may be easier to recognize that the display image / indicator image has height information.

[0014] In the peripheral monitoring device according to the aspect of this disclosure, for example, the image processing unit may change a transmittance at the time of displaying / presenting the display image / indicator image according to a shift state of the corresponding display image / indicator image. According to this configuration, for example, if the display image / indicator image is displayed so as to overlay the peripheral image / surrounding image, it is possible to continuously recognize the displayed object even if the display image / indicator image overlaps another displayed object (e.g., a portion of another vehicle) originally displayed on the peripheral image / surrounding image. As a result, even if the display image / indicator image overlaps the peripheral image / surrounding image, it may be difficult for the visibility of a surrounding situation to be reduced.

[0015] The peripheral monitoring device according to the aspect of this disclosure may further include, for example, a target determination unit configured to determine a movement target position of the vehicle on the peripheral image / surrounding image, and the image processing unit may determine the direction indicated by the display image / indicator image along which the vehicle reaches the movement target position. According to this configuration, for example, it is possible to smoothly move the vehicle to the movement target position because the display image / indicator image can be displayed to match the movement target position of the vehicle. DETAILED DESCRIPTION OF THE FIGURES

[0016] The foregoing and additional features and characteristics of this disclosure will become apparent from the following detailed description considered with reference to the accompanying drawings, in which: Fig. 1 is an exemplary schematic view for explaining an example in which a display / representation of a display image / indicator image can be applied by a peripheral monitoring device according to an embodiment, wherein Fig. 1 is a side view explaining a tractor as a vehicle in which the peripheral monitoring device can be mounted and a towing vehicle is connectable / hitchable to the tractor; Fig. 2 is an exemplary schematic view for explaining an example in which display / representation of the display image / indicator image can be applied by the peripheral monitoring device according to an embodiment, wherein Fig. 2 is a plan view explaining an operation of connecting the towing vehicle as a vehicle in which the peripheral monitoring device can be mounted and the towing vehicle is connectable to the towing vehicle; Fig. 3 is an exemplary block diagram of a configuration of a peripheral monitoring system including the peripheral monitoring device according to the embodiment; Fig. 4 is an exemplary block diagram of a configuration of a CPU of the peripheral monitoring device according to the embodiment; Fig. 5 is a view for explaining a display example of a clutch guide image as the display image / indicator image by the peripheral monitoring device according to the embodiment, wherein Fig. 5 is a schematic view illustrating the timing of the start of display of the clutch guide image; Fig. 6 is a view for explaining a display example of the clutch guide image as the display image / indicator image by the peripheral monitoring device according to the embodiment, wherein Fig. 6 is a schematic view illustrating a transient state in which the clutch guide pattern is extended rearward; Fig. 7 is a view for explaining a display example of the clutch guide image as a display image / indicator image by the peripheral monitoring device according to the embodiment, wherein Fig. 7 is a schematic view illustrating a transient state in which the clutch guide image floats; Fig. 8 is a view for explaining a display example of the clutch guide image as a display image / indicator image by the peripheral monitoring device according to the embodiment, wherein Fig. 8 is a schematic view illustrating a transition state in which the trailer coupling guide line image moves according to a steering angle of the vehicle; Fig. 9 is a flowchart for explaining an example of processing for displaying a display image / indicator image by the peripheral monitoring device according to the embodiment; and Fig. 10 is a schematic view for explaining another display example of a display image / indicator image by the peripheral monitoring device according to the embodiment. DETAILED DESCRIPTION

[0017] An exemplary embodiment disclosed herein is described below. A configuration of the embodiment described below, along with actions, results, and effects caused by the configuration, is shown as an example. The disclosure can be implemented by a configuration other than the configuration disclosed in the following embodiment, and at least one of various effects based on a basic configuration and derived effects can be achieved.

[0018] A peripheral monitoring device according to the present embodiment displays a display image to easily recognize that the display image is an image with height information when the display image with the height information is superimposed on a peripheral image / surrounding image captured by an imaging unit. As a vehicle in which the peripheral monitoring device can be mounted, Fig. 1 as a tractor 10. Fig. 1 is a side view explaining the towing vehicle 10 and a towing vehicle 12 connectable / trailable to the towing vehicle 10, and Fig. 2 is a plan view of Fig. 1. In Fig. 1, the left direction on the side is set as the front of the tractor 10 and the right direction of the side is set as the rear of the tractor 10. Fig. 3 is an exemplary block diagram of a configuration of a peripheral monitoring system 100 including the peripheral monitoring device that may be mounted in the towing vehicle 10. Thus, in the present embodiment, a case is described in which a hitch guide image / trailer hitch guide line image, which serves as an operation display of a driver driving the towing vehicle 10 when performing a driving operation for connecting the towing vehicle 10 to the towing vehicle 12, is displayed as an example of the display image.

[0019] First, a configuration of the towing vehicle 10 and the tow vehicle 12 will be described. The towing vehicle 10 may be, for example, an automobile with an internal combustion engine (not shown) as a power source (for example, an internal combustion engine vehicle), an automobile with an electric motor (not shown) as a power source (for example, an electric car or a fuel cell car), or an automobile with both the internal combustion engine and the electric motor as power sources (for example, a hybrid vehicle). The towing vehicle 10 may be a sport utility vehicle (SUV) as shown in Fig. 1 or a so-called "pickup truck" in which a loading area is provided at the rear of the vehicle. Furthermore, the towing vehicle 10 may be a general passenger vehicle. The towing vehicle 10 may be equipped with one of several transmission devices and may be equipped with various devices (e.g., systems or sections) required to drive the internal combustion engine or the electric motor. Furthermore, for example, the type, number, and layout of the devices for driving the wheels 14 (front wheels 14F and rear wheels 14E) in the towing vehicle 10 may be adjusted in various ways.

[0020] A towing device 18 (hitch / trailer hitch) protrudes from the lower portion of a rear bumper 16 of the towing vehicle 10, for example, at the center in the vehicle width direction, to tow the towing vehicle 12. The towing device 18 is fixed / attached to, for example, a frame of the towing vehicle 10. For example, the towing device 18 has a spherical coupling ball 18a provided in its tip end, which is upright in the vertical direction (vehicle vertical direction), and the coupling ball 18a is covered with a coupling 20a provided at a tip end of a connecting member 20 attached to the towing vehicle 12. As a result, the towing vehicle 10 and the towing vehicle 12 are connected to each other, and the towing vehicle 12 can be pivoted (rocked) in the vehicle width direction with respect to the towing vehicle 10.That is, the coupling ball 18a transmits forward, backward, left and right movements to the towing vehicle 12 (the connecting element 20) and additionally receives acceleration or deceleration power.

[0021] For example, in Fig. 1, the towing vehicle 12 may be of a box type having at least one cabin, a living space and a storage space, or it may be of a loading area type in which luggage (for example, a container or a boat) is loaded. Fig. The towing vehicle 12 shown in Figure 1 is a drive vehicle, having a pair of trailer wheels 22 as driven wheels, but no drive wheels or steered wheels. While the Fig. 1 is not connected to the towing vehicle 10, a stand 20b is mounted on the towing vehicle 12, which stand can be attached to the connecting element 20b and removed from the connecting element 20b, so that the towing vehicle 12 maintains its balance without tilting forward.

[0022] In addition, as in Fig. 1 and Fig. 2, the tractor vehicle 10 is equipped with a plurality of imaging units 24, for example, four imaging units 24a to 24d. The imaging unit 24 is a digital camera in which an imaging element such as a charge-coupled device (CCD) or a CMOS image sensor (CIS) is mounted. The imaging unit 24 can output video data (captured / recorded image data) at a predetermined frame rate. Each imaging unit 24 has a wide-angle lens or a fisheye lens and is capable of capturing an image of, for example, a range of 140° to 220° in the horizontal direction. Furthermore, the optical axis of the imaging unit 24 can be adjusted obliquely downward in some cases.Thus, the imaging unit 24 sequentially captures an image of the environment outside the tractor 10, including the road surface / carriageway surface on which the tractor 10 is moving, the markings (e.g., arrows, lanes, parking space lines, and lane divider lines) or objects (including obstacles such as pedestrians and vehicles) applied to the road surface.

[0023] For example, an imaging unit 24a is arranged on a wall section below a tailgate 10a on the rear of the towing vehicle 10. The imaging unit 24a can capture an image of a rear area including, for example, the rear end (the rear bumper 16) of the towing vehicle 10 and the towing device 18. Furthermore, when the towing vehicle 12 is connected to the towing vehicle 10, the imaging unit 24a can capture an image of an area including the connecting element 20 and at least the front end of the towing vehicle 12 (for example, the area defined by the two-point chain line in Fig. 1) or an image of the rear area of ​​the towing vehicle 12, which is facing the vehicle 12 from the side. The image data captured / recorded by the imaging unit 24a can be used to detect an object present in the rear area of ​​the towing vehicle 10 (for example, a parked vehicle). In addition, the captured / recorded image data, as shown in the Fig. 1 and Fig. 2, can be used to detect the towing vehicle 12 and to detect a connection state (e.g., a connection angle or the presence or absence of a connection) between the tractor vehicle 10 and the towing vehicle 12. In this case, since the connection state or the connection angle between the tractor vehicle 10 and the towing vehicle 12 can be detected based on the image data acquired / captured by the imaging unit 24a, a system configuration can be simplified and the burden of calculation or image processing can be reduced.

[0024] Furthermore, for example, the imaging unit 24b is provided at the left end of the tractor 10, for example, at a left outside mirror 10b, and captures a left side image with a range centered on the left side of the tractor 10 (for example, a range from the left front to the left rear). For example, the imaging unit 24c is provided at the front of the tractor 10, that is, an end portion at the front in the vehicle longitudinal direction, for example, a front grille 10c or a front bumper, and captures a front image including the front of the tractor 10.For example, the imaging unit 24d is provided at the right end of the tractor 10, such as the right outside mirror 10d, and acquires a right side image with a region centered on the right side of the tractor 10 (for example, a region from the right front to the right rear). Calculation or image processing may be performed based on the acquired / captured image data obtained by the plurality of imaging units 24 to generate an image with a larger / wider viewing angle or to generate a virtual bird's-eye view (plane) image of the tractor 10 viewed from above.

[0025] As in Fig. 3, a display device 26 and a sound output device 28 are provided in a vehicle cabin of the tractor vehicle 10. The display device 26 is, for example, a liquid crystal display (LCD) or an organic electroluminescent display (OLED). The sound output device 28 is, for example, a loudspeaker. Furthermore, the display device 26 is covered with a transparent operation input unit 30, such as a touch panel. An occupant (for example, a driver) can visually perceive an image displayed on a screen of the display device 26 through the operation input unit 30. Furthermore, the occupant can perform the operation input by operating the operation input unit 30 by touching, pressing, or moving with a finger, for example, at a position corresponding to the image displayed on the screen of the display device 26.The display device 26, the sound output device 28, the operation input unit 30, and the like are provided, for example, in a monitoring device 32 arranged on the central portion in the vehicle width direction, i.e., in the transverse direction of an instrument panel of the tractor vehicle 10. The monitoring device 32 may include an operation input unit (not shown) such as a switch, a dial, a joystick, or a push button. For example, the monitoring device 32 can also be used as a navigation system or an audio system.

[0026] In addition, the towing vehicle 10, as shown in the Fig. 1 and Fig. 2, for example, a four-wheeled vehicle and has two left and right front wheels 14F and two left and right rear wheels 14R. All four wheels 14 can be configured to be easily steerable. As shown in Fig. 3, the tractor vehicle 10 includes a steering system 34 that controls / steers at least two wheels 14. The steering system 34 includes an actuator 34a and a torque sensor 34b. The steering system 34 is electrically controlled, for example, by an electronic control unit (ECU) 36 to operate the actuator 34a. The steering system 34 is, for example, an electric power steering system or a steer-by-wire (SBW) system. The steering system 34 transmits torque, i.e., an assist torque, to a steering wheel through the actuator 34a to supplement a steering force, or steers the wheels 14 through the actuator 34a. In this case, the actuator 34a can control a pair of wheels 14 or multiple wheels 14. In addition, the torque sensor 34b detects, for example, the torque applied by the driver to the steering wheel.

[0027] In addition, as in Fig. 3, in the peripheral monitoring system 100 (peripheral monitoring device), in addition to the ECU 36, the monitoring device 32 and the steering system 34, for example, a braking system 38, a steering angle sensor 40, a shift sensor 42, and a wheel speed sensor 44 are electrically connected via an in-vehicle network 46 as an electrical communication link. The in-vehicle network 46 is configured, for example, as a Controller Area Network (CAN). The ECU 36 can control, for example, the steering system 34 or the braking system 38 by sending a control signal via the in-vehicle network 46. In addition, the ECU 36 can receive, for example, a detection result of the torque sensor 34b, the braking sensor 38b, the steering angle sensor 40, the shift sensor 42, or the wheel speed sensor 44, or an operation signal of the operation input unit 30 via the in-vehicle network 46.

[0028] The ECU 36 includes, for example, a central processing unit (CPU) 36a, a read-only memory (ROM) 36b, a random access memory (RAM) 36c, a display controller 36d, an audio controller 36e, and a solid-state drive (SSD) (flash memory) 36f. The CPU 36a reads a program stored / installed in a non-volatile storage device such as the ROM 36b and performs calculation processing according to the program. For example, the CPU 36a performs image processing with respect to an image displayed on the display device 26. For example, the CPU 36a performs calculation or image processing on the acquired / captured image data acquired by the imaging unit 24 to generate a peripheral image / surrounding image (e.g., a bird's-eye view image).

[0029] The RAM 36c temporarily stores various types of data used for calculation in the CPU 36a. Furthermore, the display controller 36d mainly executes a combination of image data displayed on the display device 26 between calculation processing in the ECU 36. Furthermore, the sound controller 36e mainly executes processing of the sound data output from the sound output device 28 between calculation processing in the ECU 36. The SSD 36f is a rewritable, non-volatile storage unit and can store data even when the power supply of the ECU 36 is turned off. Furthermore, for example, the CPU 36a, the ROM 36b, and the RAM 36c can be integrated into a same housing / package. Furthermore, the ECU 36 can be configured to use another logic operation processor, such as a digital signal processor (DSP) or a logic circuit, instead of the CPU 36a.In addition, a hard disk (HDD) may be provided instead of the SSD 36f, and the SSD 36f and the HDD may be provided separately from the ECU 36.

[0030] The braking system 38 includes, for example, an anti-lock braking system (ABS) that prevents brake locking, an electronic stability control (ESC) that prevents the tractor 10 from skidding sideways while cornering, an electric braking system that increases braking force (provides brake assistance), or a brake-by-wire (BBW) system. The braking system 38 applies braking force to the wheel 14 and transmits the braking force to the tractor 10 via an actuator 38a. Furthermore, the braking system 38 can detect, for example, brake locking, wheel 14 running idle, and sideways skidding based on, for example, the rotational difference between the left and right wheels 14 and perform various checks. The brake sensor 38b is, for example, a sensor that detects the position of a movable element of the brake pedal.

[0031] The steering angle sensor 40 is, for example, a sensor that detects the steering angle of the steering wheel. The ECU 36 detects, for example, the steering angle of the steering wheel by the driver or the steering angle of each wheel 14 during automatic steering from the steering angle sensor 40 and performs various checks. The shift sensor 42 is, for example, a sensor that detects the position of a movable element of a shift device unit. The wheel speed sensor 44 is a sensor that detects the rotational speed of the wheel 14 or the number of revolutions per unit time. The wheel speed sensor 44 outputs the number of wheel speed pulses as a sensor value, which indicates the detected number of revolutions. The ECU 36 can, for example, calculate the amount of movement of the tractor 10 based on a sensor value detected by the wheel speed sensor 44 and perform various checks.

[0032] Furthermore, configurations, arrangements or the electrical connection form of the various sensors and actuators described above are given only as examples and can be set (changed) in various ways.

[0033] The display device 26 may superimpose, for example, a clutch guide image or a vehicle width guide line image as an example of a display image serving as a driver's operation display, on a peripheral image acquired by the imaging unit 24 and display the superimposed image when the driver performs a driving operation. In the present embodiment, when displaying the display image, it is possible to change the display image from a first predetermined height to a second predetermined height and display a transition of the change. Moreover, the predetermined height (for example, the first predetermined height or the second predetermined height) corresponds to a three-dimensional height in a real space, of which an image is captured by the imaging unit 24.For example, the coupling guide image may hover at a height corresponding to the height of a reference position set in the towing vehicle 10, and a levitation transition may be displayed. To facilitate the coupling ball 18a of the towing vehicle 10 approaching the coupling 20a of the towing vehicle 12 when the towing vehicle 10 and the towing vehicle 12 connect, the coupling guide image may, for example, be a guide line image displayed extending from the coupling ball 18a to the coupling 20a. In this case, the coupling ball 18a is, as shown in FIG. Fig. 1, for example, at the position of a height L (the second predetermined height) from the road surface (the first predetermined height) on which the tractor 10 is located. Thus, when the hitch guide image is superimposed on the peripheral image displayed on the display device 26, the hitch guide image is displayed at a position corresponding to the height L. However, when height information is added to the hitch guide image, which is a graphic image different from a two-dimensional camera image (peripheral image) with a depth, and the high guide line image is superimposed on the camera image, the display position is easily offset according to the height information. As a result, it is difficult to recognize at first glance that the height information is reflected on the hitch guide image.Thus, in the peripheral monitoring device of the present embodiment, in order to easily recognize that the display image, for example, the clutch guide image, is displayed with the height information, a transition of movements is also displayed when the display image is displayed. This makes it easy for the driver to recognize that the display image (for example, the clutch guide image) has the height information and is displayed in a state floating above the road surface.

[0034] The CPU 36a, which is included in the ECU 36, has various modules for processing the display of the display image (e.g., the clutch guide image), which can also display the transition of a movement (e.g., levitation) as described above. The various modules are implemented by the CPU 36a reading and executing a program installed and stored in a storage device such as the ROM 36b. As shown in Fig. 4, the CPU 36a has, for example, modules such as a recording / capturing unit 48, an image processing unit 50, and a target determination unit 52.

[0035] The acquisition unit 48 has, for example, an operation mode acquisition unit 48a, a peripheral image acquisition unit 48b, a display image acquisition unit 48c, a steering angle acquisition unit 48d, and a height information acquisition unit 48e to acquire various pieces of information for realizing the display of the display image (for example, the clutch guide image) accompanying the levitation transition.

[0036] For example, the operation mode acquisition unit 48a receives / acquires an operation signal based on an operation of the operation input unit 30, such as when the driver switches between the image contents (display modes) displayed on the display device 26. The display modes include, for example, a "normal display mode," a "display image display mode," a "standard guide line image display mode," a "guide line aid image display mode," a "road surface mode," and a "transition display mode."

[0037] The "normal display mode" is a display mode in which, for example, a navigation screen or an audio screen is displayed, which is normally displayed during the travel of the tractor 10. The "display image display mode" is a display mode in which the display image (for example, the coupling guide image) is displayed as an operation display of the driver when the driver performs a driving operation, so that the peripheral image captured by the imaging unit 24 is superimposed. Furthermore, when the display image display mode is selected, it is possible to select, for example, the "standard guide line image display mode" and the "guide line aid image display mode."The "standard guidance line image display mode" is a display mode in which, for example, a vehicle width guidance line image is displayed, which is used to determine whether the tractor 10 can pass or not, or a distance guidance line image (also referred to as a distance reference line image) is displayed to easily grasp the distance behind or in front of the tractor 10 during the travel of the tractor 10. Furthermore, the "guidance aid image display mode" is a display mode in which a guidance line image for guiding the tractor 10 to a specific target position, such as the hitch guidance image indicating the direction of movement of the hitch ball 18 or a travel prediction line image indicating a predicted travel direction of the entire tractor 10, is displayed.In addition, for example, in the “guide line aid image display mode”, a parking frame guide line image indicating a parking space can be displayed when the tractor vehicle 10 is parked.

[0038] The "road surface mode" is a display mode in which the display image is drawn in contact with the road surface on which the tractor 10 is located. The "transition display mode" is a display mode in which a display position of the display image is changed from the first predetermined height to the second predetermined height, and a transition of the change is displayed. For example, when the display image is the hitch guide image, a display position of the hitch guide image floats from a state of contact with the road surface (the first predetermined height) to a reference position set in the tractor 10 (the second predetermined height), and a levitation transition is displayed.The "reference position height" set in the tractor 10 is the height determined according to the display image to be displayed, and in the case of the hitch guide image, it may be the height L of the hitch ball 18a attached to the tractor 10 from the road surface. Furthermore, the reference position height may be, for example, the height of the rear bumper 16 of the tractor 10 when the display image is, for example, the vehicle width guide line image.

[0039] For example, when the operation mode acquisition unit 48a acquires the "display image display mode," the peripheral image acquisition unit 48b acquires image information necessary for displaying a situation surrounding the tractor 10. For example, the peripheral image acquisition unit 48b acquires data of a plurality of captured images (e.g., data of a forward image, a left side image, a right side image, and a rearward image) from the imaging units 24a to 24d that capture an image of the surroundings of the tractor 10. There are a case where the captured images are real images and are sequentially displayed as they are on the display device 26, and a case where the captured image is subjected to, for example, viewpoint conversion and is sequentially displayed as a bird's-eye view image on the display device 26.

[0040] When the operation mode acquisition unit 48a acquires an operation signal indicating the "display image display mode," the display image acquisition unit 48c acquires a basic shape of, for example, the "coupling guide image," the "travel prediction line image," the "vehicle width guide line image," the "distance guide line image," or the driver-designated "parking frame guide line image" by reading the basic shape from, for example, the ROM 36b. For example, the basic shape of each guide line image is displayed so as to be in contact with the road surface on which the tractor vehicle 10 is located, and is a straight line parallel to the vehicle center line extending in the longitudinal direction of the tractor vehicle 10, or a straight line orthogonal to the vehicle center line. For example, the "coupling guide image" is a linear image extending at a predetermined length, for example, a length equal to 2.5 m from a position on the road surface immediately below the coupling ball 18a to the rear of the tractor 10. Similarly, the "travel prediction line image" is a linear image extending from a center position in the vehicle width direction of the tractor 10 into the direction of travel of the tractor 10 by, for example, a length corresponding to 2.5 m. The direction of travel of the tractor 10 can be determined from an operating position of the shift sensor 42. For example, the direction of travel of the tractor 10 is the forward direction when the shift sensor 42 is in a D-range position and a reverse direction when the shift sensor 42 is in an R-range position.The "vehicle width guide line image" is a longer image extending a length corresponding to 2.5 m from, for example, the vehicle end in the vehicle longitudinal direction, parallel to the side surfaces of the tractor vehicle 10, to which a left or right edge distance, for example, a length corresponding to 20 cm, is added to the vehicle width of the tractor vehicle 10. The "distance guide line image" is a linear image extending rearward from a position on the road surface immediately below the rear bumper 16 of the tractor vehicle 10 to a position, for example, 0.5 m, 1.0 m, or 2.5 m in a direction parallel to the rear bumper 16. Similarly, the "distance guide line image" can be displayed at a position corresponding to 0.5 m, 1.0 m, or 2.5 m from a position on the road surface / road surface immediately below a front bumper of the tractor vehicle 10.The “parking guide line image” is a linear image that defines the space corresponding to a parking space in which the tractor vehicle 10 is accommodated when a parking target position is set at the time of parking the tractor vehicle 10.

[0041] The steering angle acquisition unit 48d acquires steering angle information (information about an operating state of the steering wheel) of the tractor 10 output from the steering angle sensor 40. In other words, the steering angle acquisition unit 48d acquires a steering angle in a direction in which the driver intends to drive the tractor 10. Furthermore, the steering angle acquisition unit 48d can detect whether the tractor 10 is in a forward-moving state or a backward-moving state based on the position of a movable member detected by the switch sensor 42, and can identify whether the steering angle is a steering angle in a forward-moving state or a steering angle in a backward-moving state.

[0042] The height information acquisition unit 48e acquires height information when the display image is moved (e.g., hovered) from the first predetermined height to the second predetermined height of, for example, the ROM 36b. As described above, the height information is determined according to the display image to be displayed. For example, when the hitch guide image is displayed as a display image, the height information is the height L of the hitch ball 18a attached to the tractor 10 from the road surface, and when the vehicle width guide line image is displayed as the display image, the height information is the height of the rear bumper 16 of the tractor 10. The height information may be a predetermined fixed value based on the specification of the tractor 10 or a value appropriately set by the driver via the operation input unit 30 or the like.

[0043] The Fig. 5 to 8 are views for explaining display examples of the display image by the image processing unit 50 of the peripheral monitoring system 100 (peripheral monitoring device), and illustrate examples of displaying the coupling guide image 62 as an example of the display image that supports a hitching / connecting operation for hitching / connecting the towing vehicle 12 to the towing vehicle 10 when the towing vehicle 10 is moving backward (when the shift sensor 42 detects the position of the R range). In a case of the “display image display mode” in which the display image is displayed as shown in Fig. 5 to 8, the display device 26 can be set as a two-part screen consisting of a first display area 26a with a real image P1 and a second display area 26b with a bird's-eye view image P2. For example, the real image P1 shows a peripheral image behind the tractor 10 acquired by the imaging unit 24a. Moreover, the bird's-eye view image P2 is a panoramic image about the position of the tractor 10 obtained by performing image processing such as viewpoint conversion with respect to a position immediately above the tractor 10 as a virtual viewpoint on each of the rearward images, the left side images, the forward images, and the right side images of the tractor 10 acquired / captured by the imaging units 24a to 24d.Moreover, when the tractor 10 is moving forward, a peripheral image in front of the tractor 10 captured by the imaging unit 24c is displayed as the real image P1, and the display image is superimposed accordingly.

[0044] In the real image P1 displayed in the first display area 26a, the rear bumper 16 of the tractor 10 and the towing device 18 (coupling ball 18a) are displayed on the lower side (front side) of the screen, and a situation behind the tractor 10 is displayed above them. In addition, in the real image P1, as shown in Fig. 5, for example, the tow vehicle 12 with the connecting element 20 (the coupling 20a), which is mixed with several parked vehicles, is displayed on the top (far side) of the screen. Furthermore, a switch symbol 54, which serves as a switch for switching between the display modes of the display device 26, is displayed in a section (for example, the lower right side of the screen) of the first display area 26a. In a case of Fig. 5, the switch icon 54 is a switch that is newly displayed after selecting the “display image display mode” and has a standard mode icon 54a for switching to the “standard guideline image display mode” and a guide mode icon 54b for switching to the “guideline aid image display mode”.

[0045] Furthermore, in the bird's-eye view image P2 displayed in the second display area 26b, an own vehicle icon 56 indicating the tractor 10 from a bird's-eye view is displayed in the central area. For example, the own vehicle icon 56 can be read out from the ROM 36b and displayed. Furthermore, a vehicle width guide line image 58, a right guide line image 58a, and a left guide line image 58b are displayed behind the own vehicle icon 56. Furthermore, a plurality of distance guide line images 60 are displayed extending between the right guide line image 58a and the left guide line image 58b. The distance guide line images 60 include, for example, a first distance guide line image 60a indicating a position of 0.5 m, a second distance guide line image 60b indicating a position of 1.0 m, and a third distance guide line image 60c indicating a position of 5 m behind the rear bumper 16 of the tractor vehicle 10. Furthermore, the number of distance guide line images 60 is not limited to three and can be increased or decreased, for example, depending on the driver's setting, and the distance between the respective distance guide line images 60 can be adjusted accordingly.

[0046] In addition, in the Fig. 5 to 8, a clutch guide image 62 is displayed as an example of the display image. Fig. Fig. 5 is a schematic view illustrating the timing of the start of the display of the clutch guide image 62, and Fig. 6 is a schematic view illustrating a transient state in which the coupling guide pattern 62 extends rearwardly from the tractor 10. In addition, Fig. 7 is a schematic view illustrating a transition state in which the clutch guide image 62 is floating, and Fig. 8 is a schematic view illustrating a transient state in which the floating coupling guide pattern 62 moves according to the steering angle of the tractor 10.

[0047] The image processing unit 50 has, for example, a transition display processing unit 50a, a shadow image processing unit 50b, a display image movement processing unit 50c, and a transmission processing unit 50d for changing the movement (e.g., levitation) of the display image (e.g., the clutch guide image 62) and a display aspect after the movement.

[0048] The transition display processing unit 50a performs image processing (animation processing) of the floating of the clutch guide image 62 as shown in Fig. 7, by adding movement (floating) height information obtained by the height information acquisition unit 48e to the drawing data of the coupling guide image 62 (see Fig. 6), for example, because the display image has a basic shape in contact with the road surface captured by the display image acquisition unit 48c. The movement speed (levitation) at this time may be a predetermined fixed value that enables the detection of the presence or absence of movement, or may be a target value set by the driver via the operation input unit 30. By displaying, for example, a levitation transition of the clutch guide image 62 as a display image, the driver can easily recognize that the clutch guide image 62 (display image) is a drawn image with height information. Furthermore, the transition display processing unit 50a may, when first displaying the clutch guide image 62 with a basic shape in contact with the road surface, as shown in FIGS. Fig. 5 and Fig. 6, display an extension transition (animation) that is gradually extended to a position corresponding to 2.5 m behind the towing vehicle 10. The representation of such an extension transition makes it easier to focus the driver's attention and recognize that the coupling guidance image 62 (display image) is displayed.

[0049] As in Fig. As shown in Figure 7, when moving (e.g., levitating) the display image (e.g., the clutch guide image 62), the shadow image processing unit 50b separates a shadow image 64 (first display image / indicator image) indicating an initial state of contact with the road surface from the clutch guide image 62 (second display image) in a moving state, and displays the two images side by side in the height direction. Moreover, the height direction in this case is not necessarily the vertical direction. Since the shadow image 64 remains on the road surface, unlike the floating clutch guide image 62, the shadow image 64 can be regarded as a "shadow" of the clutch guide image 62 projected onto the road surface. In other words, by displaying the shadow image 64, it can be easier to recognize that the clutch guide image 62 is in a levitation state.Furthermore, the silhouette image 64 may be displayed in "black," for example, and the floating clutch guide image 62 may be displayed in a color that is easily distinguishable from letters and markings on the road surface, such as "green," "blue," or "yellow." Furthermore, the silhouette image 64 before floating may be displayed so that it overlaps below the clutch guide image 62. Moreover, at this time, by slightly deviating the silhouette image 64 from the clutch guide image 62 in the vehicle width direction, it may be easy for the driver to recognize the presence of the silhouette image 64 before the clutch guide image 62 is levitated.

[0050] As in Fig. As shown in Figure 8, the display image movement processing unit 50c moves (deforms) the display image (e.g., the coupling guide image 62) moved in the height direction (e.g., floating) and the shadow image 64 left on the road surface in the steering direction according to the steering information obtained from the steering angle acquisition unit 48d. That is, the coupling guide image 62 and the shadow image 64 are aligned in a direction in which the tractor 10 is pointing when the tractor 10 is traveling. For example, as shown in Fig. 8 illustrates that the towing vehicle 12 to be connected / attached is stopped behind the towing vehicle 10, it is possible to move the towing vehicle 10 toward the towing vehicle 12 by operating the steering wheel of the towing vehicle 10 so that the coupling guide pattern 62 is aligned with the towing vehicle 12. That is, it is possible to easily cause the coupling ball 18a to approach the coupling 20a.

[0051] The transmission processing unit 50d changes a transmission in displaying the display image (e.g., the clutch guide image 62) according to a shift state (e.g., levitation) of the display image. For example, as shown in FIGS. Fig. 5 and Fig. 6, when the coupling guide image 62 (which is in contact with the road surface) is extended, the coupling guide image 62 is displayed with a transmittance of “0%” so that the presence of the coupling guide image 62 is clearly visible. Then, when hovering, as shown in Fig. 6, the transmittance of the clutch guide image 62 is gradually increased so that, for example, 50% is reached when the clutch guide image 62 hovers at a position corresponding to the height information acquired by the height information acquisition unit 48e. As a result, even when the clutch guide image 62 is superimposed on an object originally present in the peripheral image acquired by the imaging unit 24a, such as the clutch 20a or a display object (e.g., lanes) on the road surface, the clutch 20a or the display object on the road surface can be recognized by the clutch guide image 62 with the increased transmittance. For example, it is easy to align the tip of the clutch guide image 62 with the position of the clutch 20a by changing the steering angle.Furthermore, the transmission processing unit 50d can also change the transmission of the shadow image 64 in the same way and achieve the same effect. The transmission can be set to a fixed value as described above or can be adjusted accordingly by the driver, for example, via the control input unit 30.

[0052] In addition, the examples of Fig. 5 to 8, the clutch guide image 62 or the shadow image 64 is represented by a dashed line. By such representation using the dashed line, when the display image such as the clutch guide image 62 or the shadow image 64 is superimposed on the real image P1, a portion overlapping, for example, the clutch 20a or a display object on the road surface is reduced, which improves the recognition of the clutch 20a or the display object. Conversely, when emphasizing the recognition of the display image such as the clutch guide image 62 or the shadow image 64, the display image may be represented with a solid line.

[0053] The target determination unit 52 is a module that can be added as an additional function. When displaying the display image as a hitch guide image 62, the target determination unit 52 can perform processing for determining a movement target position on the peripheral image. For example, when the hitch guide image 62 is displayed, the movement target position is a position where the tow vehicle 12 is located. In this case, as described above, after hovering and displaying the hitch guide image 62, the driver aligns the hitch guide image 62 with the position of the tow vehicle 12 by operating the steering wheel. As another embodiment, when using the target determination unit 52, the tow vehicle 12 can be determined by the target determination unit 52 to be in a movement target position, and the hitch guide image 62 can be automatically drawn to reach that position.The movement target position can be determined, for example, by detecting the tow vehicle 12 through image processing such as pattern matching by comparing the peripheral image acquired by the imaging unit 24a with a reference pattern (e.g., a shape pattern of the tow vehicle 12) previously held in the ROM 36b or the like. Furthermore, as another embodiment, the driver can determine the movement target position by clicking the tow vehicle 12 in the real image P1 on the screen via the operation input unit 30. When the target determination unit 52 is used, the actuator 34a of the steering system 34 can be operated in a direction in which the clutch guide image 62 is extended to the movement target position determined by the target determination unit 52, so that the steering angle is automatically adjusted.As a result, it is possible to align the coupling ball 18a and the coupling 20a more smoothly and easily, and the driver can immediately start moving the tractor 10 toward the towing vehicle 12 (the coupling 20a) without adjusting the steering angle. Furthermore, in another embodiment, in addition to displaying the coupling guide image 62 (or the shadow image 64) automatically extended to the movement target position determined by the target determination unit 52, a steering angle guide line image corresponding to the angle (steering angle) of the steering wheel of the tractor 10, which is similar to the coupling guide image 62, may be displayed, and the driver can operate the steering wheel so that the steering angle guide line image is superimposed on the coupling guide image 62 automatically aligned to the movement target position.In this case, since the displayed lines overlap, the steering adjustment for superimposing is easy compared to the alignment between the coupling guide image 62 and the towing vehicle 12 present at a distance, and the lines can be aligned more accurately with each other.

[0054] Details of display processing by the peripheral monitoring device (peripheral monitoring system 100) configured as described above will be described below with reference to the flowchart of Fig. 9. In addition, a Fig. 9 illustrates a case where the clutch guide image 62 is floated and displayed among the guide line aid images for representing a movement transition of the display image, and a case where other display images, such as the vehicle width guide line image and the distance guide line image, are not floated. Although the peripheral monitoring system 100 may float multiple types of display images, since this may complicate the display content of the display device 26 and make it difficult to grasp the display content, the number of display images to be floated and displayed may be set / adjusted to, for example, 1, which is appropriately selected by the driver.

[0055] First, the CPU 36a detects a display mode requested by the display device 26 via the mode detection unit 48a. For example, if the display mode switching switch of the operation input unit 30 displayed on the display device 26 is not operated and thus the display image display mode is not selected (No in S100), it is determined that the driver of the tractor vehicle 10 desires the "normal display mode" in which a normal screen, such as a navigation screen or an audio screen, is displayed. In this case, the CPU 36a ends this process without changing the display mode of the display device 26.

[0056] On the other hand, the peripheral image acquisition unit 48b acquires the peripheral image of the tractor 10 output by the imaging unit 24 (S102) in S100 when switching between the display modes of the operation input unit 30 displayed on the display device 26 occurs and the display mode "Display Image / Indicator Image" is selected (Yes in S100). For example, the CPU 36a displays, as shown in Fig. 5, the real image P1 uses the image captured by the imaging unit 24a in the first display area 26a, and generates the bird's-eye view image P2 using the images captured by the imaging units 24a to 24d to display the bird's-eye view image P2 in the second display area 26b. In this case, for example, the switch icon 54 is displayed at the lower right of the real image P1. The switch icon 54 displayed at this time includes the standard mode icon 54a for switching to the "standard guide line image display mode" in which the vehicle width guide line image 58 or the distance guide line image 60 is displayed, and the guidance mode icon 54b for switching to the "guide line aid image display mode" in which the clutch guide image 62 is displayed. Furthermore, the transmission processing unit 50d performs the setting of the transmission of the display image to be displayed (S104).As described above, the transmittance setting may vary depending on the display aspect of the display image. For example, when the display image is floating, the display image is strongly / clearly displayed with a transmittance of 0% when the display image is in contact with the road surface, and the transmittance is gradually increased during the floating, increasing by 50% at the time of the completion of the floating, so that the road surface on which the display image is superimposed is visible through the display image. In addition, when the display image is not floating, that is, when the display image is displayed in a state where it is in contact with the road surface, the display image is slightly displayed with a transmittance of 50% from the start of the display.In addition, the transmission can be determined accordingly by the driver, for example via the operating input unit 30, but step S104 can be omitted if a predetermined value as described above is used.

[0057] Then, when the operation mode acquisition unit 48a acquires an operation signal of the guidance mode icon 54b, that is, when the "guidance line aid image display mode" is selected (Yes in S106), the display image acquisition unit 48c acquires the coupling guide image 62 as the guidance line aid image from the ROM 36b (S108). In this case, for example, the acquired coupling guide image 62 is displayed in contact with the road surface on which the tractor 10 is located, and is a straight line parallel to the vehicle center line extending in the longitudinal direction of the tractor 10.

[0058] Subsequently, the transition display processing unit 50a, as shown in the Fig. 5 and Fig. 6 illustrates the coupling guide image 62 being gradually extended toward the rear of the tractor vehicle 10 from a position near the towing device 18 (S110). By displaying the coupling guide image 62 gradually extended in an animated manner in the real image P1, in which various objects and lines (e.g., lanes) are displayed, it is possible to make the coupling guide lines stand out, and it is easy for the driver to focus on the coupling guide image 62. The transition display processing unit 50a stretches the coupling guide image 62 to a predetermined length (e.g., a length corresponding to 2.5 m behind the tractor vehicle 10).In addition, the transition display processing unit 50a for drawing the coupling guide image 62 also displays a transition switch icon, which can be used to select whether or not to display the transition (levitation) of the coupling guide image 62 displayed in the real image P1, for example, in the lower left corner of the real image P1. When the operation mode acquisition unit 48a acquires / detects an operation signal indicating the selection of the "transition display mode" by operating the transition switch icon (Yes in S112), the height information acquisition unit 48e acquires the height information of the coupling guide image 62, for example, from the ROM 36b (S114). The height information may be the height of the coupling ball 18a of the towing device 18, which is input in advance, for example, based on the specifications of the towing vehicle 10.

[0059] Subsequently, the transition display processing unit 50a starts, as shown in Fig. 7, a transition display processing of the clutch guide image 62 (S116). Furthermore, according to the transition display processing of the clutch guide image 62 by the transition display processing unit 50a, the shadow image processing unit 50b performs drawing processing of the shadow image 64 as shown in Fig. 7. That is, when the clutch guide image 62, which extends to the point where it contacts the road surface, begins to levitate, the shadow image 64 is separated from the clutch guide image 62 and displayed to remain on the road surface. By displaying the shadow image 64 in this way, it is possible to more clearly illustrate that the clutch guide image 62 is levitating. Moreover, the clutch guide image 62 can automatically begin levitation / levitation at the time when the drawing of the clutch guide image 62 is completed. In this case, the processing of S112 can be omitted.

[0060] Subsequently, the CPU 36a determines via the steering angle detection unit 48d (S118) whether the steering angle of the tractor 10 has been changed or not. For example, if the driver, as in Fig. 8, changes the steering angle by steering the steering wheel so that the coupling guide image 62 is directed towards the towing vehicle 12 parked behind the tractor 10 (Yes in S118), as in Fig. As shown in Figure 8, the display image moving unit 50c moves the display image (the clutch guide image 62) according to the steering angle (steering angle information) acquired by the steering angle acquisition unit 48d (S120). In addition, the CPU 36a skips the processing step of S120 when the steering angle information acquired by the steering angle acquisition unit 48d does not change (No in S118). In this case, the display image moving unit 50c maintains the current display state of the display image (for example, the clutch guide image 62) displayed in the real image P1. Therefore, when the driver operates the steering wheel while changing the display state of the display image in the real image P1 (the state of the curved clutch guide image 62 in Fig. 8), the display is performed such that the display image (for example, the clutch guide image 62) is moved or maintains its state in response to the operation.

[0061] The CPU 36a confirms whether or not to stop the repeated display of the display image during the display of the display image (S122). For example, if an operation signal for switching / changing the display mode is not acquired via the operation mode acquisition unit 48a and a request to stop the display of the display image is not confirmed (No in S122), the CPU 36a returns to the processing step of S118. That is, the CPU 36a confirms whether or not the steering angle information has been changed via the steering angle acquisition unit 48d and continues to display the display image.On the other hand, when the operation signal for changing the display mode is acquired via the operation mode acquisition unit 48a and the request to stop displaying the display image is confirmed (Yes in S122), for example, when changing to the “normal display mode”, the CPU 36a returns the display of the display device 26 to, for example, a navigation screen and ends the process.

[0062] Furthermore, in the processing step of S112, when the operation mode acquisition unit 48a does not acquire the operation signal indicating the selection of the "transition display mode" by the operation of the transition switch icon (No in S112), the CPU 36a skips the processing steps of S114 and S116 and proceeds to the processing step of S118. That is, when the driver wants to display the display image (for example, the clutch guide image 62) in a state where it is in contact with the road surface, the CPU 36a moves the display image (the guide line auxiliary image) that remains in contact with the road surface based on the steering angle information acquired by the steering angle acquisition unit 48d.

[0063] Furthermore, if the "guideline aid image display mode" is not selected in the processing step of S106 (No in S106), the CPU 36a considers that the "guideline aid image display mode" has been selected. Subsequently, the display image acquisition unit 48c acquires the vehicle width guideline image 58 (the right side guideline image 58a and the left side guideline image 58b) and the distance guideline image 60 (the first distance guideline image 60a, the second distance guideline image 60b, and the third distance guideline image 60c) from a basic form from the ROM 36b (S124). Then, the transition display processing unit 50a draws, as shown in Fig. 10, the captured image is converted into the real image P1 (S126). Since the display content of the real image P1 may be complicated when the vehicle width guide line image 58 and the distance guide line image 60 are displayed in the real image P1, a center mark C indicating the center position in the vehicle width direction of the tractor vehicle 10 may also be displayed to easily detect the center of the vehicle.

[0064] Even if the vehicle width guide line image 58 or the distance guide line image 60 is being drawn, the CPU 36a shifts to the processing step of S118 and performs the processing step after S118. That is, based on the steering angle information of the tractor 10 obtained by the steering angle acquisition unit 48d, the display of the vehicle width guide line image 58 or the distance guide line image 60 is moved according to a direction in which the steering wheel is steered, as in the display of the clutch guide image 62.

[0065] The vehicle width guide line image 58 (the right side guide line image 58a and the left side guide line image 58b) and the distance guide line image 60 (the first distance guide line image 60a, the second distance guide line image 60b and the third distance guide line image 60c) can be displayed in the bird's eye view image P2 as shown in the Fig. 5 to 8 and Fig. 10, regardless of whether the "Standard Guideline Help Image Display Mode" is selected or not. In addition, as shown in Fig. 8, the direction of the vehicle width guide line image 58 or the distance guide line image 60 shown in the bird's eye view image P2 can be changed according to the steering angle of the tractor 10. When the steering angle of the tractor 10 deviates from a neutral angle (the angle of a straight-line driving state = 0 °), as shown in Fig. 8, a corner trajectory line image 66 may be displayed, which indicates a movement prediction trajectory of a corner portion located on the rear side in the traveling direction of the tractor 10 and also on the outer turning side. As described above, by displaying, for example, the vehicle width guide line image 58, the distance guide line image 60, and the corner trajectory line image 66 in the bird's-eye view image P2, it is easy to grasp, for example, the state of the steering angle of the tractor 10 and the future position of the tractor 10 when traveling at the steering angle, and it is possible to reduce the driving load of the tractor 10.

[0066] In this way, when displaying the display image with height information to be displayed in the real image P1 (peripheral image), the driver can easily recognize that the display image has height information and is displayed in a state of movement (levitation) from the road surface by further displaying a motion transition (e.g., levitation). Furthermore, since the display image (e.g., the coupling guide image 62) with the height information is displayed at a suitable location, it is possible, for example, to guide the coupling ball 18a more smoothly and accurately to the coupling 20a.

[0067] Although the flowchart of Fig. 9 illustrates an example in which the direction indicated by the hitch guide image 62 is changed by the driver's steering of the steering wheel, as described above. Furthermore, a movement target position may be set by the target determination unit 52, and the direction indicated by the hitch guide image 62 may be automatically determined. In this case, after acquiring the display image (for example, the hitch guide image 62) in the processing step of S108, for example, pattern matching processing may be performed on the real image P1 to extract the tow vehicle 12. Subsequently, the hitch guide line image 62 directed toward the tow vehicle 12 in the real image P1 is generated by the display image movement processing unit 50c and drawn by the transition display processing unit 50a.In this state, the transition display processing of the clutch guide image 62 can be executed. In this case, the ECU 36 can control the actuator 34a of the steering system 34 to change the steering angle to correspond to the drawn clutch guide image 62. As a result, the driver can easily and precisely perform a driving operation by bringing the clutch ball 18a closer to the clutch 20a by operating only an accelerator pedal and a brake. Furthermore, the movement target position of the tow vehicle 12 (the position of the tow vehicle 12) in the real image P1 can be determined by the driver specifying a desired movement target position (e.g., the position of the tow vehicle 12) in the real image P1 via the operation input unit 30 (e.g., clicks).In this case, it is not necessary to perform image processing such as pattern recognition processing, which can contribute to simplifying the system or reducing computational processing. Furthermore, when the target determination unit 52 is used, the processing steps of S118 and S120 in FIG. Fig. 9 can be omitted, which can simplify processing. Conversely, if the destination determination unit 52 is not used, the destination determination unit 52 can be omitted from the CPU module 36a, which can contribute to simplifying the system.

[0068] Furthermore, the above-described embodiment described an example in which, when displaying the coupling guide image 62 in contact with the road surface, an animation display is performed in which the coupling guide image is gradually extended rearward from a position near the towing device 18. In another embodiment, instead of executing the animation display, a coupling guide image 62 with a long length corresponding to, for example, 2.5 m rearward of the towing vehicle 10 may be displayed. In this case, the display control of the coupling guide image 62 is simplified, which can contribute to reducing the processing load of the CPU 36a.Moreover, in this case, it is possible to improve the recognition of the clutch guide image 62 by flashing the clutch guide image 62 for a predetermined period of time after displaying the clutch guide image 62 or by displaying the clutch guide image 62 with increased luminance, such as an animation display in which the clutch guide image 62 is extended.

[0069] Although the Fig. While the flowchart shown in FIG. 9 illustrates an example of performing levitation display processing on the coupling guide image 62, any other display image may be subjected to transition display processing. For example, a travel prediction line image (a display image extending in the vehicle longitudinal direction from the center position in the vehicle width direction of the tractor vehicle 10) indicating a predicted travel direction of the tractor vehicle 10 may be displayed in a floating manner, which may facilitate recognition of the travel direction of the tractor vehicle 10. Furthermore, in another embodiment, for example, the vehicle width guide line image 58 or the distance guide line image 60, which is an example of the display image, may be raised to the level of the rear bumper 16 and displayed.In this case, when the tractor 10 is reversing, it is easy to determine whether the tractor 10 is coming into contact with a high object. Furthermore, the parking frame guide line image indicating a parking target position at a distant position in the real image P1 can be displayed in a floating / raised manner and presented as an example of the display image. By floating and displaying the parking frame guide line image displayed at the distant position, it is possible to improve the recognition of the parking frame guide line image or the position of a parking space / parking gap.Furthermore, by presenting a levitation transition when hovering and displaying, for example, the driving prediction line image, the vehicle width guide line image 58, the distance guide line image 60, or the parking frame guide line image, it can be easily recognized by the driver that the display image to be displayed has height information and is displayed in a state where the height information is added. As a result, it is possible to reduce the erroneous recognition of the display content of the real image P1 displayed on the display device 26. Furthermore, it may be difficult to feel a sense of incompatibility with the display content.

[0070] Moreover, although the above-described embodiment described an example in which the silhouette image 64 is displayed even when the clutch guide image 62 is hovered, in another embodiment, the display of the silhouette image 64 may be omitted, or the silhouette image 64 may no longer be displayed after the guide line image is hovered. In this case, it is possible to prevent the display content of the real image P1 from becoming complicated. Whether or not to omit the silhouette image 64 can be selected by the driver, or the silhouette image 64 may be automatically omitted according to the display content of the real image P1.

[0071] The Fig. The flowchart shown in Figure 9 illustrates an example in which, when the clutch guide image 62 is displayed as a display image, the driver operates the switch to switch between display modes to display the clutch guide image. In another embodiment, for example, in the case of an assist mode in which the tow vehicle 12 is connected to the tractor vehicle 10, the display and levitation of the clutch guide image 62 can be started when a shift operation unit is operated to a reverse position, and the same effect can be achieved.

[0072] In addition, although the above-described embodiment described an example in which the tractor 10 is movable rearward, the display image (for example, the coupling guide image 62) accompanying a levitation transition is displayed on the real image P1 indicating the rear. In another embodiment, the display image accompanying a levitation transition may be displayed on the real image P1 indicating the front. In this case, the display image may be, for example, the vehicle width guide line image 58, the distance guide line image 60, or the travel prediction line image.

[0073] The Fig.5 to 8 and 10 illustrate a case where, when displaying the display image, the display of the display device 26 is configured with a dual screen including the real image P1 and the bird's-eye view image P2. In this case, it is easy for the driver to determine, for example, a traveling direction accompanying a levitation transition with the display image while grasping a situation around the own vehicle from the bird's-eye view image P2. As a result, it is possible to provide a heightened sense of safety during driving / travel, particularly during reversing driving / travel. Furthermore, in another embodiment, the bird's-eye view image P2 may be omitted. In this case, the display area of ​​the real image P1 can be enlarged, which can make it easier for the driver to recognize the display content from the display image.

[0074] The above-described embodiment described an example in which, when the first predetermined height is set to, for example, the road surface and the second predetermined height is set to, for example, the height of the hitch ball 18a, a display position of the hitch guide image 62 as a display image moves (levitates) from the first predetermined height to the second predetermined height, and a transition of the movement (levitation) is displayed. That is, a case was described in which the first predetermined height is lower than the second predetermined height. In another embodiment, the first predetermined height may be higher than the second predetermined height. In other words, the display image may be moved so as to descend from the first predetermined height to the second predetermined height.For example, when the display image is a driving prediction line image, the driving prediction line image is first drawn at a height (first predetermined height) corresponding to the height of the rear bumper 16, and then lowered toward the road surface (second predetermined height), for example. In this case, since the driving prediction line image is initially displayed at a high position, it is impossible to improve the driver's visibility. Furthermore, by displaying a downward transition of the driving prediction line image that descends from the position to the road surface, it is easy to draw the driver's attention to the road surface and grasp the road surface condition.In this case, shadow images can also be displayed as the display image (e.g., the predictive route line image or the first display image) at the first predetermined height, and the second display image at the second predetermined height. That is, the display is designed so that the shadow image descends from the predictive route line image and is ultimately projected onto the road surface. This can contribute to both clarifying the predicted direction of travel through the predictive route line image and increasing attention to the road surface through the shadow image.

[0075] Furthermore, according to the traveling direction (forward or reverse) of the vehicle (tractor 10) and the type of display image, it can be determined whether the display image should be displayed for levitation or descent. For example, in the case of displaying the coupling guide image 62, a levitation transition of the coupling guide image 62 may be displayed when the tractor 10 approaches the towing vehicle 12, and a descent transition of the coupling guide image 62 may be displayed when the tractor 10 moves far away from the towing vehicle 12. This further reduces the possibility of the coupling guide image 62 being displayed at an unnatural position with respect to the towing vehicle 12, which changes position relative to the tractor 10.Furthermore, for example, in the case of displaying the parking frame guide line image as a display image, a descent transition of the parking frame guide line image from the height of a reference position set in the vehicle (for example, the height corresponding to the height of the rear bumper 16) can be displayed when the vehicle (tractor vehicle 10) approaches a parking space, and a levitation transition of the parking frame guide line image from the height in contact with the road surface can be displayed when the vehicle moves far away from the parking space. In this case, when the vehicle is far away from the parking space, the parking frame guide line image is displayed in a state suspended from the road surface, which can improve the recognition of the position of the parking space / parking space.On the other hand, as the vehicle approaches the parking space, the parking frame guide line image approaches the road surface, which can improve the recognition of the size of the parking space or the like.

[0076] In this way, by switching the direction of the transition of the movement of the display image between levitation and descent, that is, by selecting the position of the first predetermined height and the position of the second predetermined height according to the type of display image and the state at the time of display, a sense of incompatibility is further reduced, and it is easy for the driver to recognize that the display image is an image containing altitude information. Furthermore, by presenting a levitation transition or a descent transition, such as the coupling line image, the driving prediction line image, the vehicle width guide line image, the distance guide line image, or the parking frame guide line image, as the display image, it can facilitate the driver's recognition of the displayed display image.

[0077] Moreover, although the present embodiment has described a case where the display image is a linear image, the display image is not limited to this and may be configured with a mark such as a circle or a rectangle, letters or numbers, and the same effect can be achieved.

[0078] A peripheral monitoring program executed by the CPU 36a of the present embodiment may be a file in an installable format or an executable format, and may be recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

[0079] Furthermore, the peripheral monitoring program can be provided by storing it in a computer connected to a network, such as the Internet, and downloading it over the network. Furthermore, the peripheral monitoring program implemented in the present embodiment can be configured to be provided or distributed over a network, such as the Internet.

[0080] Although the embodiments and variations of the present disclosure have been described, these embodiments and variations are provided only as examples and are not intended to limit the scope of this disclosure. These novel embodiments may be implemented in various other modes, and various omissions, substitutions, and changes may be made without departing from the spirit of the embodiments disclosed herein. These embodiments and variations thereof are included within the spirit or scope of this disclosure and belong to the appropriate range of this disclosure as defined in the claims.

[0081] The principles, preferred embodiment, and mode of operation of the present invention have been described in the foregoing specification. However, the invention to be protected is not to be construed as limited to the disclosed embodiments. Moreover, the embodiments described herein are to be considered as illustrative rather than restrictive. Variations and changes may be made by others and substituted equivalents without departing from the spirit of the present invention. Accordingly, it is expressly intended to embrace all variations, changes, and equivalents that fall within the spirit and scope of the present invention as defined by the claims.

Claims

[1] Peripheral monitoring device (100) comprising: a pickup unit (48) configured to acquire a peripheral image captured by an imaging unit (24, 24a to 24d) provided in a vehicle (10) and acquiring an image of a periphery of a vehicle, and a display image capable of superimposing the peripheral image and serving as an operation display of a driver when the driver performs a driving operation of the vehicle; and an image processing unit (50) configured to change the display image from a first predetermined height to a second predetermined height and to display a transition of the change. [2] The peripheral monitoring device according to claim 1, wherein the first predetermined height is lower than the second predetermined height. [3] The peripheral monitoring device according to claim 1, wherein the first predetermined height is higher than the second predetermined height. [4] A peripheral monitoring device according to claim 2, wherein the first predetermined height is a height in contact with a road surface on which the vehicle is present. [5] A peripheral monitoring device according to claim 3, wherein the second predetermined height is a height in contact with a road surface on which the vehicle is present. [6] A peripheral monitoring device according to any one of claims 1 to 3, wherein the first predetermined height or the second predetermined height is a height of a reference position set in the vehicle. [7] A peripheral monitoring device according to any one of claims 1 to 6, wherein the image processing unit changes a direction indicated by the display image / indicator image according to a current steering angle of the vehicle. [8] A peripheral monitoring device according to any one of claims 1 to 7, wherein the image processing unit displays a first display image / indicator image at the first predetermined height, and a second display image / indicator image at the second predetermined height side by side in a height direction. [9] A peripheral monitoring device according to any one of claims 1 to 8, wherein the image processing unit changes a transmittance at the time of displaying the display image / indicator image according to a shift state of the corresponding display image / indicator image. [10] A peripheral monitoring device according to any one of claims 1 to 9, further comprising a target determination unit (52) configured to determine a movement target position of the vehicle on the peripheral image / surrounding image, wherein the image processing unit determines the direction indicated by the display image / indicator image along which the vehicle reaches the movement target position.

Images