Peripheral monitoring device

The composite image is generated by the peripheral monitoring device, and the virtual vehicle images are displayed overlappingly at the position of the vehicle image, solving the problem that the driver finds difficult to recognize the relationship between the object and the vehicle position and improving driving safety.

CN112477758BActive Publication Date: 2024-12-03AISIN CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202010962500.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-09-12
Filing Date
2020-09-14
Publication Date
2024-12-03
Estimated Expiration
2040-09-14

AI Technical Summary

Technical Problem

The prior art is difficult to help the driver intuitively identify the positional relationship between the object and the vehicle when the vehicle detects an object that needs attention, resulting in difficulty in avoiding contact.

Method used

Design a peripheral monitoring device to generate a composite image by obtaining the current steering angle and surroundings of the vehicle, and the virtual vehicle image is displayed overlappingly at the position of the vehicle image. The virtual vehicle image can be moved, deformed and enlarged to help the driver identify objects Location and vehicle relationship.

Benefits of technology

It improves the convenience of the driver to identify the position of an object, reduces the risk of vehicle contact with objects, and enhances driving safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112477758B_ABST
    Figure CN112477758B_ABST
Patent Text Reader

Abstract

The present invention provides a surrounding monitoring device that allows a user to easily and intuitively grasp the positional relationship between the object and the vehicle when an object requiring attention is detected. The surrounding monitoring device of the embodiment includes, for example: an acquisition unit that acquires the current steering angle of the vehicle; an image acquisition unit that acquires a captured image from a capturing unit that captures the surroundings of the vehicle; a detection unit that acquires detection information of objects in the surroundings of the vehicle; and a control unit that causes a composite image to be displayed on a display unit, the composite image including a vehicle image representing the vehicle and a surrounding image representing the surroundings of the vehicle based on the captured image, and when an object is detected on a route where the vehicle advances a predetermined distance at the current steering angle, the control unit causes a virtual vehicle image representing the shape of the vehicle to be displayed in the composite image in an overlapping manner on the route to the object based on the position of the vehicle shown in the vehicle image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] An embodiment of the present invention relates to a periphery monitoring device. Background Art

[0002] A technology is proposed that detects objects (such as obstacles) around a vehicle based on an image captured by a camera of the vehicle's surroundings, calculates an expected path of the wheels, and reports if the object is on the expected path.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2008-90655

[0004] When an object that may come into contact with the vehicle (an object that requires attention) is discovered, if it is not only reported but also the positional relationship between the object and the vehicle while the vehicle continues to move in a state that the user (driver, etc.) can more intuitively recognize is provided, it will be easier to avoid contact more smoothly and safely. Summary of the invention

[0005] Therefore, one of the objects of the embodiment is to provide a surrounding monitoring device that allows a user to easily and intuitively understand the positional relationship between the object and a vehicle (host vehicle) when an object requiring attention is detected.

[0006] The surrounding monitoring device of the embodiment includes, for example, an acquisition unit that acquires the current steering angle of the vehicle; an image acquisition unit that acquires a captured image from a capturing unit that captures the surroundings of the vehicle; a detection unit that acquires detection information of an object in the surroundings of the vehicle; and a control unit that causes a composite image to be displayed on a display unit, the composite image including a vehicle image representing the vehicle and a surrounding image representing the surroundings of the vehicle based on the captured image, and when an object is detected on a route where the vehicle moves forward at the current steering angle for a predetermined distance, the control unit causes a virtual vehicle image representing the shape of the vehicle to be displayed in the composite image in a superimposed manner on the route to the object based on the position of the vehicle shown in the vehicle image. According to this structure, for example, a display can be performed so that the user can easily and intuitively recognize whether there is an object that needs attention on the route where the vehicle moves forward at the current steering angle, depending on whether the virtual vehicle image is displayed in the composite image.

[0007] In addition, in the surrounding monitoring device of the embodiment, the virtual vehicle image can be moved, for example, from the position of the vehicle toward the position of the object. According to this structure, for example, a display can be made that allows the driver to easily recognize the existence of the virtual vehicle image, in other words, a display can be made that makes it easy to recognize that there is an object that needs attention on the route. In addition, by moving the virtual vehicle image toward the object, a display that makes it easier to recognize the position of the object can be made.

[0008] In addition, the control unit of the surrounding monitoring device of the embodiment may change the display mode of the partial image that contacts the object in the virtual vehicle image, and stop the movement of the virtual vehicle image at the position where the vehicle contacts the object or at the position after the vehicle advances to the position just before the contact with the object in the synthetic image. According to this structure, for example, it is possible to make a display based on the virtual vehicle image so that the driver can easily recognize the position of the vehicle body that contacts the object or the position just before the contact, and thus it is easy to drive the vehicle. In addition, since the virtual vehicle image is stopped at the position that contacts the object or the position just before the contact, it is possible to make a display that makes it easier to recognize the position of the object, and it is possible to make a display that makes it easy to avoid the contact between the detected object and the vehicle.

[0009] In addition, the virtual vehicle image of the surrounding monitoring device of the embodiment may be, for example, an image representing the shape of the vehicle composed of polygons, and the partial image is a polygon of a portion of the polygons constituting the virtual vehicle image that is in contact with an object. According to this structure, for example, a display can be made based on the virtual vehicle image so that the driver can easily recognize the position in the vehicle body where an object may come into contact, thereby making it easier for the driver to drive the vehicle. In addition, a display can be made so that the contact between the detected object and the vehicle can be easily avoided.

[0010] In addition, the control unit of the surrounding monitoring device of the embodiment may change the display mode of the partial image, for example, according to the distance between the position of the object and the position of the vehicle shown in the virtual vehicle image. According to this structure, for example, by confirming the change in the display mode of the partial image, the positional relationship between the vehicle and the object can be grasped in more detail, which makes it easier for the driver to drive the vehicle without contacting the detected object.

[0011] In addition, the control unit of the surrounding monitoring device of the embodiment may change the display mode of the partial image to a first display mode indicating contact when the distance is below a predetermined value, and change the display mode from the first display mode to a second display mode indicating contact avoidance when the distance changes from being below the predetermined value to exceeding the predetermined value. According to this structure, for example, a display that makes it easy for the driver of the vehicle to judge whether there is a possibility of contact or non-contact with the object can be performed.

[0012] In addition, in the surrounding monitoring device of the embodiment, the vehicle image may be, for example, a bird's-eye view image of the vehicle. According to this configuration, for example, a display can be provided that allows the driver to easily and accurately understand the positional relationship between the vehicle and surrounding objects.

[0013] In addition, in the surrounding monitoring device of the embodiment, the virtual vehicle image may be, for example, an image showing a three-dimensional shape of the vehicle. According to this configuration, for example, a more realistic virtual vehicle image can be displayed on the display unit.

[0014] In addition, in the surrounding monitoring device of the embodiment, the virtual vehicle image is, for example, a semi-transparent image showing the shape of the vehicle. According to this structure, for example, it is possible to display the virtual vehicle image and the vehicle image so that the driver can easily distinguish them. In addition, it is possible to display the virtual vehicle image so that the driver can easily and intuitively recognize that the virtual vehicle image is an image showing the future position of the vehicle.

[0015] In addition, in the surrounding monitoring device of the embodiment, the virtual vehicle image is, for example, an image that emphasizes the outline of the vehicle. According to this structure, for example, it is possible to display the future position of the vehicle based on the virtual vehicle image so that the driver can easily recognize it. In addition, the portion overlapping with the virtual vehicle image can be visually confirmed, and it is easy to suppress the reduction in visibility of the surrounding conditions.

[0016] In addition, in the surrounding monitoring device of the embodiment, the virtual vehicle image is, for example, an image whose transmittance increases as it moves from the outline of the vehicle toward the inside. According to this structure, for example, a display based on the virtual vehicle image can be performed so that the driver can easily recognize the future position of the vehicle. In addition, the portion overlapping with the virtual vehicle image can be visually confirmed, and the visibility of the surrounding conditions can be easily suppressed.

[0017] In addition, the control unit of the surrounding monitoring device of the embodiment may, for example, display the composite image in an enlarged manner compared to the display state before the object is detected when the detection unit detects an object. According to this structure, for example, it is possible to display the enlarged display so that the driver can easily and intuitively recognize that an object has been detected and that he needs to pay attention to the surrounding conditions. In addition, it is possible to display the enlarged display so that the driver can easily confirm the details of the surrounding conditions and easily give a sense of security.

[0018] In addition, the control unit of the surrounding monitoring device of the embodiment may, for example, reduce the degree of magnification display when the vehicle is in a state where the vehicle can move backwards, compared to a state where the vehicle can move backwards. According to this structure, for example, when the vehicle is in a state where the vehicle can move backwards, the detection of the object can be easily and intuitively recognized by the magnified display, and the display range of the surroundings of the vehicle can be expanded compared to a state where the vehicle can move backwards, making it easy to confirm a wide range.

[0019] In addition, the control unit of the surrounding monitoring device of the embodiment may, for example, perform a staggered display when displaying the composite image in an enlarged manner, in which the display position of the vehicle image is staggered in the direction opposite to the turning direction based on the current steering angle. According to this configuration, for example, since the display area of ​​the turning direction is increased, it is possible to perform a display that allows the driver to easily understand the situation in the direction in which the driver will be heading from now on.

[0020] In addition, the control unit of the surrounding monitoring device of the embodiment may be configured not to perform the staggered display when the vehicle is in a state where the vehicle can move backwards, for example. According to this configuration, for example, when the vehicle is in a state where the vehicle can move backwards, a display can be performed that allows the driver to easily recognize the conditions related to the entire area around the vehicle.

[0021] In addition, the control unit of the surrounding monitoring device of the embodiment may restore the synthetic image to the display state before the object is detected, for example, when the virtual vehicle image is not displayed. According to this structure, for example, a display that allows the driver to easily recognize that the display of the virtual vehicle image has ended can be performed. In addition, a display that allows the driver to easily and intuitively recognize that the possibility of contact with the object has been reduced or eliminated can be performed.

[0022] In addition, the control unit of the surrounding monitoring device of the embodiment may, for example, temporarily hide the display of the virtual vehicle image, and cancel the hiding of the display when a predetermined condition is satisfied. According to this structure, for example, when the display of the virtual vehicle image is not necessary, it can be hidden, and the hiding state can be suppressed from continuing.

[0023] In addition, the control unit of the surrounding monitoring device of the embodiment may be configured to enable redisplay only during the non-display period, for example, when the display of the virtual vehicle image is set to be temporarily non-displayed. According to this configuration, for example, even when the virtual vehicle image is not displayed, it can be redisplayed as needed, which can improve the convenience of use.

[0024] In addition, it is also possible that: after the control unit of the surrounding monitoring device of the embodiment stops the movement of the virtual vehicle image at the position where the vehicle contacts the object or at the position close to the contact between the vehicle and the object, the steering angle is changed from the current steering angle, and it is set to be able to restart the movement of the virtual vehicle image in the direction of the changed steering angle when the vehicle is no longer in contact with the object. According to this structure, for example, it is possible to more clearly let the driver recognize the situation where the contact with the object can be eliminated by changing the steering angle, and the situation where the contact with the object has been eliminated. In addition, it is possible to display the area in the front so that it is easy to continue driving while paying attention to the possibility of contact with other objects. In addition, when it can be confirmed that there is no contact with the object, the virtual vehicle image may not be displayed after a specified period of time.

[0025] In addition, the surrounding monitoring device of the embodiment may also include, for example, an operating switch, and the operating switch can switch whether to display the virtual vehicle image on the display unit, and when no object is detected on the route when the vehicle moves forward a specified distance at the current steering angle, and when the virtual vehicle image can be displayed on the display unit according to the operation of the operating switch, the control unit changes the display mode of the vehicle image. According to this structure, for example, even in the state before the object is detected on the route, the display mode of the vehicle image can be changed, and the display is a display standby state of the virtual vehicle image. Therefore, when an object is detected on the route, it is possible to make it easy for the driver to recognize that it is a state of immediate notification through the virtual vehicle image, and it is easy to give a sense of security.

[0026] In addition, the control unit of the surrounding monitoring device of the embodiment may, for example, enlarge and display at least one of the length in the vehicle length direction and the length in the vehicle width direction of the virtual vehicle image compared to the corresponding length of the vehicle image when displaying a virtual vehicle image. According to this structure, for example, the size of the displayed virtual vehicle image is displayed larger than the vehicle image. In other words, when the vehicle is moved in consideration of an object (such as an obstacle), the virtual vehicle image can be displayed in a state including a safe and appropriate surplus distance (safety margin). As a result, a display that easily avoids contact with or excessive proximity to an object can be performed. In addition, conversely, a display that easily suppresses taking a surplus distance greater than necessary can be achieved.

[0027] In addition, the control unit of the surrounding monitoring device of the embodiment may, for example, only enlarge the length of the virtual vehicle image in the vehicle width direction. For example, when the vehicle is moved forward or turns, there is a tendency that the safety margin in the vehicle length direction is easy to identify, while on the other hand, the safety margin in the vehicle width direction is difficult to identify compared with the vehicle length direction. According to this structure, for example, when the vehicle is moved forward or turns, the minimum necessary, safe and appropriate margin distance in the vehicle width direction can be ensured, and a display that easily prompts the driver to a more appropriate movement path can be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is an illustrative and schematic perspective view showing a state in which a part of a vehicle compartment of a vehicle on which the surrounding monitoring device according to the embodiment is mounted is seen through.

[0029] Figure 2 It is an illustrative and schematic plan view of a vehicle equipped with the surrounding monitoring device according to the embodiment.

[0030] Figure 3 This is an exemplary and schematic block diagram showing a functional configuration of a control system of a vehicle including the surrounding monitoring device according to the present embodiment.

[0031] Figure 4 This is a block diagram schematically illustrating the configuration of a surrounding area monitoring device according to an embodiment.

[0032] Figure 5 This is an illustrative and schematic diagram of a display image of the surrounding monitoring device according to the first embodiment.

[0033] Figure 6 This is an illustrative and schematic diagram of another display screen of the surrounding monitoring device according to the first embodiment.

[0034] Figure 7 This is an illustrative and schematic diagram for explaining an example of a method for displaying a virtual vehicle image by the surrounding monitoring device according to the first embodiment.

[0035] Figure 8 This is an illustrative and schematic diagram for explaining a display method when a vehicle image is displayed in polygonal form by the surrounding monitoring device according to the first embodiment.

[0036] Fig. 9 This is an illustrative and schematic diagram for explaining a method of determining the colors of polygons constituting a virtual vehicle image by the surrounding monitoring device according to the first embodiment.

[0037] Fig.10It is an illustrative and schematic diagram for explaining the display of a virtual vehicle image by the surrounding monitoring device according to the first embodiment.

[0038] Fig.11 This is an illustrative and schematic diagram for explaining a process of emphasizing and displaying a partial image by the surrounding monitoring device according to the first embodiment.

[0039] Fig.12 This is an illustrative and schematic diagram for explaining a process of emphasizing and displaying a partial image by the surrounding monitoring device according to the first embodiment.

[0040] Fig.13 This is an illustrative and schematic diagram of a display screen of the surrounding monitoring device according to the second embodiment.

[0041] Fig.14 This is an illustrative and schematic diagram of another display screen of the surrounding monitoring device according to the second embodiment.

[0042] Fig.15 Is based on Fig.14 An illustrative and schematic diagram of the display of a proximity indicator of a peripheral monitoring device is shown.

[0043] Fig.16 Is based on Fig.14 An exemplary and schematic diagram of other displays of a proximity indicator of a perimeter monitoring device is shown.

[0044] Fig.17 This is an illustrative and schematic diagram of a display screen of the surrounding monitoring device according to the third embodiment.

[0045] Fig.18 It is an exemplary and schematic diagram showing how the display mode of the display screen of the surrounding monitoring device according to the third embodiment is changed between a state in which the vehicle can move backward and another state.

[0046] Fig.19 This is a schematic diagram showing an example of a display screen capable of hiding and redisplaying a virtual vehicle image according to the surrounding monitoring device according to the third embodiment.

[0047] Fig. 20 It is a schematic diagram showing an example of a display screen corresponding to the operation state of a main switch (operation switch) for selecting whether or not to display a vehicle image in the surrounding area monitoring device according to the present embodiment.

[0048] Fig.21 This is an illustrative and schematic diagram showing a display image in which the size of a virtual vehicle image is changed when it is displayed in the surrounding monitoring device according to the present embodiment to ensure a margin distance (safety margin) to an object (for example, an obstacle).

[0049] Fig. 22 This is an illustrative and schematic diagram of a display screen of a surrounding monitoring device according to a modification.

[0050] Explanation of the reference numerals: 1…vehicle, 8…display device, 10…operation input unit, 10a…main switch, 14…ECU, 14a…CPU, 14b…ROM, 14c…RAM, 14d…display control unit, 14f…SSD, 15…shooting unit, 16…radar, 17…sonar, 400…image acquisition unit, 401…acquisition unit, 402…detection unit, 403…control unit. DETAILED DESCRIPTION

[0051] The following discloses an exemplary embodiment of the present invention. The structure of the embodiment shown below, and the action, result and effect brought about by the structure are an example. The present invention can be implemented by structures other than those disclosed in the following embodiments, and at least one of the various effects and derived effects based on the basic structure can be obtained.

[0052] (Description of the overall system)

[0053] Figure 1 This is an illustrative and schematic stereoscopic view of a state of a portion of a vehicle compartment 2a of a vehicle 1 equipped with a surrounding monitoring device according to an embodiment. The vehicle equipped with the surrounding monitoring device according to this embodiment may be a vehicle that uses an internal combustion engine (engine) as a driving source (internal combustion engine vehicle), a vehicle that uses an electric motor (motor) as a driving source (electric vehicle, fuel cell vehicle, etc.), or a vehicle that uses both of them as driving sources (hybrid vehicle). In addition, the vehicle may be equipped with various transmission devices, various devices (systems, components, etc.) required for driving an internal combustion engine or an electric motor. In addition, the method, number, layout, etc. of the devices related to the driving of the wheels in the vehicle may be variously set.

[0054] like Figure 1 As shown in the example, the vehicle body 2 of the vehicle 1 constitutes a compartment 2a for passengers (not shown) to ride in. In the compartment 2a, a steering control unit 4, an acceleration control unit 5, a brake control unit 6, a speed change control unit 7, etc. are provided in a state facing a seat 2b for a driver as a passenger. The steering control unit 4 is, for example, a steering wheel protruding from a dashboard 24. The acceleration control unit 5 is, for example, an accelerator pedal located under the driver's feet. The brake control unit 6 is, for example, a brake pedal located under the driver's feet. The speed change control unit 7 is, for example, a shift lever protruding from a center console.

[0055] In addition, a display device (display unit) 8 and a sound output device 9 as a sound output unit are provided in the vehicle compartment 2a. The display device 8 is, for example, an LCD (liquid crystal display), an OELD (organic electroluminescent display), etc. The sound output device 9 is, for example, a speaker. In addition, the display device 8 is covered by a transparent operation input unit 10 such as a touch panel. Passengers (users) can visually confirm the image displayed on the display screen of the display device 8 through the operation input unit 10. In addition, passengers can operate the operation input unit 10 by touching, pressing or moving the position corresponding to the image displayed on the display screen of the display device 8 with a finger, etc., thereby performing operation input. These display device 8, sound output device 9, operation input unit 10, etc. are provided, for example, on a monitor device 11 located in the center of the vehicle width direction, i.e., the left-right direction, of the instrument panel. The monitor device 11 can have an operation input unit not shown, such as a switch, a dial, a joystick, a button, etc. Furthermore, a sound output device (not shown) may be provided at another position in the vehicle compartment 2a different from the monitor device 11, and sound may be output from a sound output device different from the sound output device 9 of the monitor device 11. The monitor device 11 may also serve as a navigation system or an audio system, for example.

[0056] Figure 2 FIG. 1 is an illustrative and schematic top view of a vehicle 1 equipped with the surrounding monitoring device according to the present embodiment. Figure 1 as well as Figure 2 As shown, the vehicle 1 is a four-wheeled vehicle or the like, and has two left and right front wheels 3F and two left and right rear wheels 3R. All or part of the four wheels 3 are steerable.

[0057] For example, four photographing units 15a to 15d are provided on the vehicle body 2 as a plurality of photographing units 15. The photographing unit 15 is, for example, a digital camera with built-in photographing elements such as CCD (charge coupled device) and CIS (CMOS image sensor). The photographing unit 15 can output video data at a specified frame rate. The photographing units 15 respectively have a wide-angle lens or a fisheye lens, and can capture a range of 140° to 220° in the horizontal direction, for example. In addition, the optical axis of the photographing unit 15 is set to face obliquely downward. Thus, the photographing unit 15 can sequentially capture the external environment around the vehicle body 2, including the road surface on which the vehicle 1 can move and the area where the vehicle 1 can park, and output it as captured image data.

[0058] The photographing unit 15a is, for example, located at the end 2e on the rear side of the vehicle body 2, and is provided on the wall below the rearview window of the trunk door 2h, so as to photograph the condition of the rear area of ​​the vehicle 1. The photographing unit 15b is, for example, located at the end 2f on the right side of the vehicle body 2, and is provided on the right rearview mirror 2g, so as to photograph the condition of the area including the right front, right side, and right rear of the vehicle 1. The photographing unit 15c is, for example, located at the front side of the vehicle body 2, that is, the end 2c on the front side in the front-to-rear direction of the vehicle, and is provided on the front bumper, etc., so as to photograph the condition of the area in front of the vehicle 1. The photographing unit 15d is, for example, located at the left side of the vehicle body 2, that is, the end 2d on the left side in the vehicle width direction, and is provided on the rearview mirror 2g as a protruding portion on the left side, so as to photograph the condition of the area including the left front, left side, and left rear of the vehicle 1. The ECU 14 (refer to Figure 3 ) can perform calculation processing and image processing based on the captured image data obtained by multiple imaging units 15 to generate an image with a wider field of view, or generate a virtual bird's-eye view image of the vehicle 1 observed from above.

[0059] In addition, the vehicle 1 has a plurality of radars 16 as distance measuring units capable of measuring the distance to an object existing outside the vehicle 1. The radar 16 is, for example, a millimeter wave radar, etc., and can measure the distance to an object existing in the traveling direction of the vehicle 1. In the present embodiment, the vehicle 1 has a plurality of radars 16a to 16d. The radar 16a is, for example, disposed at the end portion of the left side of the rear bumper of the vehicle 1, and can measure the distance to an object existing to the left rear of the vehicle 1. In addition, the radar 16b is disposed at the end portion of the right side of the rear bumper of the vehicle 1, and can measure the distance to an object existing to the right rear of the vehicle 1. The radar 16c is disposed at the end portion of the right side of the front bumper of the vehicle 1, and can measure the distance to an object existing to the right front of the vehicle 1. In addition, the radar 16d is disposed at the end portion of the left side of the front bumper of the vehicle 1, and can measure the distance to an object existing to the left front of the vehicle 1.

[0060] In addition, the vehicle 1 has a sonar 17 that can measure the distance to an external object that is relatively close to the vehicle 1 using ultrasonic waves. In the present embodiment, the vehicle 1 has a plurality of sonars 17a to 17h. The sonars 17a to 17d are provided on the rear bumper of the vehicle 1 and can measure the distance to an object that is behind the vehicle. The sonars 17e to 17h are provided on the front bumper of the vehicle 1 and can measure the distance to an object that is in front of the vehicle 1.

[0061] Figure 3 1 is an exemplary and schematic block diagram showing the functional configuration of a control system 100 of a vehicle 1 including the surrounding monitoring device according to the present embodiment. Figure 3As shown in the example, in the control system 100, in addition to the ECU 14, the monitoring device 11, the steering system 13, the radar 16, the sonar 17, etc., the brake system 18, the steering angle sensor 19, the acceleration sensor 20, the shift sensor 21, the wheel speed sensor 22, the GPS (Global Positioning System) receiver 25, the direction indicator 26, etc. are electrically connected via the in-vehicle network 23 as an electrical communication line. The in-vehicle network 23 is configured as a CAN (controller area network), for example. The ECU 14 can control the steering system 13, the brake system 18, etc. by sending a control signal via the in-vehicle network 23. In addition, the ECU 14 can receive the detection results of the torque sensor 13b, the brake sensor 18b, the steering angle sensor 19, the radar 16, the sonar 17, the acceleration sensor 20, the shift sensor 21, the wheel speed sensor 22, the GPS receiver 25, etc., and the operation signal of the operation input unit 10 or the operation lever of the direction indicator 26, etc. via the in-vehicle network 23.

[0062] The steering system 13 is an electric power steering system, a SBW (Steer By Wire) system, etc. The steering system 13 includes an actuator 13a and a torque sensor 13b. The steering system 13 is electrically controlled by the ECU 14, etc., so that the actuator 13a is actuated to add torque to the steering unit 4 to supplement the steering force, thereby turning the wheels 3. The torque sensor 13b detects the torque applied by the driver to the steering unit 4, and sends the detection result to the ECU 14.

[0063] The brake system 18 includes an ABS (Anti-lock Brake System) for controlling the locking of the brakes of the vehicle 1, an anti-skid device (ESC: Electronic Stability Control) for suppressing the side slip of the vehicle 1 when turning, an electric brake system for enhancing the braking force to assist the brake, and BBW (Brake By Wire). The brake system 18 has an actuator 18a and a brake sensor 18b. The brake system 18 is electrically controlled by the ECU 14, etc., and applies a braking force to the wheel 3 via the actuator 18a. The brake system 18 detects the locking of the brake, the idling of the wheel 3, and the signs of side slip based on the rotation difference between the left and right wheels 3, etc., and performs control to suppress the locking of the brake, the idling of the wheel 3, and the side slip. The brake sensor 18b is a displacement sensor that detects the position of the brake pedal, which is a movable part of the brake operating unit 6, and sends the detection result of the position of the brake pedal to the ECU 14.

[0064] The steering angle sensor 19 is a sensor for detecting the steering amount of the steering control unit 4 such as a steering wheel. The steering angle sensor 19 is composed of a Hall element or the like, detects the rotation angle of the rotating part of the steering control unit 4 as the steering amount, and sends the detection result to the ECU 14. The acceleration sensor 20 is a displacement sensor for detecting the position of the accelerator pedal as the movable part of the acceleration operation unit 5, and sends the detection result to the ECU 14.

[0065] The shift sensor 21 is a sensor that detects the position of the movable part (lever, arm, button, etc.) of the speed change operating part 7, and sends its detection result to the ECU 14. The wheel speed sensor 22 has a Hall element, etc., and is a sensor that detects the rotation amount of the wheel 3 or the number of revolutions of the wheel 3 per unit time, and sends its detection result to the ECU 14. The GPS receiver 25 obtains the current position of the vehicle 1 based on the radio waves received from the artificial satellite. The direction indicator 26 is provided, for example, on the left and right rearview mirrors 2g. The operation signal of the movable part (operating lever, etc.) operated by the driver is sent to the ECU 14, and is sent to the direction indicator 26, which notifies the surroundings of the direction in which the vehicle 1 is about to turn by flashing or lighting.

[0066] In addition, the structures, arrangements, electrical connection methods, etc. of the various sensors and actuators described above are merely examples and can be set (changed).

[0067] The ECU 14 is composed of a computer and the like, and is responsible for overall control of the vehicle 1 through the cooperation of hardware and software. Specifically, the ECU 14 includes a CPU (central processing unit) 14a, a ROM (read only memory) 14b, a RAM (random access memory) 14c, a display control unit 14d, a sound control unit 14e, and an SSD (solid state drive) 14f, etc. The CPU 14a, the ROM 14b, and the RAM 14c may be provided in the same circuit board.

[0068] The CPU 14a can read out a program installed and stored in a non-volatile storage device such as the ROM 14b, and perform arithmetic processing according to the program. For example, the CPU 14a can perform arithmetic and control of image processing related to the image displayed on the display device 8. In addition, the CPU 14a can perform a distortion correction process in which the image data (curved image data) of the wide-angle image obtained by the imaging unit 15 is subjected to arithmetic processing and image processing to correct the distortion, or generate a bird's-eye view image (surrounding image) in which, for example, a vehicle image (vehicle icon) representing the vehicle 1 is displayed at the center position based on the image data captured by the imaging unit 15, and display it on the display device 8. In addition, when generating a bird's-eye view image, the CPU 14a can change the position of the virtual viewpoint to generate a bird's-eye view image such as a bird's-eye view image facing the vehicle image from directly above, or a bird's-eye view image such as a bird's-eye view image facing the vehicle image from an oblique direction.

[0069] The CPU 14a can perform the following processing: when an object is detected on the route when the vehicle 1 advances a predetermined distance at the current steering angle by analyzing the captured image data, analyzing the distance measurement data based on the radar 16 or the sonar 17, etc., a virtual vehicle image that can move based on the position of the vehicle 1 shown in the vehicle image and represents the shape of the vehicle 1 is displayed in an overlapping manner in the bird's-eye view image. In addition, in other embodiments, when an object is detected on the route when the vehicle 1 advances a predetermined distance at the current steering angle, the CPU 14a may display the virtual vehicle image in an overlapping manner between the position of the object on the route and the position of the vehicle 1 shown in the vehicle image in the bird's-eye view image (in the synthetic image described later). In other words, when an object is detected, a display may be performed such that the virtual vehicle image appears at a predetermined position on the route (for example, a position equivalent to 2 meters in front of the vehicle 1). In addition, the route when the vehicle 1 advances a predetermined distance does not need to be completely consistent with the actual movement trajectory of the vehicle 1, and for example, it may be set to a route with a predetermined amount of margin (error or margin).

[0070] ROM14b stores various programs and parameters required for the execution of the programs. RAM14c temporarily stores various data used for calculations in CPU14a. Display control unit 14d mainly performs image processing of image data obtained from camera 15 and output to CPU14a in the calculation processing of ECU14, converts image data obtained from CPU14a into image data for display on display device 8, etc. Sound control unit 14e mainly performs sound processing obtained from CPU14a and output to sound output device 9 in the calculation processing of ECU14. SSD14f is a rewritable non-volatile storage unit, which continues to store data obtained from CPU14a even when the power supply of ECU14 is disconnected. In addition, CPU14a, ROM14b, RAM14c, etc. can be integrated in the same package. In addition, ECU14 can also be a structure using other logic operation processors such as DSP (digital signal processor) or logic circuits instead of CPU14a. In addition, a HDD (hard disk drive) may be provided instead of the SSD 14 f , or the SSD 14 f or the HDD may be provided separately from the ECU 14 .

[0071] Figure 4 1 is a block diagram schematically showing a configuration in which the surrounding monitoring device according to the embodiment is implemented by the CPU 14a. Figure 4 As shown, the CPU 14a includes an image acquisition unit 400, an acquisition unit 401, a detection unit 402, and a control unit 403. The CPU 14a implements the functions of the image acquisition unit 400, the acquisition unit 401, the detection unit 402, and the control unit 403 by executing a peripheral monitoring program stored in a storage medium such as the ROM 14b or the SSD 14f. In addition, a part or all of the image acquisition unit 400, the acquisition unit 401, the detection unit 402, and the control unit 403 may be constituted by hardware such as a circuit. Figure 4 Although not shown in the figure, the CPU 14 a can also realize various modules required for the driving of the vehicle 1 .

[0072] The image acquisition unit 400 acquires an image of the surroundings of the vehicle 1 acquired by the image acquisition unit 15. The acquisition unit 401 acquires the current steering angle of the vehicle 1. The acquisition unit 401 acquires the steering amount detected by the steering angle sensor 19 as the current steering angle of the vehicle 1.

[0073] The detection unit 402 can obtain detection information of objects that may come into contact with the vehicle 1. Specifically, the detection unit 402 detects objects that may come into contact with the vehicle 1 on the route when the vehicle 1 moves forward a predetermined distance at the current steering angle based on the captured image obtained by the imaging unit 15 capturing the traveling direction of the vehicle 1, the distance measured by the radar 16 (the distance between the vehicle 1 and the object existing in the traveling direction of the vehicle 1), etc. The detection unit 402 can obtain (detect) detection information of both stationary objects and moving objects that approach the vehicle 1 and may come into contact with it as objects that may come into contact with the vehicle 1.

[0074] For example, the detection unit 402 detects an object that may come into contact with the vehicle 1 by performing image processing (e.g., optical flow) on the captured image obtained by the imaging unit 15. Alternatively, the detection unit 402 detects an object that may come into contact with the vehicle 1 based on a change in the distance measured by the radar 16.

[0075] The detection unit 402 detects an object that may come into contact with the vehicle 1 based on the captured image obtained by the imaging unit 15 or the distance measurement result of the radar 16, but may also detect an object that exists at a relatively close distance to the vehicle 1. In this case, the object that may come into contact with the vehicle 1 can be detected based on the distance measurement result of the sonar 17.

[0076] The detection unit 402 transitions to an operating state (ON) or an inoperative state (OFF) according to the operation of a detection start switch (not shown) of the vehicle 1. Here, the operating state is a state of detecting an object in contact with the vehicle 1. On the other hand, the inoperative state is a state of not detecting an object in contact with the vehicle 1.

[0077] The detection unit 402 is transferred to the operating state or the non-operating state according to the operation of the detection start switch, but is not limited to this. For example, when the speed of the vehicle 1 based on the detection result of the rotation speed of the wheel 3 by the wheel speed sensor 22 becomes below a preset speed (for example, 12 km / h), the detection unit 402 automatically (independent of the operation of the detection start switch) transfers to the operating state. In addition, when the speed of the vehicle 1 is faster than the preset speed, the detection unit 402 automatically (independent of the operation of the detection start switch) transfers to the non-operating state.

[0078] The control unit 403 displays a display screen including a captured image obtained by capturing the traveling direction of the vehicle 1 by the capturing unit 15 and a composite image including a vehicle image and a surrounding image on the display device 8 via the display control unit 14d. In the present embodiment, the control unit 403 displays a display screen including a composite image and a captured image on the display device 8, but it is sufficient to display a display screen including at least a composite image on the display device 8.

[0079] Here, the vehicle image is an image representing the vehicle 1. The vehicle image is a bird's-eye view image of the vehicle 1 obtained by looking down on it from above. In this way, the positional relationship between the vehicle 1 and the objects around it can be accurately understood. The vehicle image can be an image in the form of a bitmap, or an image representing the shape of the vehicle composed of multiple polygons. Here, the vehicle image composed of multiple polygons refers to the three-dimensional shape of the vehicle 1 expressed by multiple polygons (for example, triangular polygons).

[0080] The surrounding image is an image showing the surroundings (surroundings) of the vehicle 1, which is generated based on the captured image obtained by capturing the surroundings of the vehicle 1 by the capturing unit 15. The surrounding image is a bird's-eye view image obtained by looking down from above at the surroundings (surroundings) of the vehicle 1. The surrounding image is a bird's-eye view image of the surroundings of the vehicle 1 centered on, for example, the center of the rear wheel axle of the vehicle image.

[0081] In addition, when the detection unit 402 detects an object on the route when the vehicle 1 moves forward at the current steering angle for a predetermined distance, the control unit 403 causes a virtual vehicle image that can move based on the position shown in the vehicle image and represents the shape of the vehicle 1 to be displayed in an overlapping manner in the synthetic image. For example, the image acquisition unit 400 can display the virtual vehicle image by moving the predetermined distance from the position of the vehicle image along the route based on the current steering angle. On the other hand, when the detection unit 402 does not detect an object on the route when the vehicle 1 moves forward at the current steering angle for a predetermined distance, the control unit 403 causes the virtual vehicle image not to be displayed. Thus, the driver of the vehicle 1 can easily recognize whether there is an object that may be contacted on the route when the vehicle 1 moves forward at the current steering angle based on whether the virtual vehicle image is included on the display screen displayed on the display device 8. In addition, as described above, when the detection unit 402 detects an object on the route when the vehicle 1 moves forward at the current steering angle for a predetermined distance, the control unit 403 causes the virtual vehicle image to appear at a predetermined position on the route.

[0082] Here, the predetermined distance is a pre-set distance, for example, 1.0 to 2.0 m, but the setting can be changed appropriately. In addition, the current steering angle is the steering angle at the current position of the vehicle 1. The control unit 403 obtains the steering angle obtained by the acquisition unit 401 as the steering angle at the current position of the vehicle 1.

[0083] In addition, the virtual vehicle image can be a virtual image composed of a plurality of polygons that represents the shape of the vehicle 1. The virtual vehicle image composed of a plurality of polygons refers to the shape of the three-dimensional vehicle 1 (the three-dimensional shape of the vehicle 1) displayed by a plurality of polygons (for example, triangular polygons). Thus, a more realistic virtual vehicle image can be displayed on the display device 8.

[0084] The control unit 403 includes an image representing the shape of the vehicle 1 composed of a plurality of polygons as a virtual vehicle image in the synthesized image. Alternatively, for example, an image representing the shape of the vehicle 1 in a bitmap format may be included as a virtual vehicle image in the synthesized image.

[0085] When the detector 402 is in an operating state and the shift sensor 21 detects that the position of the speed change operation unit 7 is in the D gear, and an object is detected on the route when the vehicle 1 moves forward a predetermined distance at the current steering angle, the control unit 403 displays a virtual vehicle image in front of the vehicle 1. For example, the virtual vehicle image is moved in a manner separated from the vehicle image along the route determined by the current steering angle. In this way, it is possible to realize a display that allows the driver to easily recognize an object on the route in front of the vehicle 1.

[0086] Similarly, when the detector 402 is in the operating state and the shift sensor 21 detects that the position of the speed change operation unit 7 is in the R position, and when an object is detected on the route when the vehicle 1 is backing up a predetermined distance at the current steering angle, the control unit 403 displays a virtual vehicle image behind the vehicle 1. In this way, it is possible to realize a display that allows the driver to easily recognize an object on the route behind the vehicle 1.

[0087] On the other hand, when the detection unit 402 does not detect a contacting object on the route when the vehicle 1 advances a specified distance, the control unit 403 does not display the virtual vehicle image in the synthetic image. That is, the control unit 403 can realize a display indicating that there is no object that may come into contact with the vehicle 1 on the route when the vehicle 1 advances a specified distance at the current steering angle. In addition, although it will be described later, there is a situation where there is no object on the route when the virtual vehicle image is already displayed, for example, when the steering angle is changed or the object itself moves. In this case, the virtual vehicle image may not be displayed immediately, but may be displayed after a specified period of time. By displaying the virtual vehicle image for a specified period of time, it is possible to avoid a complex display such as repeatedly switching between non-display and display of the virtual vehicle image in a short period of time when there are objects around the route.

[0088] In addition, when the detection unit 402 detects an object in contact with the vehicle 1, the control unit 403 may change the display mode of the image of the portion in contact with the detected object (hereinafter referred to as the partial image) in the virtual vehicle image. In addition, the control unit 403 may also change the display mode of the partial image to be different from other portions of the virtual vehicle image by flashing the partial image, changing the display color, or emphasizing the outline of the partial image. In addition, when the virtual vehicle image is composed of polygons, the control unit 403 can determine the polygon of the portion in contact with the object among the polygons constituting the virtual vehicle image as the partial image. Moreover, the control unit 403 may also change the display mode of the determined polygon.

[0089] Thus, it is possible to display a display that allows the driver of the vehicle 1 to easily recognize which part of the vehicle body of the vehicle 1 contacts the object when the vehicle is traveling at the current steering angle from the displayed virtual vehicle image. As a result, it is possible to display a driving operation that is easy to perform to avoid contact between the detected object and the vehicle 1.

[0090] In addition, when an object in contact with the vehicle 1 is detected, the control unit 403 can move the virtual vehicle image to the contact position where the vehicle 1 contacts the object in the synthetic image, and then stop the virtual vehicle image from the contact position. In other words, when the distance is less than the predetermined distance at which the virtual vehicle image can be moved, the control unit 403 also stops the virtual vehicle image at the contact position with the object. When an object in contact with the vehicle 1 is detected, the control unit 403 does not move the virtual vehicle image from the contact position where the vehicle 1 contacts the object in the synthetic image, but fixes it, but the present invention is not limited thereto. For example, the control unit 403 can also stop the movement of the virtual vehicle image at a position before the contact position, and does not move the virtual vehicle image from the position, but fixes it. That is, the control unit 403 overlaps and displays the virtual vehicle image at the contact position where the vehicle 1 contacts the object or the position when the virtual vehicle image moves forward to the position before the contact, as the position when the vehicle 1 moves forward. As a result, the driver of the vehicle 1 can easily recognize at which position on the route based on the current steering angle the vehicle 1 contacts the object.

[0091] In addition, it is assumed that after detecting an object in contact with the vehicle 1, the virtual vehicle image is fixed (stopped) at the contact position, the current steering angle is changed to change the direction of travel of the vehicle 1, and the object in contact with the vehicle 1 is not detected (no contact with the object) on the new route by the detection unit 402. In this case, the control unit 403 can also release the fixation of the virtual vehicle image that was fixed (stopped) at the contact position before the steering angle was changed. In addition, the control unit 403 can also restart the movement of the virtual vehicle image along the route according to the changed steering angle. As a result, the control unit 403 can move the position of the virtual vehicle image in the synthetic image along with the change of the steering angle of the vehicle 1. As a result, it is possible to perform a display that allows the driver to more clearly recognize that the contact with the object can be eliminated by changing the steering angle, and the possibility of eliminating the contact. In addition, it is easy to continue the driving of the vehicle 1 while paying attention to the possibility of contact with other objects in the previous area. In addition, the virtual vehicle image can also be not displayed when it can be confirmed that the vehicle 1 is not in contact with the object by changing the steering angle or moving the object (moving body). In this case, for example, the virtual vehicle image may not be displayed after a predetermined period, for example, after a predetermined time has passed or after a predetermined distance has been traveled, has passed since it was confirmed that the vehicle 1 is not in contact with the object.

[0092] Furthermore, when the detector 402 detects an object approaching the path of the vehicle 1 as it moves forward a predetermined distance, the controller 403 may display an approaching object indicator (direction indicator) relative to the traveling direction of the vehicle 1 shown in the vehicle image in the composite image. Thus, the driver of the vehicle 1 can easily recognize the existence of an object approaching the vehicle 1 and its approaching direction based on whether the display screen displayed on the display device 8 includes the approaching object indicator based on the display screen displayed on the display device 8.

[0093] Here, the approaching object indicator is an indicator that can identify the direction in which an object is gradually approaching the vehicle 1 (hereinafter referred to as the approaching direction). The approaching object indicator is, for example, an arrow indicating the approaching direction. In addition, the approaching object indicator is an indicator that can identify the approaching direction of a moving body among objects that may come into contact with the vehicle 1.

[0094] Furthermore, when an object approaching the vehicle 1 is detected by the detector 402, the controller 403 changes the display mode of the approaching object indicator. Thus, the driver of the vehicle 1 can easily recognize from which direction the object that may come into contact with the vehicle 1 is approaching by visually confirming the approaching object indicator whose display mode has been changed. The controller 403 can also change the color of the approaching object indicator or make it flash so that the display mode of the approaching object indicator is different when an object approaching the vehicle 1 is not detected and when an object approaching the vehicle 1 is detected.

[0095] In the above example, when a stationary object is detected as an object that may come into contact with the vehicle 1, the control unit 403 changes the display mode of a portion of the image in the virtual vehicle image. When a moving object is detected as an object that may come into contact with the vehicle 1, the control unit 403 changes the display mode of the approaching object indicator. In other embodiments, when a moving object is detected as an object that may come into contact with the vehicle 1, the display mode of a portion of the image in the virtual vehicle image may be changed. In this case, the control unit 403 may include the approaching object indicator in the synthetic image, or may not include the approaching object indicator in the synthetic image.

[0096] Next, use Figure 5 to Figure 12 , a specific example of a display screen displayed on the display device 8 by the control unit 403 is described.

[0097] (First Embodiment)

[0098] Figure 5 FIG. 1 is an illustrative and schematic diagram showing a display screen of the surrounding monitoring device according to the first embodiment. Figure 5, the display processing of the display screen when the position of the speed change operation part 7 is detected to be in the D position by the shift sensor 21 is described. In the first embodiment, the control unit 403 displays the display screen G on the display device 8, and the display screen G includes a composite image G3 including the vehicle image G1 and the surrounding image G2, and a captured image G4 obtained by capturing the moving direction of the vehicle 1 (for example, the front of the vehicle 1) by the capturing unit 15.

[0099] Furthermore, when the detection unit 402 is in an operating state and an object is detected on the route when the vehicle 1 advances a predetermined distance at the current steering angle acquired by the acquisition unit 401, the control unit 403 causes the virtual vehicle image G5 to be displayed in a superimposed manner in the surrounding image G2 at a position P2 where the vehicle 1 advances a predetermined distance at the steering angle of the current position P1 based on the position represented by the vehicle image G1. Here, as Figure 5 As shown, the virtual vehicle image G5 is a semi-transparent image showing the shape of the vehicle 1. Thus, the virtual vehicle image G5 can be easily distinguished from the vehicle image G1, and the driver of the vehicle 1 can intuitively recognize that the virtual vehicle image G5 is an image showing the future position P2 of the vehicle 1.

[0100] In addition, the control unit 403 may also display an image whose transmittance increases as it moves from the outline of the vehicle 1 toward the inside as the virtual vehicle image G5. This display makes it easier to distinguish the virtual vehicle image G5 from the vehicle image G1, and allows the driver of the vehicle 1 to intuitively recognize that the virtual vehicle image G5 is an image representing the future position of the vehicle 1.

[0101] Furthermore, the control unit 403 may also display the outline of the virtual vehicle image G5 in a different display mode (e.g., different color, flashing, overlapping of frame lines) from other parts of the virtual vehicle image G5 to emphasize the outline. Thus, the visibility of the virtual vehicle image G5 is improved, and the display can be performed so that the driver can easily recognize the future position of the vehicle 1.

[0102] By displaying the virtual vehicle image G5 semi-transparently or in outline in this way, even the portion (synthesized image G3) overlapping with the virtual vehicle image G5 can be visually confirmed, and it is easy to suppress a reduction in visibility of the surrounding situation.

[0103] In addition, when the detector 402 is in an operating state, Figure 5As shown, the control unit 403 displays the approaching object indicator G6 in the traveling direction of the vehicle image G1 (e.g., a position about 1 m in front of the vehicle 1) in the surrounding image G2. At this time, if the detector 402 does not detect an object, the control unit 403 displays the approaching object indicator G6 in a gray scale.

[0104] Figure 6 4 is a diagram showing another display example of the display screen of the surrounding monitoring device according to the first embodiment. When the detection unit 402 does not detect an object on the route when the vehicle 1 moves forward at the current steering angle for a predetermined distance, the control unit 403 does not display the virtual vehicle image G5 in the surrounding image G2. On the other hand, when the detection unit 402 detects an object O (e.g., a wall, a base of a wall) in contact with the vehicle 1 on the route when the vehicle 1 moves forward at the current steering angle for a predetermined distance, the control unit 403 does not display the virtual vehicle image G5 in the surrounding image G2. Figure 6 As shown, the control unit 403 moves the virtual vehicle image G5 from the vehicle image G1 to the contact position P3 with the detected object O in the surrounding image G2. Thereafter, even if the vehicle 1 moves while maintaining the current steering angle, the control unit 403 does not move the virtual vehicle image G5 from the contact position P3 but keeps it fixed.

[0105] At this time, if Figure 6 As shown, the control unit 403 makes the display mode of the partial image PG in contact with the detected object O in the virtual vehicle image G5 different from the display mode of other parts of the virtual vehicle image G5. For example, when the distance between the object O and the virtual vehicle image G5 is less than a predetermined value, the display mode of the partial image PG is changed to a first display mode indicating contact. For example, the control unit 403 displays the partial image PG in red and displays the parts other than the partial image PG in the virtual vehicle image G5 in white. As a result, the driver of the vehicle 1 can understand which part of the vehicle body 2 of the vehicle 1 is in contact with the detected object O when the vehicle 1 is driven at the current steering angle. As a result, a display that allows the driver to easily judge which driving operation is better to be performed in order to prevent the vehicle 1 from contacting the detected object O can be performed.

[0106] In addition, the control unit 403 can also change the display mode of the partial image PG according to the distance between the position of the detected object O and the position of the virtual vehicle image G5. Thus, by confirming the change in the display mode of the partial image PG, the positional relationship between the vehicle 1 and the object O can be grasped in more detail. Specifically, as the distance between the position of the detected object O and the position shown by the virtual vehicle image G5 becomes shorter, the control unit 403, for example, increases the degree of red of the partial image PG displayed in red, or makes the partial image PG flash, and emphasizes the display of the partial image PG. As a result, it is easier for the driver to pay more attention to the object.

[0107] On the contrary, as the distance between the position of the detected object O and the position shown by the virtual vehicle image G5 becomes longer, the control unit 403 reduces the degree of redness of the partial image PG of the virtual vehicle image G5, or changes it to a color other than red, such as yellow, or extends the interval of flashing of the partial image PG to cancel the emphasized display of the partial image PG. When the driver of the vehicle 1 performs steering operation on the steering control unit 4 to change the direction of travel of the vehicle 1, and the object O in contact with the vehicle 1 is not detected by the detection unit 402, the control unit 403 returns the display mode of the partial image PG to the same display mode as the other parts of the virtual vehicle image G5. In addition, the control unit 403 releases the fixation of the virtual vehicle image G5 at the contact position P3, and again moves the position of the virtual vehicle image G5 in the synthetic image G3 as the vehicle 1 moves. As a result, the driver can clearly recognize that the contact with the object O has been eliminated, and can also display the previous area so that it is easy to continue driving while paying attention to the possibility of contact with other objects. In addition, in Figure 6 In the case where it is confirmed that no object is detected on the route when the vehicle 1 advances a specified distance at the changed steering angle, the virtual vehicle image G5 may not be displayed after a specified period, for example, after a specified time or after traveling a specified distance.

[0108] Here, since it is assumed that the object O detected by the detector 402 is a stationary object such as a wall or a wall base, the control unit 403 changes the display mode of the partial image PG in the virtual vehicle image G5, but does not change the display mode of the approaching object indicator G6. In other words, the driver of the vehicle 1 can recognize whether the object detected by the detector 402 is a stationary object or a moving object gradually approaching the vehicle 1 by confirming the display state of the approaching object indicator G6.

[0109] exist Figure 6 In the case of the vehicle 1 being in contact with the detected moving body, the control unit 403 displays the approaching object indicator G6 in a gray scale, which is the display mode of the approaching object indicator G6 when the detection unit 402 does not detect a moving body that may be in contact with the vehicle 1. However, in the case where the object detected by the detection unit 402 is a moving body such as another vehicle or a pedestrian, the control unit 403 changes the display mode of the approaching object indicator G6 in the direction in which the detected moving body exists among the approaching object indicators G6 included in the peripheral image G2. At this time, the control unit 403 may also change the display mode of the approaching object indicator G6, and when there is a moving body on the route when the vehicle 1 advances a specified distance at the current steering angle, change the display mode of the partial image PG in contact with the detected moving body in the virtual vehicle image G5.

[0110] In addition, in the case where the object detected by the detection unit 402 is a moving body gradually approaching the vehicle 1, the control unit 403 changes the display mode of the approaching object indicator G6 in the direction in which the detected object is gradually approaching, as described above. For example, the control unit 403 changes the color of the approaching object indicator G6 in the direction in which the detected object is gradually approaching to yellow, etc., or makes the approaching object indicator G6 flash. Alternatively, in the case where each approaching object indicator G6 includes a plurality of arrows, it can also be displayed as an animation display in which the control unit 403 sequentially changes the display mode starting from the arrow away from the virtual vehicle image G5 to scroll the plurality of arrows included in the approaching object indicator G6 displayed in the direction in which the detected moving body exists.

[0111] Figure 7 as well as Figure 8 FIG. 1 is a diagram for explaining an example of a method for displaying a virtual vehicle image by the surrounding monitoring device according to the first embodiment. Figure 7 In FIG. 1 , the X axis corresponds to the vehicle width direction of the vehicle 1, the Z axis corresponds to the moving direction of the vehicle 1, and the Y axis corresponds to the height direction of the vehicle 1. In the case where the virtual vehicle image G5 is composed of a plurality of polygons PL, as shown in FIG. Figure 7 as well as Figure 8 As shown, the control unit 403 obtains the value of the normal vector n of the vertices V1, V2, and V3 of each polygon PL in the Y-axis direction (direction perpendicular to the road surface) (hereinafter referred to as the Y component). Furthermore, the control unit 403 determines the transmittance of the polygon PL based on the Y component of the normal vector n.

[0112] Specifically, the control unit 403 obtains the Y component of the normal vector n of the vertices V1, V2, and V3 of the polygon PL. Next, the control unit 403 determines the pixels included in the polygon PL based on the Y component of the normal vector n of the vertices V1, V2, and V3. At this time, the control unit 403 increases the transmittance as the Y component of the normal vector n increases. In other words, the control unit 403 can increase the transmittance of the position parallel to the ground (the roof of the vehicle, the hood, etc.). As a result, an image that is transmitted from the outline of the vehicle 1 toward the inside can be displayed as a virtual vehicle image G5. In the present embodiment, the color of the pixels constituting the polygon PL is set to white, but it is not limited to this. For example, it can also be set to any color such as the color of the body of the vehicle 1.

[0113] Fig. 9 FIG. 1 is a diagram for explaining an example of a method for determining the color of a polygon constituting a virtual vehicle image by the surrounding monitoring device according to the first embodiment. Fig. 9, the horizontal axis represents the type of color (e.g., RGB) of the vertices of the polygons forming the virtual vehicle image G5, and the vertical axis represents the value of the color (e.g., RGB value) of the vertices of the polygons forming the virtual vehicle image G5. In the present embodiment, when an object is detected on the route when the vehicle 1 advances a predetermined distance at the current steering angle and the virtual vehicle image G5 is displayed, and when the distance between the object and the virtual vehicle image G5 is less than a predetermined value, the control unit 403 makes the display mode of the partial image PG in contact with the object in the virtual vehicle image G5 different from the display mode of the other parts of the virtual vehicle image G5.

[0114] Specifically, when an object is detected on the route when the vehicle 1 advances a predetermined distance at the current steering angle, but the distance between the object and the virtual vehicle image G5 is greater than a predetermined value, as shown in FIG. Fig. 9 As shown, the control unit 403 makes the values ​​of the RGB colors of the vertices of the polygon constituting the virtual vehicle image G5 equal. Furthermore, the control unit 403 interpolates the colors of the areas within the polygons enclosed by the vertices by linear interpolation or the like based on the values ​​of the RGB colors of the vertices of the polygons to determine the color of the entire polygon. Thus, the control unit 403 can display the virtual vehicle image G5 in white.

[0115] On the other hand, when the distance between the object detected on the route and the virtual vehicle image G5 is less than a predetermined value, as shown in FIG. Fig. 9 As shown, the control unit 403 makes the value of GB, for example, of each vertex of the polygon constituting the partial image among the polygons constituting the virtual vehicle image G5 smaller than the value of R of each vertex. In this case, the control unit 403 also interpolates the color of the area within the polygon surrounded by each vertex by linear interpolation or the like based on the values ​​of the RGB colors of each vertex of the polygon, and determines the color of the entire polygon. Thus, the control unit 403 can display the partial image PG in red.

[0116] In the present embodiment, the control unit 403 displays the partial image in red by making the value of GB of each vertex of the polygon constituting the partial image PG smaller than the value of R of each vertex, thereby emphasizing the display of the partial image PG, but the present invention is not limited thereto. For example, the control unit 403 may also display the partial image PG in green by making the value of RB of each vertex of the polygon constituting the partial image PG smaller than the value of G of each vertex, thereby emphasizing the display of the partial image PG.

[0117] At this time, the control unit 403 may also reduce the value of GB of each vertex of the polygon of the partial image PG constituting the virtual vehicle image G5 as the distance between the position of the detected object and the position of the virtual vehicle image G5 decreases. Thus, the control unit 403 increases the degree of redness of the partial image PG and emphasizes the display of the partial image PG. Thus, it is easier to recognize which position of the vehicle body of the vehicle 1 is in contact with the object, and it is easier to recognize the degree of attention when moving the vehicle 1. In addition, the control unit 403 always makes the values ​​of RGB colors of each vertex of the polygon other than the partial image PG among the polygons constituting the virtual vehicle image G5 equal. Thus, the control unit 403 can display the polygon other than the partial image PG in white.

[0118] Fig.10 FIG. 4 is a diagram showing an example of a display screen of the surrounding monitoring device according to the first embodiment. Fig.10 As shown, at time t1, when the detection unit 402 detects an object O (stationary object) existing on the route when the vehicle 1 advances a specified distance at the current steering angle, the control unit 403 causes the virtual vehicle image G5 to be displayed overlappingly on the surrounding image G2 in a manner that the virtual vehicle image G5 advances to a specified distance based on the position of the vehicle image G1. Moreover, as the virtual vehicle image G5 advances, the distance to the detected object O decreases. When it becomes below a specified value, the portion of the image PG of the virtual vehicle image G5 that is in contact with the object O is changed to a first display mode (e.g., a red highlighted display) indicating contact with the object O. In addition, Fig.10 In FIG. 1 , the guide line m is the predicted movement trajectory of the vehicle image G1 when the vehicle 1 moves at the current steering angle, and is displayed at a position having a predetermined margin relative to the vehicle width of the vehicle image G1 . The guide line m may not be displayed.

[0119] Specifically, the control unit 403 obtains an XZ plane parallel to the road surface (see Figure 7) is the Euclidean distance from each vertex of the polygon constituting the partial image PG to the object O. Next, the control unit 403 makes the value of GB of each vertex smaller than the value of R of each vertex based on the Euclidean distance between each vertex of the polygon constituting the partial image PG and the object O. Moreover, the control unit 403 determines the color of the pixels contained in the polygon constituting the partial image PG through the fragment shader based on the RGB value of each vertex of the polygon constituting the partial image PG. In addition, the control unit 403 interpolates the color of the area within the polygon enclosed by each vertex through linear interpolation or the like based on the RGB color values ​​of each vertex of the polygon, and determines the color of the entire polygon. Thus, the control unit 403 displays the partial image PG in red. In addition, when calculating the RGB value of the polygon constituting the virtual vehicle image, the RGB value of the polygon constituting the partial image can also be calculated simultaneously based on the distance between each vertex of the polygon and the object. Thus, the virtual vehicle image G5 and the partial image PG can be generated without separation.

[0120] After that, the vehicle 1 continues to move, and at time t2 after time t1, when the virtual vehicle image G5 reaches the contact position P3 or the position in front of the contact position P3 with the object O, the control unit 403 temporarily stops the virtual vehicle image G5 at the contact position P3 and continues to display the position. In this case, as shown at time t2, the display color of the partial image PG may be changed according to the distance from the object O. For example, the color tone of the partial image PG may be enhanced (the degree of redness may be enhanced) to indicate that the object is closer. Alternatively, the red color tone may be enhanced for a position where the distance from the object O is less than a specified value, and the yellow color tone may be enhanced for a position where the distance exceeds the specified value.

[0121] Furthermore, at time t3 after time t2, the steering angle of the vehicle 1 is changed, and the vehicle 1 is no longer likely to come into contact with the object O, that is, the object O is not detected by the detector 402 on the route based on the changed steering angle. At time t3, the contact between the object O and the virtual vehicle image G5 can be avoided, as shown in FIG. Fig.10 As shown in FIG. 1 , the control unit 403 releases the fixation of the virtual vehicle image G5 at the contact position P3, and displays the virtual vehicle image G5 at the position P2 when the vehicle 1 has moved forward a predetermined distance based on the position of the vehicle 1 represented by the vehicle image G1b at time t3. That is, the movement of the virtual vehicle image G5 is allowed to be restarted. As a result, a display is performed that makes it easy to continue driving while keeping an eye on the possibility of contact with other objects for the area further ahead.

[0122] At this time, the control unit 403 may also change the partial image PG in the virtual vehicle image G5 from a first display mode, such as red, indicating contact between the object O and the virtual vehicle image G5 to a second display mode, such as yellow, indicating contact avoidance. In addition, in other embodiments, the emphasized display in red may also be released. That is, at time t3, the control unit 403 returns the display mode of the partial image PG in the virtual vehicle image G5 to the same display mode (e.g., white) as the other parts of the virtual vehicle image G5. Thus, a display indicating that contact with the object O can be avoided by the current steering angle (changed steering angle) of the vehicle 1 can be provided to the driver of the vehicle 1.

[0123] Fig.11 as well as Fig.12 4 is a diagram for explaining an example of a process of emphasizing and displaying a partial image PG by the surrounding monitoring device according to the first embodiment. Fig.11 As shown, the control unit 403 first obtains the position of each vertex V of the polygon constituting the partial image PG in the virtual vehicle image G5 with respect to the XZ plane 1100 ( Figure 7 The control unit 403 finds the point V′ at which a perpendicular line 1101 to a plane defined by the X-axis and the Z-axis intersects the XZ plane 1100. Next, the control unit 403 obtains the Euclidean distance L between the point V′ in the XZ plane 1100 and the position of the object O.

[0124] Next, the control unit 403 Fig.12 The intensity distribution 1200 shown determines the degree of emphasis display corresponding to the obtained Euclidean distance L. Here, the intensity distribution 1200 is a distribution of the degree of emphasis display when emphasizing a part of the image, and as the Euclidean distance L decreases, the degree of emphasis display increases.

[0125] In this embodiment, the intensity distribution 1200 is a concentric intensity distribution in which the degree of emphasis is reduced to a smaller extent as the Euclidean distance L increases, with the position of the object O as the center. For example, the intensity distribution 1200 is represented by a high-order curve in which the degree of emphasis increases sharply when the Euclidean distance L is less than a preset distance (e.g., 1.7 to 3.0 m). For example, the intensity distribution 1200 is an intensity distribution in which the value of GB decreases sharply and R is emphasized when the Euclidean distance L is less than a preset distance.

[0126] Therefore, if Fig.12 As shown in FIG. 1 , among the polygons constituting the virtual vehicle image G5, the control unit 403 highlights the polygons whose Euclidean distance L to the object O is closer in red. As a result, Fig.12As shown, the control unit 403 can highlight the polygons constituting the partial image PG among the polygons constituting the virtual vehicle image G5 in red.

[0127] In this way, according to the vehicle 1 involved in the first embodiment, it is possible to enable the driver to easily recognize whether there is an object O on the route when the vehicle 1 moves forward a specified distance at the current steering angle based on whether the virtual vehicle image G5 is included in the display screen displayed on the display device 8.

[0128] (Second Embodiment)

[0129] The second embodiment is an example in which a display screen including a three-dimensional image of the surroundings of the vehicle is displayed on a display device instead of a captured image obtained by capturing the vehicle's traveling direction by a capturing unit. In the following description, description of the same configuration as the first embodiment is omitted.

[0130] Fig.13 as well as Fig.14 FIG. 1 is a diagram showing an example of a display screen of a surrounding monitoring device of a vehicle 1 according to a second embodiment. Fig.13 As shown, the control unit 403 displays a display screen G on the display device 8, and the display screen G includes a synthesized image G3 and a three-dimensional image G7 of the vehicle 1 and its surroundings (hereinafter referred to as a three-dimensional surrounding image). Thus, in addition to the synthesized image G3, the three-dimensional surrounding image G7 is also visually confirmed, thereby making it easy to grasp the positional relationship between the vehicle 1 and the objects around it in more detail.

[0131] Here, as described above, the three-dimensional surrounding image G7 is a three-dimensional image of the vehicle 1 and its surroundings. In the present embodiment, the three-dimensional surrounding image G7 is an image obtained by pasting an image of the surroundings of the vehicle 1 captured by the imaging unit 15 onto a three-dimensional surface in a bowl or cylinder shape. Fig.13 As shown, the three-dimensional surrounding image G7 includes a three-dimensional vehicle image G8 which is a three-dimensional image of the vehicle 1. In the present embodiment, the three-dimensional vehicle image G8 is composed of a plurality of polygons like the virtual vehicle image G5 and is an image showing the three-dimensional shape of the vehicle 1.

[0132] In addition, in the present embodiment, the control unit 403 displays the vehicle position information I that can identify the position of the three-dimensional vehicle image G8 on the road surface in the three-dimensional surrounding image G7. For example, the vehicle position information I is information that displays the position of the three-dimensional vehicle image G8 on the road surface in the three-dimensional surrounding image G7 by a gray scale or a line (e.g., a dotted line) surrounding the position where the three-dimensional vehicle image G8 exists.

[0133] Moreover, if Fig.14As shown in FIG. 1 , similarly to the first embodiment, the control unit 403 displays the proximity indicator G6 in the surrounding image G2 and also displays the proximity indicator G9 for the three-dimensional surrounding image G7. Fig.14 As shown, the control unit 403 displays the approaching object index G9 included in the three-dimensional surrounding image G7 with a gray scale. In order to easily understand the positional relationship between the vehicle image G1 and the surrounding objects, the control unit 403 does not display the virtual vehicle image G5 in the three-dimensional surrounding image G7, but is not limited to this, and the virtual vehicle image G5 may also be displayed for the three-dimensional surrounding image G7.

[0134] Fig.15 as well as Fig.16 FIG. 4 is a diagram showing an example of a display screen of the surrounding monitoring device according to the second embodiment. In this embodiment, when the detector 402 detects an object approaching the vehicle 1 (for example, an object approaching from the left side of the traveling direction of the vehicle 1), the Fig.15 As shown, the control unit 403 changes the display mode of the approaching object indicators G6 and G9 included in the surrounding image G2 and the three-dimensional surrounding image G7 in the direction in which the detected object approaches (for example, the left side).

[0135] In addition, when the detector 402 detects that the vehicle 1 is approaching from both the left and right sides of the traveling direction, as shown in FIG. Fig.16 As shown, the control unit 403 changes the display color of the approaching object indicators G6 and G9 existing on the left and right sides of the traveling direction (e.g., the front) of the vehicle 1 to yellow or the like, or makes the approaching object indicators G6 and G9 flash. Alternatively, when the approaching object indicators G6 and G9 include a plurality of arrows, the control unit 403 may scroll the plurality of arrows included in the approaching object indicators G6 and G9 displayed in the direction where the detected moving object exists by an animation display that sequentially changes the display mode starting from the arrow away from the virtual vehicle image G5.

[0136] In the second embodiment, when the detection unit 402 is in an operating state and when an object approaching the vehicle 1 is not detected, approaching object indicators G6 and G9 are displayed in advance by gray scales, etc., thereby indicating that the object approaching the vehicle 1 is detected by the detection unit 402. In addition, when an object approaching the vehicle 1 is detected, the display method of the approaching object indicators G6 and G9 is changed. By changing the display method, the driver of the vehicle 1 can easily recognize that the object approaching the vehicle 1 is detected. In the second embodiment, the control unit 403 displays both the virtual vehicle image G5 and the approaching object indicators G6 and G9 on the display screen G, but at least the virtual vehicle image G5 is displayed.

[0137] As described above, according to the surrounding monitoring device according to the second embodiment, by visually checking the three-dimensional surrounding image G7 in addition to the synthesized image G3 , it is possible to grasp the positional relationship between the vehicle 1 and the surrounding objects in more detail.

[0138] (Third Embodiment)

[0139] The third embodiment is an example in which the above-mentioned virtual vehicle image G5 and the surrounding conditions of the vehicle image G1 are displayed in a more easily recognizable state. In the following description, the description of the same configuration as the first embodiment is omitted.

[0140] In the third embodiment, when the object O is detected by the detection unit 402 on the route when the vehicle 1 advances a specified distance at the current steering angle, the control unit 403 displays (enlarges) the synthetic image G3 in an enlarged manner compared to before the object O is detected. The enlargement of the synthetic image G3 can be performed, for example, by bringing the virtual viewpoint close to the vehicle image G1. By performing the enlarged display, the driver of the vehicle 1 can easily and intuitively recognize that the object O that may come into contact with the vehicle 1 has been detected and needs to pay attention to the surrounding conditions. In addition, by enlarging the display, the details of the surrounding conditions of the vehicle image G1 and the virtual vehicle image G5 are displayed more clearly. As a result, it is possible to provide the driver of the vehicle 1 with a display that makes it easy to confirm the surrounding conditions and give a sense of security.

[0141] In addition, in the third embodiment, when the synthetic image G3 is displayed in an enlarged manner, the control unit 403 performs a staggered display in which the display position (for example, the display position of the vehicle image G1) is staggered in the direction opposite to the turning direction based on the current steering angle. For example, the position of the virtual viewpoint is moved in the direction opposite to the turning direction to stagger the display of the synthetic image G3. When the synthetic image G3 is displayed in an enlarged manner in the limited display area of ​​the display device 8, it is easy to recognize the details around the vehicle image G1 or the virtual vehicle image G5, but the display range (field of view) centered on the vehicle image G1 or the virtual vehicle image G5 becomes narrower. When the vehicle 1 turns, if it can be confirmed that it does not contact the object O that has been detected in the direction opposite to the turning direction, the vehicle 1 can pass. On the other hand, there may be other objects in the turning direction. Therefore, the control unit 403 staggers the display position of the vehicle image G1 in the direction opposite to the turning direction, expands the display range (field of view) of the turning direction, and can display a wider range.

[0142] In addition, in the third embodiment, when the control unit 403 is performing the enlarged display of the synthetic image G3, when the object O is far away, the enlarged display is canceled (for example, the display size before the enlargement is returned), and the virtual vehicle image G5 may not be displayed. By restoring (reducing) the vehicle image G1 to the original display size and not displaying the virtual vehicle image G5, the synthetic image G3 as a whole is significantly changed, and the display that the display of the virtual vehicle image G5 is ended can be easily recognized by the driver of the vehicle 1. In addition, the driver of the vehicle 1 can easily and intuitively recognize that the possibility of reducing or eliminating the contact with the object O. In addition, although it will be described later, the virtual vehicle image G5 in the enlarged display can also be temporarily hidden by the driver's operation.

[0143] Fig.17 FIG. 2 is a diagram showing an example of a display screen of a periphery monitoring device of a vehicle 1 according to a third embodiment. Fig.17 The time t1 is the state before the object O is detected by the detection unit 402 on the route when the vehicle 1 advances a predetermined distance at the current steering angle, and shows the display state of the normal size before the enlarged display is performed. Before the object is detected on the route, the vehicle image G1 is displayed in a non-enlarged state at approximately the center of the synthetic image G3.

[0144] Time t2 is the state where the vehicle image G1 (vehicle 1) moves forward from time t1, detects object O on the route, starts displaying the virtual vehicle image G5, and performs enlarged display of the vehicle image G1 and the virtual vehicle image G5. In this case, the magnification of the composite image G3 including the vehicle image G1 and the virtual vehicle image G5 is, for example, "2". In this case, the control unit 403, for example, moves the virtual viewpoint closer to the vehicle image G1 based on the vehicle image G1. As a result, a display change is performed such as enlarging the composite image G3 as a bird's-eye view image. In addition, at this time, the control unit 403 executes the movement of the virtual viewpoint in the direction opposite to the turning direction based on the current steering angle of the vehicle 1 acquired by the acquisition unit 401. Fig.17 In the case of the vehicle image G1 or the virtual vehicle image G5, the display position of the object O is shifted to the left in the figure, and the display position of the synthetic image G3 is changed. By performing the enlarged display process, the surrounding conditions of the vehicle image G1 and the virtual vehicle image G5, such as the distance to the object O, can be more easily recognized. In addition, by performing the staggered display process, the display area in the turning direction according to the current steering angle of the vehicle image G1 and the virtual vehicle image G5, that is, the right direction in the figure, is expanded, and the presence or absence of other objects in the turning direction can be easily confirmed.

[0145] In addition, the magnification when the control unit 403 enlarges the synthetic image G3 (including the image of the vehicle image G1 or the virtual vehicle image G5, etc.) may be a fixed value as described above, or may be changed according to the distance between the object O and the virtual vehicle image G5 (vehicle image G1). For example, the magnification may be made smaller as the virtual vehicle image G5 (vehicle image G1) is farther away from the object O. The smaller the magnification, the wider the field of view can be ensured, and the display can be made easier to grasp the overall situation of the surroundings of the vehicle image G1 and the virtual vehicle image G5. On the other hand, the closer the virtual vehicle image G5 (vehicle image G1) is to the object O, the larger the magnification. In this case, the field of view becomes narrower, but since it is magnified and displayed, it is possible to display the recent situation of the vehicle image G1 or the virtual vehicle image G5 more easily than when it is far away. As a result, the driver is more likely to pay attention to the object O and decide the direction or steering angle to avoid contact with the object O.

[0146] In addition, the control unit 403 may change the magnification of the synthetic image G3 according to the speed of the vehicle 1. For example, when approaching the object O at a low speed (for example, when approaching a wall for parking), the closer to the object O, the larger the magnification, and the easier it is to recognize the positional relationship with the object O in more detail. On the other hand, when the vehicle 1 is traveling at a predetermined speed or higher (for example, 12 km / h or higher) at a position far from the object, the magnification is reduced, and the synthetic image G3 with a wider field of view is displayed. In this case, a wider field of view can be ensured, and the situation of the traveling direction can be more easily grasped than at a low speed (when the magnification is large), and a display that is easy to give a sense of security can be performed.

[0147] In addition, time t2 shows that the distance between the object O and the virtual vehicle image G5 is less than a predetermined value, and the display mode of the partial screen PG of the virtual vehicle image G5 that may come into contact with the object O is changed to a first display mode (for example, red) indicating the contact with the object O. In this case, a display is performed that allows the driver to easily recognize which position of the vehicle 1 may come into contact with the object O when the vehicle 1 is driven at the current steering angle.

[0148] Fig.17 The time t3 and Fig.10In the same manner as described above, in the virtual vehicle image G5, as the distance to the object O becomes shorter, the redness of the partial image PG displayed in red increases, so that the driver can easily recognize the approach of the object O. In addition, compared with time t2, the virtual vehicle image G5 is closer to the object O, and the position on the virtual vehicle image G5 where the virtual vehicle image G5 and the object O are likely to contact is further concentrated, and the position with a higher possibility of contact is displayed with a higher degree of redness. In this way, the synthetic image G3 is displayed in an enlarged manner, and the display mode of the partial area PG with a higher possibility of contact is changed, so that the driver can recognize the approach state relative to the object O in more detail.

[0149] At time t4, a state is shown in which the guide line m indicating the predicted movement trajectory of the vehicle image G1 moves closer to the front than the object O by changing the steering angle. In this case, the virtual vehicle image G5 is also shown to move away from the object O and to a contact avoidance state (a passable state) due to the change in route based on the change in the steering angle. In this case, the synthetic image G3 is also displayed in an enlarged manner, so that the driver can easily recognize that the possibility of contact between the object O and the virtual vehicle image G5 (vehicle image G1) is reduced. In addition, at time t4, a state is shown in which the distance between the object O and the virtual vehicle image G5 exceeds a prescribed value by changing the steering angle, and the contact is avoided. In this case, the display color of the partial image PG of the virtual vehicle image G5 displayed at time t3 is changed from the first display mode (for example, red) indicating contact to the second display mode (for example, white) indicating contact avoidance. The driver can also easily recognize that the contact with the object O can be avoided according to the current steering angle (the changed steering angle) by changing the display mode of the partial image PG of the virtual vehicle image G5.

[0150] Time t5 is a display example indicating that the vehicle can continue to travel by maintaining the current steering angle (the steering angle after the change) and avoiding contact with the object O detected by the detection unit 402. In this case, the enlarged display of the synthetic image G3 is canceled. In this case, the vehicle image G1 and the surrounding area of ​​the virtual vehicle image G5 can be displayed in a wide range (wide field of view). The control unit 403 cancels the enlarged display of the synthetic image G3 and returns to the zoomed-out state (the same magnification "1" as time t1). As a result, the driver can more clearly recognize that the contact with the object O can be eliminated by changing the steering angle, and the contact with the object O has been eliminated. In addition, by continuing to display the virtual vehicle image G5, it is easy to continue driving while keeping an eye on the possibility of contact with other objects for the previous area. In this state, the distance between the object O and the virtual vehicle image G5 exceeds the specified value and expands, so the partial image PG of the virtual vehicle image G5 is displayed in a second display mode (for example, white) indicating that the contact has been avoided. Therefore, a display is performed that allows the driver to easily recognize that the contact with the object O has been avoided, also by the display color of the partial image PG of the virtual vehicle image G5. In this case, the offset display process may be set to be canceled or not.

[0151] At time t5, when the speed of the vehicle 1 is a prescribed speed, for example, 12 km / h or above, or when the vehicle 1 avoids contact with the object O, when a prescribed period has passed, for example, after a prescribed time (for example, 2 seconds have passed), or after a prescribed distance (for example, 5 meters) has been traveled, the display of the virtual vehicle image G5 may be terminated and not displayed. In other words, it may be shown that there is no object that may be contacted on the route when the vehicle 1 advances a prescribed distance at the current steering angle (the changed steering angle). In addition, when an object is newly detected on the route, as shown at time t2, the virtual vehicle image G5 is displayed, and the composite image G3 is enlarged and displayed in an offset manner.

[0152] Fig.18 An example is shown in which the display mode of the synthetic image G3 (an image including the vehicle image G1 , the virtual vehicle image G5 , and the like) is changed according to the shift state of the speed change operation unit 7 of the vehicle 1 .

[0153] The scene S1 shown in the center is a state where the synthetic image G3 (including the image of the vehicle image G1) is not enlarged and displayed (magnification "1") in a non-detection case where the detection unit 402 does not detect the object O on the route when the vehicle 1 moves forward a specified distance at the current steering angle. On the other hand, scene S2 is a case where the speed change operating unit 7 is in the forward (D) gear or the parking (P) gear. In other words, the vehicle 1 is not in a state where it can travel in reverse, for example, the steering operation is performed to the right and the vehicle 1 is about to turn right based on the current steering angle. Moreover, it is an example of detecting an object O on the route when the detection unit 402 is used to detect the object O. In the case of such a right turn, the area that the driver needs to pay attention to is the area of ​​the left front end of the vehicle in the direction opposite to the turning direction, especially the relative distance to the object O. Therefore, in such a case, Fig.17 As shown, the control unit 403 magnifies and displays the synthetic image G3, for example, at a higher magnification, such as a magnification of "2". On this basis, the driver also needs to pay attention to other objects that may exist in the turning direction. Therefore, the control unit 403 performs a staggered display process to shift the display position of the synthetic image G3 in the direction opposite to the turning direction determined by the steering angle. In this case, the control unit 403 can determine the amount of stagger, for example, so that the detected object O and the vehicle image G1 and the virtual vehicle image G5 converge within the display area, or can perform the process at a predetermined magnification such that they converge. As a result, the driver of the vehicle 1 can confirm the enlarged image of the distance to the object O in front (whether there is contact) while paying attention to the wider area in the turning direction and continuing to drive.

[0154] On the other hand, scene S3 is a case where the speed change operating unit 7 is in the reverse (R) gear, in other words, the vehicle 1 is in a state where it can travel backward, for example, the steering operation is performed to the right, and the vehicle 1 is in a state where it is turning right and moving backward based on the current steering angle. Furthermore, the detection unit 402 detects an object O (such as a pillar) on the route when the vehicle 1 moves forward (backward) a specified distance at the current steering angle, and an example where an object O (such as a wall) is detected on the left side of the vehicle 1. In such a case of right turn and reverse driving, the areas that the driver mainly needs to pay attention to are the left front end of the vehicle in the direction opposite to the turning direction and the right rear end of the vehicle in the turning direction. In other words, it is necessary to pay attention to the diagonal areas in front and behind the vehicle image G1 (virtual vehicle image G5). Therefore, as Fig.18As shown in the scene S3, the control unit 403 sets the magnification of the synthetic image G3 to, for example, "1.5" which is smaller than when the vehicle 1 is moving forward, and displays the magnified image G3. In addition, when the vehicle 1 can move backward, the staggered display process is not performed, and the left and right display areas of the vehicle image G1 are substantially evenly secured. By suppressing the magnification and not performing the staggered display process, the surroundings (left and right sides) of the vehicle image G1 (virtual vehicle image G5) can be confirmed in a wider range (wide field of view) than when the vehicle is moving forward, and the display can be easily reversed.

[0155] In addition, as described above, when an object is detected on the route when the vehicle 1 moves forward a specified distance at the current steering angle, the virtual vehicle image G5 is displayed based on the position of the vehicle image G1, and is further moved along the route, thereby having the effect of making it easy to focus on the object that may be touched. On the other hand, if the virtual vehicle image G5 is displayed continuously, the driver who recognizes the object on the route at once may feel annoyed. In addition, when the distance between the object and the virtual vehicle image G5 is small, it may be difficult to confirm the details of the surroundings of the object due to the virtual vehicle image G5. In addition, if part of the image PG is changed to an accent color due to approaching the object, the details of the surroundings of the object may be difficult to confirm. Therefore, the control unit 403 can set the display of the virtual vehicle image G5 to be temporarily hidden. For example, if Fig.19 As shown in the scene S4 of FIG. 4 , the control unit 403 can overlap and display the non-display switch 30 (e.g., Ghost Off switch) of the virtual vehicle image G5 at a position related to the virtual vehicle image G5 being displayed, such as on the virtual vehicle image G5. When the driver determines that the display of the virtual vehicle image G5 is unnecessary, for example, the driver has grasped the existence of the object O, or has recognized the relative positional relationship between the object O and the virtual vehicle image G5 (vehicle image G1, vehicle 1). In this case, by operating (pressing) the non-display switch 30, the virtual vehicle image G5 can be temporarily non-displayed, thereby improving the visibility of the surroundings of the object O. In addition, the display position of the non-display switch 30 can be any position on the display screen G, or can be set at a position outside the display screen G, but it is preferably set at a position related to the virtual vehicle image G5. For example, by displaying the non-display switch 30 overlapping with the virtual vehicle image G5, the driver can recognize the existence of the virtual vehicle image G5, that is, the existence of the object to be touched, and can perform the operation of non-displaying the virtual vehicle image G5, so that the virtual vehicle image G5 can be effectively used and the visibility of the surroundings of the object can be improved.

[0156] By operating (pressing) the non-display switch 30, the enlarged and displayed synthetic image G3 (including the vehicle image G1 and the virtual vehicle image G5) can be displayed as shown in FIG. Fig.19As shown in the scene S5 of FIG. 1 , the display returns to the non-enlarged state, or the enlarged state may be maintained without displaying the virtual vehicle image G5. In addition, when the staggered display process is executed during the enlarged display, the staggered display process is also canceled, and the vehicle image G1 returns to the approximately central position of the synthetic image G3, and the entire periphery of the vehicle image G1 returns to a display that is approximately uniform and can be confirmed.

[0157] When the virtual vehicle image G5 is not displayed by operating the non-display switch 30, the non-display of the virtual vehicle image G5 is canceled when a predetermined condition is satisfied. In other words, when an object continues to exist on the route when the vehicle 1 advances a predetermined distance at the current steering angle, the virtual vehicle image G5 is displayed again. For example, when it is confirmed that the vehicle 1 has stopped, for example, when the shift state is changed by the speed change operation unit 7, for example, when "shift P" is selected, the virtual vehicle image G5 is displayed again after a predetermined period of time has passed since it was non-displayed by operating the non-display switch 30, for example, when a predetermined time has passed, when a predetermined distance has been traveled, etc. In other words, the driver can recognize the existence of the object that is about to be touched. In this way, even when the non-display switch 30 is operated, the virtual vehicle image G5 is restored to a state where it can be displayed by satisfying the predetermined condition. As a result, for example, it is possible to avoid the virtual vehicle image G5 from being permanently non-displayed.

[0158] In addition, an example is shown in which the non-display switch 30 can be used under the condition that the virtual vehicle image G5 is displayed, but it can also be set to be usable from the stage before the virtual vehicle image G5 is displayed. For example, there are cases where the driver is familiar with the situation behind, such as parking. In addition, the driver sometimes knows the timing of displaying the virtual vehicle image G5. In such a case, the display of the virtual vehicle image G5 can be temporarily canceled before the virtual vehicle image G5 is displayed. In this case, the non-display of the virtual vehicle image G5 based on the operation of the non-display switch 30 is also temporary. When the specified conditions are met, the virtual vehicle image G5 is returned to a state where it can be displayed, and permanent non-display can be avoided.

[0159] In addition, a restoration switch 32 (eg, a Ghost ON switch) may be provided. When the virtual vehicle image G5 is not displayed by operating the non-display switch 30, the restoration switch 32 is turned on. Fig.19As shown in the scene S5 of FIG. 5 , the virtual vehicle image G5 can be redisplayed only during the non-display period of the virtual vehicle image G5. The restore switch 32 can be displayed, for example, from the non-display of the virtual vehicle image G5 until the gear is changed by the shift operation unit 7, for example, superimposed on the vehicle image G1. By providing the restore switch 32, the non-displayed virtual vehicle image G5 can be restored at a desired timing, and the confirmation of the surrounding conditions using the virtual vehicle image G5 can be easily restarted. In addition, the display position of the restore switch 32 can be any position on the display screen G, and can also be set at a position other than the display screen G, but by setting it at a position related to the vehicle image G1, the operation for redisplaying the virtual vehicle image G5 can be smoothly performed. In addition, in the case of scene 5, the virtual vehicle image G5 is not displayed, and the vehicle image G1 returns to a non-magnified state. In other embodiments, an enlargement switch for performing an enlarged display of the synthetic image G3 can also be provided separately from the restore switch 32. In this case, a display that can easily confirm the positional relationship between the vehicle image G1 and the object O can be performed without displaying the virtual vehicle image G5.

[0160] Thus, according to the surrounding monitoring device of the third embodiment, the magnification of the synthetic image G3 (including images such as the vehicle image G1 and the virtual vehicle image G5) and the presence or absence of the execution of the staggered display process are determined according to the traveling direction of the vehicle 1. In addition, the display of the virtual vehicle image G5 can be temporarily hidden or displayed again by the driver's operation. As a result, it is possible to realize the display of the surrounding confirmation suitable for the current situation according to the driving state of the vehicle 1 or the driver's operation, which can easily improve the driver's sense of security.

[0161] In the above-mentioned embodiment, an example is shown in which the display of the virtual vehicle image G5 is started when the object O is detected on the route when the vehicle 1 moves forward at the current steering angle for a predetermined distance. In other embodiments, the CPU 14a may also determine based on the operation state of the main switch 10a (operation switch) for virtual display that switches whether to display the virtual vehicle image G5 in the synthetic image G3 (bird's-eye view image). Figure 3 As shown in the example, the main switch 10a can be set as a part of the operation input unit 10 (touch sensor type switch) set in the monitor device 11, or can be set as another mechanical physical switch. When the vehicle 1 moves forward at the current steering angle and the object O is not detected on the route, and when the virtual vehicle image G5 can be displayed on the display device 8 by operating the main switch 10a (when the main switch is turned on), the CPU 14a displays the vehicle image G1 in a different manner from when the main switch 10a is turned off. For example, Fig. 20The display color of the vehicle image G1 shown is changed from the vehicle color corresponding to the vehicle 1 to another color (for example, green), or flashing, to show that the virtual vehicle image G5 is in a display standby state. The change in the display method in this case only needs to enable the driver and others to recognize that the virtual vehicle image G5 is in a display standby state. The display brightness of the vehicle image G1 may be changed or the display transmittance may be changed, and the display method may be appropriately changed. In addition, when the display method of the vehicle image G1 is changed based on the main switch 10a being in the on state, a message reporting that the virtual vehicle image G5 is in a display standby state may be displayed on the synthetic image G3 or other display areas, or a sound or reporting sound may be output so that the driver can recognize it. In this way, by changing the display method of the vehicle image G1 based on the operating state of the main switch 10a, when an object O is detected on the route, it is possible to make it easy for the driver to recognize that the existence of the object O is immediately notified through the virtual vehicle image G5, and the display can be easily given a sense of security. In addition, in Fig.19 In the case of the non-display switch 30 described above, even if the virtual vehicle image G5 is not displayed by operating the non-display switch 30, it is restored to a state where it can be displayed if a predetermined condition is satisfied. On the other hand, when the main switch 10a is turned off, the virtual vehicle image G5 is not displayed regardless of the presence or absence of the object O on the route. In this case, the virtual vehicle image G5 can be non-displayed as desired by the driver, and a simple synthetic image G3 can be provided.

[0162] In addition, in the third embodiment, the control unit 403 displays the surrounding conditions of the vehicle image G1 and the surrounding conditions of the virtual vehicle image G5 in an easily recognizable state by enlarging and displaying the synthetic image G3 (enlarging) compared to before the object O is detected. However, when the driver actually drives the vehicle 1, it is necessary to perform steering so that the vehicle 1 can pass safely and smoothly relative to the object O existing in the surroundings when moving. In other words, it is necessary to perform steering so that the virtual vehicle image G5 can ensure a necessary and sufficient margin (safety margin) relative to the object O. If the margin is insufficient, an unfavorable situation of being too close to the object O or contacting occurs. On the contrary, if the margin is excessively taken to be more than necessary, an unfavorable situation such as divergence from the appropriate moving path occurs. In addition, in this case, the margin is sometimes reduced on the opposite side of the direction in which the margin is considered, and on the opposite side, the possibility of contact is sometimes increased, or it is not possible to escape from a narrow space. In this way, the appropriate ensuring of the margin is sometimes affected by the driver's experience, skills, etc., and when the margin cannot be set appropriately, it sometimes becomes a cause of giving the driver a sense of uneasiness. In addition, there may be a case where the image capturing unit 15 used to capture and generate the composite image G3 has an installation error, or there may be a case where an error occurs in the display position of the surrounding image G2 due to the inclination of the road surface, etc. Such a case may also cause the margin distance to be unable to be appropriately set, and may sometimes cause the driver to feel uneasy.

[0163] Therefore, when displaying the virtual vehicle image G5, the control unit 403 performs a residual enlargement process of enlarging at least one of the length in the vehicle length direction and the length in the vehicle width direction of the virtual vehicle image G5 compared to the length corresponding to the vehicle image G1 currently displayed and displays it. In this case, the difference enlargement rate can be determined in advance by experiments, etc. according to the vehicle model, and can be set to, for example, 1.3 times the vehicle image G1 displayed corresponding to the size of the vehicle 1. This difference enlargement rate is an example, and the driver, etc. can appropriately set and change it. In addition, the residual enlargement process is a process of enlarging only the virtual vehicle image G5 independently of the vehicle image G1 or the surrounding image G2 displayed in the synthetic image G3.

[0164] Fig.21 It is an exemplary and schematic diagram for explaining the remaining amount enlargement process performed on the virtual vehicle image G5 with respect to the vehicle image G1 being displayed. Fig.21 The scene S6 shown is a state where only the vehicle image G1 is displayed in the synthesized image G3. That is, the vehicle image G1 and the surrounding image G2 are displayed at the same magnification.

[0165] When an object O is detected on the route when the vehicle 1 advances a predetermined distance at the current steering angle and the virtual vehicle image G5 is displayed, as shown in scene S7, the control unit 403 enlarges at least one of the length L1 in the vehicle length direction and the length W1 in the vehicle width direction of the virtual vehicle image G5 based on a predetermined magnification ratio ( Fig.21 In the case of scene S7, it is both sides). For example, when the length of the vehicle length direction of the vehicle image G1 before the margin magnification processing is set to L0, the length of the virtual vehicle image G5 in the vehicle length direction is enlarged to, for example, 1.3 times the length L1. Similarly, when the length of the vehicle width direction of the vehicle image G1 before the margin magnification processing is set to W0, the length of the virtual vehicle image G5 in the vehicle width direction is enlarged to, for example, 1.3 times the length W1. In this case, the distance relationship between the vehicle image G1 and the object O included in the surrounding image G2 is displayed to roughly correspond to the actual distance relationship. On the other hand, the virtual vehicle image G5 is displayed enlarged in the vehicle length direction and the vehicle width direction compared to the vehicle image G1. Therefore, in the case of moving the virtual vehicle image G5 so as not to come into contact with the object O, the virtual vehicle image G5 is displayed in a state containing (taking into account) a prescribed safety margin. As a result, if the driver determines the steering angle so that the guide line m (refer to Fig.12 If the vehicle does not come into contact with the object O, a safety margin is properly ensured regardless of the driver's experience, skills, the installation state of the camera 15, the inclination of the road surface, etc. In other words, when avoiding the object O, the vehicle avoids the unfavorable conditions of being too close to the object O and being too far away from the object O, and a display that easily gives the driver a sense of security can be performed.

[0166] In addition, when the control unit 403 performs the residual enlargement processing, the enlargement base point in the vehicle length direction can be set, for example, to the center position in the front-rear direction of the vehicle image G1 (vehicle 1). Similarly, the enlargement base point in the vehicle width direction can be set, for example, to the center position in the vehicle width direction of the vehicle image G1 (vehicle 1). In addition, as in scene S7, when enlarging the vehicle length direction and the vehicle width direction, it can be set to the center position of the front, rear, left, and right in the vehicle image G1 (vehicle 1).

[0167] However, for example, when an object O exists in front of the right side of the vehicle image G1 (vehicle 1), and the vehicle image G1 (vehicle 1) is turned right while avoiding the object O, if at least a margin distance between the left front corner of the vehicle image G1 (vehicle 1) and the object O can be ensured, the object O can be avoided. In addition, when the vehicle 1 is moved forward and turned, the safety margin in the vehicle length direction is easy to recognize, while the safety margin in the vehicle width direction tends to be difficult to recognize compared to the length direction of the vehicle 1. Therefore, the margin enlargement processing when the vehicle image G1 (vehicle 1) is turned while avoiding the object O is as follows. Fig.21 As shown in the scene S8 of FIG. 1 , only the length in the vehicle width direction may be enlarged. That is, the length in the vehicle length direction of the virtual vehicle image G5 is always set to the length L0 that maintains the length L0 in the vehicle length direction of the vehicle image G1 before the margin enlargement process. On the other hand, the length in the vehicle width direction is set to the length W2 that is enlarged by, for example, 1.3 times the length W0 in the vehicle width direction of the vehicle image G1 before the margin enlargement process. As a result, it is possible to suppress excessive safety margin around the vehicle image G1 (vehicle 1) (front and rear directions) when turning.

[0168] Furthermore, when the vehicle image G1 (vehicle 1) is moving forward, the control unit 403 may implement the margin enlargement process in the vehicle length direction only in front of the virtual vehicle image G5. Similarly, when the vehicle image G1 (vehicle 1) is moving backward, the control unit 403 may implement the margin enlargement process in the vehicle length direction only behind the virtual vehicle image G5. In addition, the safety margin in the vehicle width direction may be implemented only in the necessary direction according to the vehicle's traveling direction (turning direction), etc.

[0169] When the control unit 403 executes the remaining space enlargement processing of the virtual vehicle image G5, the virtual vehicle image G5 may be instantly enlarged to the target size, but the enlargement processing is preferably executed gradually so that the driver or the like can easily recognize that the remaining space enlargement processing is being executed.

[0170] In addition, when executing the remaining capacity enlargement processing of the virtual vehicle image G5, the virtual vehicle image G5 can be enlarged to the target size while moving on the route when the vehicle 1 advances a specified distance. However, in order to make it easy for the driver and the like to recognize that the remaining capacity enlargement processing is being executed, it is preferred to start the movement of the virtual vehicle image G5 after the remaining capacity enlargement processing is completed.

[0171] In addition, if Fig.17As shown in FIG. 4 , when the composite image G3 is enlarged and displayed, the control unit 403 may perform the remaining enlargement processing for the virtual vehicle image G5 after the enlargement processing of the composite image G3 is completed, or may perform the remaining enlargement processing for the virtual vehicle image G5 simultaneously with the enlargement processing of the composite image G3. In this case, the virtual vehicle image G5 is enlarged at a different enlargement ratio from that of the vehicle image G1 and the surrounding image G2.

[0172] When the remaining power enlargement process is executed for the virtual vehicle image G5, the control unit 403 may display the virtual vehicle image G5 in a manner different from when the remaining power enlargement process is not executed. For example, the display color may be changed or the display may be blinked to allow the driver to recognize the execution of the remaining power enlargement process.

[0173] In addition, in the above-mentioned embodiment, when the object O is detected on the route when the vehicle 1 advances a predetermined distance, the example in which the virtual vehicle image G5 is displayed and the remaining amount enlargement process is performed at this time is shown. In a modified example, regardless of the presence or absence of the object O (obstacle), the virtual vehicle image G5 can be displayed at an arbitrary timing by the driver's operation using the main switch 10a, etc. Moreover, the remaining amount enlargement process can also be performed at this time.

[0174] In addition, in the above-mentioned embodiment, an example is shown in which the process of enlarging the synthetic image G3 is performed, and the process of enlarging the margin of G5 is also performed. In other embodiments, the process of enlarging the synthetic image G3 may not be performed, and only the process of enlarging the margin of the virtual vehicle image G5 may be performed. In this case, the display of appropriately ensuring the remaining distance (safety margin) with respect to the object O on the route displayed on the synthetic image G3 can also be performed, and the same effect can be obtained.

[0175] (Variation Example)

[0176] In the above-described embodiments, an example is shown in which the virtual vehicle image G5 is displayed when an object O is detected on the route when the vehicle 1 moves forward a predetermined distance at the current steering angle. Fig. 22 In the modified example shown, regardless of the steering angle, the direction indicator 26 (see Figure 3), the virtual vehicle image G5 can be displayed. For example, in the case of coming to the road from an area outside the road, the driver operates the direction indicator 26 before determining the steering angle by operating the steering control unit 4 (steering wheel). In other words, the operating direction of the steering control unit 4 can be estimated in advance. Therefore, when the direction indicator 26 is operated, the acquisition unit 401 searches for objects between the steering angle "0" (neutral position) and the maximum steering angle of the indicated direction of the direction indicator 26. In other words, from then on, for the areas where the vehicle 1 may travel, the routes are set in sequence to search for objects. As a result, when an object is detected, such as Fig. 22 As shown, regardless of the current steering angle, a path including the detected object is set, and the virtual vehicle image G5 is moved starting from the vehicle image G1 according to the path. Furthermore, the virtual vehicle image G5 is displayed so as to stop at a position in contact with the object or a position close to the object. In this case, as described above, the vehicle image G1 or the virtual vehicle image G5 is displayed in an enlarged manner, and a staggered display process is performed in a direction opposite to the direction indicated by the direction indicator 26. In addition, the display color of the partial image PG may be changed to, for example, red according to the distance between the virtual vehicle image G5 and the object.

[0177] For example, when entering the road through a gap in a guardrail installed on the roadside, it is necessary to determine the steering angle so as not to touch the end of the guardrail. Fig. 22 As shown in the display example, before the steering angle is determined by the operation of the steering control unit 4, the virtual vehicle image G5 starts to move, and the display of the virtual vehicle image G5 is fixed at the contact position with the detected object (for example, the contact position with the end of the guardrail). As a result, it is easy for the driver to recognize the steering angle required to get to the road without contacting the object. In other words, the steering control unit 4 is operated to determine the steering angle at a position before the steering angle corresponding to the displayed virtual vehicle image G5. In this case, the virtual vehicle image G5 corresponding to the steering angle changed by the operation of the steering control unit 4 can also be displayed separately from the virtual vehicle image G5 displayed by the operation of the direction indicator 26. In this case, by adjusting the steering angle so that the virtual vehicle image G5 corresponding to the steering angle does not overlap on the previously displayed virtual vehicle image G5, it is possible to determine the steering angle that can avoid contact with the object. As a result, a smooth start can be made without contact with the object.

[0178] In addition, the driver may be informed of the steering angle corresponding to the virtual vehicle image G5 displayed by the operation of the direction indicator 26. For example, the maximum steering angle that does not contact the object may be displayed or sounded to prompt the driver to operate, or the operation amount of the steering control unit 4 may be determined by automatic steering operation. Figure 5As shown, an approaching object indicator G6 may also be displayed to display the existence or approaching direction of objects gradually approaching the vehicle 1, such as pedestrians, bicycles, other vehicles, etc., in a state that is easy for the driver to recognize. In addition, in the case where the virtual vehicle image G5 is to be displayed by operating the direction indicator 26, even if the maximum steering angle is searched but no object that may be contacted is detected, the virtual vehicle image G5 may be displayed as moving in the direction corresponding to the maximum steering angle, or may not be displayed. In this case, for example, a display or message such as "Even if operating at the maximum steering angle, the possibility of contact with an object is low." may be output. In addition, as Fig.21 As described above, the margin enlargement process may also be performed on the virtual vehicle image G5. In this case, a display that easily enhances the driver's sense of security when the vehicle is driving while making a sharp turn can be performed.

[0179] The peripheral monitoring program executed by the CPU 14 a of the present embodiment may be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, a floppy disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

[0180] Furthermore, the surrounding monitoring program may be provided by storing it on a computer connected to a network such as the Internet and downloading it via the network. In addition, the surrounding monitoring program executed in this embodiment may be provided or distributed via a network such as the Internet.

[0181] Although the embodiments and variations of the present invention have been described, these embodiments and variations are provided as examples and do not limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the main purpose of the invention. These embodiments and their variations are included in the scope and main purpose of the invention, and are included in the invention described in the claims and their equivalents.

Claims

1. A perimeter monitoring device, comprising: An acquisition unit, for acquiring a current steering angle of the vehicle; An image acquisition unit that acquires a captured image from a capturing unit that captures the periphery of the vehicle; A detection unit, which obtains detection information of objects around the vehicle; and a control unit that causes a composite image to be displayed on a display unit, the composite image including a vehicle image representing the vehicle and a surrounding image representing the surroundings of the vehicle based on the captured image, Only when the object is detected on the route when the vehicle advances a specified distance at the current steering angle, the control unit causes a virtual vehicle image representing the shape of the vehicle to be displayed in the composite image in an overlapping manner on the route to the object based on the position of the vehicle shown in the vehicle image. When the object is detected by the detection unit, the control unit displays the synthesized image in an enlarged manner compared to a display state before the object is detected. When the composite image is displayed in the enlarged manner, the control unit performs a shifted display in which a display position of the vehicle image is shifted and displayed in a direction opposite to a turning direction based on the current steering angle.

2. The perimeter monitoring device according to claim 1, wherein: The virtual vehicle image is movable from a position of the vehicle toward a position of the object.

3. The perimeter monitoring device according to claim 2, wherein: The display mode of the portion of the image in contact with the object within the virtual vehicle image is changed, and the movement of the virtual vehicle image is stopped at the position where the vehicle contacts the object or at a position after advancing to a position close to the position where the vehicle contacts the object within the composite image.

4. The perimeter monitoring device according to claim 3, wherein: The virtual vehicle image is an image composed of polygons and representing the shape of the vehicle. The partial image is a portion of the polygon constituting the virtual vehicle image that is in contact with the object.

5. The perimeter monitoring device according to claim 3, wherein: The control unit changes a display mode of the partial image according to a distance between a position of the object and a position of the vehicle shown in the virtual vehicle image.

6. The perimeter monitoring device according to claim 5, wherein: When the above-mentioned distance is below the specified value, the above-mentioned control unit changes the above-mentioned display mode of the above-mentioned partial image to a first display mode indicating contact, and when the above-mentioned distance changes from a state where the state is below the above-mentioned specified value to a state where the state exceeds the above-mentioned specified value, the above-mentioned control unit changes the above-mentioned display mode from the above-mentioned first display mode to a second display mode indicating contact avoidance.

7. The perimeter monitoring device according to any one of claims 1 to 6, wherein: The vehicle image is a bird's-eye view image of the vehicle.

8. The perimeter monitoring device according to any one of claims 1 to 6, wherein: The virtual vehicle image is an image representing a three-dimensional shape of the vehicle.

9. The perimeter monitoring device according to any one of claims 1 to 6, wherein: The virtual vehicle image is a semi-transparent image showing the shape of the vehicle.

10. The perimeter monitoring device according to any one of claims 1 to 6, wherein: The virtual vehicle image is an image in which the outline of the vehicle is highlighted.

11. The perimeter monitoring device according to any one of claims 1 to 6, wherein: The virtual vehicle image is an image whose transmittance increases as it moves from the outline of the vehicle toward the inside.

12. The perimeter monitoring device according to claim 1, wherein: When the enlarged display is performed, when the vehicle is in a state where the vehicle can move backward, the control unit reduces the degree of the enlarged display compared to a state other than the state where the vehicle can move backward.

13. The perimeter monitoring device according to claim 1, wherein: When the vehicle is in a state where it can move backward, the control unit is configured not to execute the offset display.

14. The perimeter monitoring device according to any one of claims 1 to 6 and claims 12 to 13, wherein: When the virtual vehicle image is not to be displayed, the control unit returns the synthetic image to a display state before the object is detected.

15. The perimeter monitoring device according to any one of claims 1 to 6 and claims 12 to 13, wherein: The control unit temporarily hides the display of the virtual vehicle image, and releases the hiding of the display when a predetermined condition is satisfied.

16. The perimeter monitoring device according to claim 15, wherein: When temporarily suspending display of the virtual vehicle image, the control unit enables redisplay only during the suspension period.

17. The perimeter monitoring device according to any one of claims 2 to 6, claims 12 to 13, and claim 16, wherein: After the control unit stops the movement of the virtual vehicle image at the position where the vehicle contacts the object or at a position close to the position where the vehicle contacts the object, it changes the steering angle from the current steering angle, and is configured to restart the movement of the virtual vehicle image in the direction of the changed steering angle when the vehicle is no longer in contact with the object.

18. The perimeter monitoring device according to any one of claims 1 to 6, claims 12 to 13, and claim 16, wherein: The surrounding monitoring device further includes an operation switch, and the operation switch can switch whether to display the virtual vehicle image on the display unit. When the object is not detected on the route when the vehicle advances a specified distance at the current steering angle, and when the virtual vehicle image can be displayed on the display unit according to the operation of the operating switch, the control unit changes the display mode of the vehicle image.

Citation Information

Patent Citations

  • Driving support device

    JP2008090655A

  • Bird's-eye video generation device, bird's-eye video generation system, bird's-eye video generation method and program

    JP2017114218A

  • Driving assistance device, driving assistance method and program

    JP2018184091A

  • Surroundings monitoring device

    JP2018203031A