Peripheral image generation device and display control method

Abstract

A surrounding image generation apparatus and a display control method are provided. A surrounding image generation apparatus for a vehicle includes an image acquisition unit (F1), an image storage unit (M1), and a composite image generation unit (F7). The image acquisition unit acquires a plurality of camera images obtained from a plurality of cameras each of which captures a surrounding of the vehicle. The image storage unit stores a travel direction camera image which is a part or all of an image included in the plurality of camera images and captured at least in a travel direction of the vehicle. The composite image generation unit generates a bottom transparent image which is a composite image that transmits a bottom of the vehicle, based on the plurality of camera images and the travel direction camera image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a display control method and a surrounding image generation apparatus for displaying an image of a road surface located below a vehicle body. BACKGROUND

[0002] US 2018 / 111553 A1 (corresponding to JP 2016-197785 A) discloses a technique for displaying an image of a road surface located below a vehicle. By using an image of a front camera acquired while the vehicle is traveling forward, a road surface image is synthesized with another camera image. Since the synthesized image showing the road surface below the vehicle body in this way is an image in which the vehicle body bottom is transmitted, it will also be referred to as an underfloor transparent image hereinafter.

[0003] JP 3797343 B2 discloses a configuration in which a range in which a vehicle exists is shown with a frame line in a bird's-eye view image showing the surroundings of the vehicle. SUMMARY

[0004] US 2018 / 111553 A1 does not disclose a configuration in which a vehicle boundary line indicating a range of the vehicle is superimposed on the underfloor transparent image. In such a configuration, it is difficult for the user to recognize where the vehicle body exists even by watching the underfloor transparent image. If the user erroneously recognizes the range of existence of the vehicle body, the possibility of the vehicle coming into contact with an obstacle increases.

[0005] In response to such a difficulty, a configuration in which a frame line corresponding to the vehicle boundary line of JP 3797343 B2 is displayed is also assumed. However, in the underfloor transparent image, various lines can be superimposed, such as a tire boundary line indicating a range of existence of a tire and a guide line indicating a predicted trajectory of the vehicle according to a steering angle. When the frame is simply arranged as the vehicle boundary line on the underfloor transparent image, it can be difficult to distinguish the vehicle boundary line from other display lines. Therefore, it is still difficult for the user to recognize the range of existence of the vehicle.

[0006] In view of the above-described difficulties, an object of the present disclosure is to provide a display control method and a surrounding image generation apparatus capable of displaying an underfloor transparent image in which a user can easily recognize a range of existence of a vehicle body.

[0007] According to an aspect of the present disclosure, a surrounding image generation device for a vehicle includes an image acquisition unit, an image storage unit, and a composite image generation unit. The image acquisition unit acquires a plurality of camera images obtained from a plurality of cameras each of which captures a surrounding of the vehicle. The image storage unit stores a travel direction camera image that is a part or all of an image included in the plurality of camera images and captured at least in a travel direction of the vehicle. The composite image generation unit generates a bottom transparent image that is a composite image that transmits a bottom of the vehicle, based on the plurality of camera images and the travel direction camera image. The composite image generation unit generates an image in which a vehicle boundary line indicating a boundary of a range of a vehicle body is superimposed on an image showing a part or all of a ground surface located below the vehicle and a decoration having a predetermined pattern is added to the vehicle boundary line, as the bottom transparent image.

[0008] According to the above-described configuration, the vehicle boundary line to which the decoration having the predetermined pattern is added is displayed. Since the vehicle boundary line is highlighted by the decoration, a user can easily recognize a range in which the vehicle body exists.

[0009] According to another aspect of the present disclosure, a display control method controls display of an image for supporting a driving operation of a vehicle. The display control method includes acquiring a plurality of camera images obtained from a plurality of cameras each of which captures a surrounding of the vehicle, storing a travel direction camera image that is included in the plurality of camera images and captured in a travel direction of the vehicle, and generating a bottom transparent image that is a composite image that transmits a bottom of the vehicle, based on the plurality of camera images and the travel direction camera image. The bottom transparent image is an image in which a vehicle boundary line indicating a boundary of a range of a vehicle body is superimposed on an image showing a part or all of a ground surface located below the vehicle and a decoration having a predetermined pattern is added to the vehicle boundary line.

[0010] According to the above-described configuration, an effect similar to that of the surrounding image generation device can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description made in conjunction with the accompanying drawings. In the drawings:

[0012] Figure 1 is a block diagram showing an overall configuration of a surrounding display system;

[0013] Figure 2 is a diagram showing an example of a mounting position and a capturing range of each camera 2;

[0014] Figure 3This is a diagram showing an example of the installation location and capture range of each sonar 8;

[0015] Figure 4 This is a block diagram illustrating the function of the image generation ECU 1;

[0016] Figure 5 This is a diagram showing the virtual viewpoint VP;

[0017] Figure 6 This is a diagram showing the projection surface TS;

[0018] Figure 7 This is a diagram showing the image data used to generate the under-transparent image;

[0019] Figure 8 This is an example diagram showing a transparent image at the bottom;

[0020] Figure 9 yes Figure 8 An enlarged view of the area surrounded by dashed lines;

[0021] Figure 10 This is a schematic diagram showing the decorative image Ef;

[0022] Figure 11 This is a diagram showing the decorative image Ef added to the vehicle boundary line Lvc;

[0023] Figure 12 This is a flowchart illustrating the operation process of the image generation ECU 1;

[0024] Figure 13 This is a diagram illustrating an example configuration for displaying an image DP;

[0025] Figure 14 This is a diagram showing an example of a modified decorative image Ef;

[0026] Figure 15 This is a diagram showing an example of a modified decorative image Ef;

[0027] Figure 16 This is an example diagram showing a transparent image at the bottom;

[0028] Figure 17 yes Figure 16 An enlarged view of the area surrounded by dashed lines;

[0029] Figure 18 This is a diagram showing an example of a modified decorative image Ef;

[0030] Figure 19 This is a diagram showing an example of a modified decorative image Ef;

[0031] Figure 20is a diagram showing a modification example of the decorative image Ef;

[0032] Figure 21 is a diagram showing a modification example of the decorative image Ef;

[0033] Figure 22 is a diagram showing an example of a control mode in which the display of the decorative image Ef is changed in accordance with the moving state of the vehicle; and

[0034] Figure 23 is a diagram showing another setting example of the virtual viewpoint VP. DETAILED DESCRIPTION

[0035] Hereinafter, an embodiment of a surround display system Sys according to the present disclosure will be described with reference to the drawings. The surround display system Sys displays a surround image of a vehicle equipped with the system on a display. Hereinafter, the vehicle V equipped with the surround display system Sys will also be referred to as a subject vehicle.

[0036] The subject vehicle of the present embodiment is, for example, a four-wheel drive vehicle, which is assumed to travel not only on a road on which asphalt or the like is paved (i.e., on a road), but also off-road, and has a normal mode and an off-road mode as driving modes. The normal mode is a driving mode suitable for a road, and the off-road mode is a driving mode suitable for off-road. Each driving mode has a different method of controlling the distribution of driving force to front wheels, rear wheels, left wheels, and right wheels. Off-road here mainly refers to a ground having a large irregularity, such as a rocky road. Of course, off-road can also be understood as a ground other than a road, i.e., an unmaintained ground. The present disclosure can also be applied to a vehicle that is not expected to travel off-road. The subject vehicle can be a vehicle whose driving source is an engine, or can be an electric vehicle or a hybrid vehicle having a motor as a driving source.

[0037] In the following description, each of the front-rear, left-right, and up-down directions is defined with reference to the subject vehicle. Specifically, the front-rear direction corresponds to the longitudinal direction of the subject vehicle. The left-right direction corresponds to the width direction of the subject vehicle. The up-down direction corresponds to the height direction of the subject vehicle. From another viewpoint, the up-down direction corresponds to a direction perpendicular to a plane parallel to the front-rear direction and the left-right direction. In the present disclosure, the plane perpendicular to the height direction is also referred to as a vehicle horizontal plane. Furthermore, a direction perpendicular to the height direction and including the front-rear direction and the left-right direction is also referred to as a horizontal direction of the vehicle. The horizontal direction of the vehicle corresponds to a direction away from the subject vehicle.

[0038] In addition, "parallel" in the present disclosure is not limited to a state of being completely parallel. For example, "parallel" can be inclined by about 20 degrees. That is, the term "parallel" includes a state of being substantially parallel. The expression "vertical" in the present disclosure is not limited to a state of being completely vertical, but can be inclined by about 20 degrees.

[0039] (Explanation of overall configuration)

[0040] Figure 1 is a drawing showing an example of a schematic configuration of a surrounding display system Sys according to the present disclosure. As shown in Figure 1 , the surrounding display system Sys includes an image generation ECU 1, a plurality of cameras 2, a display 3, a touch panel 4, an operation button 5, a vehicle state sensor 6, a sonar ECU 7, and a plurality of sonars 8. ECU in each name is an abbreviation of an electronic control unit.

[0041] The image generation ECU 1 is communicably connected to each of the plurality of cameras 2, the display 3, the touch panel 4, the operation button 5, the vehicle state sensor 6, and the sonar ECU 7. The plurality of sonars 8 is connected to the sonar ECU 7. The above-described various devices or sensors and the image generation ECU 1 can be connected individually by dedicated lines, or can be connected via an in-vehicle network as a communication network built in a vehicle. For example, the camera 2 and the image generation ECU 1 can be directly connected by a dedicated video signal line.

[0042] The image generation ECU 1 generates a composite image of a vehicle surrounding area viewed from an arbitrary viewpoint based on image data input from each camera 2 as an image for supporting a driving operation of the vehicle V, and displays the generated image on the display 3. The image generation ECU 1 corresponds to a surrounding image generation device. The image generation ECU 1 is implemented by using a computer. That is, the image generation ECU 1 includes a processing unit 11, a RAM 12, a storage device 13, an I / O 14, a bus or the like connecting these configurations.

[0043] The processing unit 11 is a hardware (in other words, an arithmetic core) for arithmetic processing in combination with the RAM 12. The processing unit 11 is, for example, a CPU. The processing unit 11 accesses the RAM 12 to execute various processes for providing functions of respective functional blocks. The RAM 12 is a volatile storage medium.

[0044] The storage 13 includes a non-volatile storage medium such as a flash memory. The storage 13 stores an image generation program Pg as firmware and various drawing data for use in generating a composite image by the composite image generation unit F7. The drawing data includes data indicating the shape of the projection surface TS (which will be described later), data of a 3D model of the appearance of the subject vehicle, data of a 3D model of a predetermined component of the subject vehicle, and the like. Examples of the component for which the 3D model data is prepared include a tire, a steering wheel, an instrument panel, a pillar, and a body panel. The execution of the image generation program Pg by the processing unit 11 corresponds to the execution of a display control method, which is a method corresponding to the image generation program Pg. The I / O 14 is a circuit module for communicating with another device. The I / O 14 can be implemented by using an analog circuit element, an IC, or the like. Details of the image generation ECU 1 will be described later.

[0045] The cameras 2 are on-vehicle cameras that capture the surroundings of the subject vehicle and output data of the captured images to the image generation ECU 1. Each camera 2 includes at least a lens and an image sensor, and electronically acquires an image showing the periphery of the subject vehicle. Each of the plurality of cameras 2 is attached to a predetermined position in the subject vehicle in a predetermined posture so as to capture a different range from each other. As shown in Figure 2 The periphery display system Sys of the present embodiment includes a front camera 2F, a rear camera 2B, a left side camera 2L, and a right side camera 2R as the cameras 2. These four cameras 2 are arranged at different positions in the subject vehicle and capture different directions around the subject vehicle. Specific examples are described below.

[0046] The front camera 2F is a camera that takes an image of the front of the vehicle at a predetermined angle of view. The front camera 2F is attached to the front end of the subject vehicle, for example, the front grille, with the optical axis 2F of the front camera 2F facing the front of the subject vehicle. The rear camera 2B is a camera that takes an image of the rear of the vehicle at a predetermined angle of view. The rear camera 2B is arranged at a predetermined position on the rear surface of the vehicle body, for example, near the rear license plate or the rear window, in a posture with the optical axis 2Ba pointing toward the rear of the subject vehicle. The left side camera 2L is a camera that captures the left side of the subject vehicle. The left side camera 2L is attached to the left side mirror with the optical axis 2La facing the left side of the subject vehicle. The right side camera 2R is a camera that captures the right side of the subject vehicle. The right side camera 2R is attached to the right side mirror with the optical axis 2Ra of the right side camera 2R facing the right side of the subject vehicle.

[0047] A wide-angle lens such as a fisheye lens is used as the lens for each camera 2, and each camera 2 has a viewing angle θ of 180 degrees or more. Thus, by using four cameras 2, it is possible to take a picture of the entire surroundings of the subject vehicle. The mounting position of each camera 2 described above can be changed as appropriate. The front camera 2F can be attached to a rearview mirror, an upper end of a windshield, or the like. The right camera 2R or the left camera 2L can be disposed near the base of an A-pillar or a B-pillar. The surround display system Sys can include a camera 2 mounted on the roof. Some or all of the cameras 2 can be retrofitted cameras on the roof, on the instrument panel, near the window frame, or the like.

[0048] Data indicating the mounting position and posture of each camera 2 in the vehicle V (hereinafter, mounting position data) is stored in the storage device 13. The mounting position of each camera 2 can be expressed as a point on a three-dimensional coordinate system centered on an arbitrary position of the vehicle V (hereinafter, vehicle three-dimensional coordinate system), for example. The X-axis forming the vehicle three-dimensional coordinate system can be an axis parallel to the right-left direction of the vehicle, for example. Further, the Y-axis can be an axis parallel to the front-rear direction. The Z-axis can be an axis parallel to the height direction. It is assumed that the right direction of the vehicle corresponds to the positive direction of the X-axis, the front of the vehicle corresponds to the positive direction of the Y-axis, and the upper direction of the vehicle corresponds to the positive direction of the Z-axis. The center of the vehicle three-dimensional coordinate system can be the center of the rear axle, for example.

[0049] The display 3 is provided with a thin display panel such as a liquid crystal display, and is a device for displaying various information and images. The display 3 is disposed on an instrument panel or the like of the subject vehicle so that the user can visually recognize the screen. The display 3 can be integrated with the image generation ECU 1 by being disposed in the same housing as the image generation ECU 1. Of course, the display 3 can be a device separate from the image generation ECU 1. The display 3 includes a touch panel 4 stacked on the display panel, and is capable of accepting user operations. The touch panel 4 is a capacitive touch panel, for example, and outputs a signal indicating the touch position of the user. The user here refers mainly to the occupant of the driver's seat (so-called driver). In addition to the driver, the user can include an occupant in the passenger's seat, or the like.

[0050] The operation button 5 is an operation member that receives a user's operation on the display content of the display 3. The operation button 5 is a switch for displaying a composite image generated by the image generation ECU 1 on the display 3 and changing a virtual viewpoint of the composite image. The operation button 5 is provided, for example, on a steering wheel of the subject vehicle, and mainly receives an operation from a driver. The user can perform various operations on the surround display system Sys via the operation button 5 and the touch panel 4. When the user operates any one of the operation button 5 and the touch panel 4, an operation signal indicating the operation content is input to the image generation ECU 1. The operation button 5 can also be referred to as a multi-information switch. The operation button 5 can be arranged on an instrument panel.

[0051] The surround display system Sys can include a travel mode switch as an operation member for the driver to switch the driving mode of the vehicle V to the off-road mode. The operation member for switching the driving mode can be of a dial type. The shift lever can have a function as the driving mode switch.

[0052] The vehicle state sensor 6 is a sensor that detects a state quantity related to the travel control of the subject vehicle. The vehicle state sensor 6 includes a shift sensor, a vehicle speed sensor, a steering angle sensor, an acceleration sensor, and the like. The shift sensor is a sensor that detects a shift position, in other words, a sensor that detects a setting state of a transmission. The vehicle speed sensor is a sensor that detects a travel speed of the subject vehicle. The steering angle sensor is a sensor that detects a rotation angle (so-called steering angle) of a steering wheel. The acceleration sensor is a sensor that detects an acceleration in at least one of a vehicle front-rear direction, a vehicle left-right direction, and a vehicle up-down direction acting on the subject vehicle. Here, a 3-axis acceleration sensor is assumed to be employed as the acceleration sensor. The detection value of the acceleration sensor can be used to determine a vehicle attitude with respect to a horizontal plane. The type of the sensor for the surround display system Sys as the vehicle state sensor 6 can be designed as appropriate, and it is not necessary to include all of the above-described sensors. Furthermore, a vehicle height sensor, a gyro sensor, a direction angle sensor, and the like can be included in the vehicle state sensor 6. Each sensor outputs data indicating a current value (i.e., a detection result) of a physical state quantity to be detected to the image generation ECU 1.

[0053] The sonar ECU 7 is an ECU that controls the operation of the sonar 8. The sonar 8 emits an ultrasonic wave as a probe wave, and receives a reflection wave reflected by the ultrasonic wave to detect an object present around the subject vehicle. Furthermore, the sonar 8 detects a distance from the sonar 8 to the object based on a time from the emission of the ultrasonic wave to the return as the reflection wave. Distance information of the object detected by each sonar 8 is input to the image generation ECU 1 via, for example, the sonar ECU 7.

[0054] In this embodiment, as an example, as Figure 3Eight sonars 8 are provided as shown. Specifically, four sonars 8A to 8D are arranged from the right corner portion to the left corner portion of the front bumper. In addition, four sonars 8E to 8H are arranged from the right corner portion to the left corner portion of the rear bumper of the subject vehicle. With such an arrangement of the sonars 8, the sonar ECU 7 is able to detect an object present in front of or behind the subject vehicle. The arrangement of the sonars 8 is an example, and is not limited to Figure 3 The example shown. In addition, the sonars 8 can be attached to a side sill, a fender, or a door panel so as to emit ultrasonic waves toward the side of the vehicle.

[0055] In addition, the sonar ECU 7 is able to recognize the relative position of an object present around the subject vehicle by combining the detection results of the plurality of sonars 8. For example, based on the distance to the same object detected by each of two or more sonars 8, the direction of the object relative to the subject vehicle is derived as the direction of the detected object. When the sonar 8 detects an object, the sonar ECU 7 inputs data indicating the direction and distance of the object to the image generation ECU 1 as a detection result. In addition, the sonar ECU 7 can determine the height of the detected object by analyzing the reflection intensity and the received waveform, and output the position and height information as the detected object information to the image generation ECU 1. The sonar ECU 7 can be integrated with the image generation ECU 1.

[0056] (Configuration of the image generation ECU)

[0057] As Figure 4 shown, the image generation ECU 1 includes, as functional units, an image acquisition unit F1, an operation reception unit F2, a vehicle state acquisition unit F3, an obstacle information acquisition unit F4, an image recognition unit F5, a display control unit F6, a composite image generation unit F7, a display image generation unit F8, and an image output unit F9. In addition, the image generation ECU 1 includes an image storage M1, which is a memory for temporarily storing image data. The image storage M1 is implemented by using, for example, a part of a storage area included in the RAM 12. In the image storage M1, the position information and the orientation information of the vehicle V at the time of capturing an image can be stored in association with the image of each camera. The image storage M1 corresponds to the image storage unit.

[0058] The image acquisition unit F1 acquires a camera image, which is an image generated by each of the four cameras 2. By combining the images of the cameras 2, image data for the entire surroundings of the subject vehicle is obtained. The image acquisition unit F1 converts the image signals input from the cameras 2 into digital image data in a predetermined data format, and then outputs the image signals to the image recognition unit F5 and the composite image generation unit F7.

[0059] Further, for example, each time the subject vehicle moves a predetermined storage distance, the image acquisition unit F1 stores the position information and the body attitude information of the subject vehicle, which are acquired respectively, in association with each other in the image storage M1. The storage distance can be, for example, 0.1 m, 0.3 m, 0.5 m, or the like. When the vehicle travels forward, the image acquisition unit F1 can store captured image data of, for example, the front camera 2F including the ground from immediately below the front end of the vehicle to 3 m ahead in the image storage M1. Further, when the vehicle travels backward, the image acquisition unit F1 can store captured image data of the rear camera 2B including the ground from immediately below the rear end of the vehicle to 3 m behind in the image storage M1.

[0060] In addition, various modes can be employed as the storage mode of the image data in the image storage M1. For example, the image acquisition unit F1 overwrites and saves new data in the oldest updated area in the image storage M1. That is, the image storage M1 can be configured as a ring buffer. The ring buffer is a storage area logically arranged in a ring shape. The image acquisition unit F1 can store image frames acquired within a predetermined recent time in an area different from the above-described image data for each fixed distance. The composite image generation unit F7 and the display image generation unit F8 refer to new image data stored in the image storage M1. The image acquisition unit F1 can store images acquired from the cameras 2 after performing image processing such as distortion correction, enlargement or reduction, or cropping on the images acquired from the cameras 2 according to the lens characteristics.

[0061] The operation reception unit F2 receives operation signals output from the operation buttons 5 and the touch panel 4. The operation signal is a signal indicating the content of the user's operation on the operation buttons 5 and the touch panel 4. As a result, the operation reception unit F2 receives the user's instruction operation for displaying the composite image or the camera image. The operation reception unit F2 inputs data corresponding to the received operation signal to the display control unit F6.

[0062] The vehicle state acquisition unit F3 acquires information indicating the state of the subject vehicle from other devices (e.g., the vehicle state sensor 6) provided in the subject vehicle. The vehicle state acquisition unit F3 acquires the shift position, acceleration for each detected axis direction, vehicle speed, steering angle, and the like from various sensors. The vehicle state acquisition unit F3 can acquire various state quantities from other ECUs. For example, information such as the steering angle can be acquired from an ECU that constitutes a steering system. Further, the vehicle state acquisition unit F3 can acquire the amount of change in the moving distance, direction, and the like with respect to a predetermined time ago from the image recognition unit F5. That is, the vehicle state acquisition unit F3 can acquire various information related to the state of the subject vehicle not only from sensors but also from another ECU, the image recognition unit F5, and the like. Further, the vehicle state acquisition unit F3 can combine a plurality of types of information to calculate the travel distance of the vehicle, the amount of change in the vehicle body direction, and the like. The direction of the vehicle can include not only the direction in which the vehicle faces (so-called yaw angle) but also the pitch angle, roll angle, and the like.

[0063] The obstacle information acquisition unit F4 acquires information on a three-dimensional object existing around the vehicle from the sonar ECU 7. That is, the size, height, relative position, and the like of the detected three-dimensional object are acquired. Further, when the sonar ECU 7 is provided with a discriminator that identifies the type of the detected object, such as a low-profile object or an obstacle, by analyzing a characteristic quantity of the signal waveform of the received reflected wave, the recognition result of the detected object by the sonar ECU 7 is also acquired. The low-profile object here is a three-dimensional object having a height of a predetermined value (e.g., 4 cm) or less, and conceptually refers to a three-dimensional object that the vehicle V can overcome. For example, the low-profile object refers to a rock area protruding from the road surface by an amount less than a predetermined threshold, a barrier placed in a parking lot, and the like. Further, the obstacle here conceptually refers to a three-dimensional object that the vehicle V should avoid. For example, the obstacle refers to a rock, guardrail, wall, or another moving body having a height such that contact with the vehicle body of the vehicle V is made. In addition to other vehicles, other moving objects can include bicycles, pedestrians, animals, and the like. The obstacle information acquisition unit F4 can perform so-called sensor fusion that detects an object by combining the information from the sonar ECU 7 with the recognition result of the image recognition unit F5.

[0064] The image recognition unit F5 detects the position of a predetermined detection target, the type of a predetermined detection target, and the like by analyzing an image input from the camera 2. The image recognition unit F5 has a function as a discriminator for recognizing the type of an object based on, for example, an image feature vector. The image recognition unit F5 can recognize an object by using, for example, a convolutional neural network (CNN) or a deep neural network (DNN) technique that applies deep learning. The detection target includes a pedestrian, another vehicle, a road mark (for example, a lane mark that can be given to a paved road), or a road edge. In addition to a line as a lane mark, the lane mark line can include a line indicating a parking frame or the like.

[0065] Further, the image recognition unit F5 can estimate the amount of movement and the amount of change in orientation of the subject vehicle based on the amount of change in the position of a feature point appropriately set on the image between image frames. For example, the image recognition unit F5 can estimate the amount of change in the current position with respect to a certain reference point, the orientation of the vehicle body, the yaw rate, and the like by using an optical flow method or the like, which is an inter-frame difference method. The optical flow is information that shows the movement of an object reflected in image data as a vector. The object recognition result of the image recognition unit F5 or the like and the estimation result of the position of the subject vehicle are output to the display control unit F6, the vehicle state acquisition unit F3, the composite image generation unit F7, the display image generation unit F8, and the like.

[0066] The display control unit F6 comprehensively controls the entire image generation ECU 1. For example, the display control unit F6 controls the composite image generation unit F7 and the display image generation unit F8 based on the information input from the operation reception unit F2 and the vehicle state acquisition unit F3, and causes the composite image generation unit F7 and the display image generation unit F8 to generate a composite image and a display image DP corresponding to the running state of the subject vehicle and the setting of the user.

[0067] The display control unit F6 determines whether the running direction of the subject vehicle is a forward direction or a rearward direction based on, for example, a signal from a shift sensor or the direction of rotation of a tire. The display control unit F6 can determine whether the running environment is off-road based on an input signal from a drive mode switch.

[0068] Further, the display control unit F6 sets the position and the line-of-sight direction of the virtual viewpoint VP based on at least one of the running direction of the subject vehicle, a signal from the touch panel 4, and a signal from the operation button 5 for generating a composite image described later. For example, as shown in FIG. 6, the display control unit F6 sets the position and the line-of-sight direction of the virtual viewpoint VP based on the running direction of the subject vehicle, the signal from the touch panel 4, and the signal from the operation button 5. Figure 5As illustrated, a bird's-eye view perspective VPb, a driver's eye point VPd, and the like can be employed as virtual viewpoints VP that can be set. The bird's-eye view perspective VPb is a setting mode of the virtual viewpoint VP in which the viewpoint position is directly above the subject vehicle and the viewing direction is directly below. The bird's-eye view perspective VPb is applied when a bird's-eye view image that is an image of the subject vehicle and its surrounding environment viewed from directly above the vehicle is generated. The viewing angle of the bird's-eye view perspective VPb can be appropriately adjusted so as to include a predetermined range around the vehicle. The viewpoint position of the bird's-eye view perspective VPb is not limited to directly above the vehicle, and can be a position moved to the rear side, the front side, or the lateral direction from directly above the vehicle. The bird's-eye view perspective VPb corresponds to an example in which the virtual viewpoint VP is arranged at an outdoor viewpoint outside the vehicle interior.

[0069] The driver's eye point VPd is a virtual viewpoint VP in which the viewpoint position is set to a presumed position of the driver's eyes in the vehicle interior. For example, the line-of-sight direction of the driver's eye point VPd can be set to be obliquely forward and obliquely downward so as to include the vicinity of the front wheels. The obliquely downward direction can be, for example, a direction pointing downward by about 20° to 30° from the horizontal plane of the vehicle. The line-of-sight direction of the driver's eye point VPd can be changed to an arbitrary direction based on the user's operation (e.g., swiping) of the touch panel 4, while using the obliquely forward downward direction as a reference direction. The viewing angle of the driver's eye point VPd can also be appropriately adjusted so as to include a predetermined range around the vicinity of the front wheels. For example, the viewing angle of the driver's eye point VPd can be 100 degrees in the horizontal direction, 60 degrees in the vertical direction, or the like.

[0070] The presumed position of the driver's eyes is set, for example, in the vicinity of the headrest of the driver's seat. An eyelip set for each vehicle type can be used as the presumed position of the driver's eyes. The eyelip is a virtual elliptical form defined based on an eye range that statistically represents the spatial distribution of the eye points of the occupants. The driver's eye point VPd corresponds to an example in which the virtual viewpoint VP is arranged in a vehicle interior viewpoint in the vehicle interior. Furthermore, the position of the driver's eye point VPd can be arranged at a position deviated from the presumed position of the driver's eyes. For example, the driver's eye point VPd can be arranged at a position deviated from the presumed position of the driver's eyes by a predetermined amount, for example, at an intermediate position between the driver's seat and the passenger seat.

[0071] ​​The image generation ECU 1 of this embodiment includes a floor display mode as an operation mode for displaying a floor transparent image CP, which is a composite image in which the floor of the vehicle body is transmitted. The display control unit F6 can switch the display mode based on at least one of the shift lever being set to a predetermined position and a user operation. The display control unit F6 sets the display mode to the floor display mode, for example, based on the fact that the transparent floor display condition described later is satisfied. The display control unit F6 causes the composite image generation unit F7 to generate a floor transparent image CP in the floor display mode. The floor transparent image CP generated in a state in which the virtual viewpoint VP is set to the driver viewpoint VPd is also referred to as a driver viewpoint image CPd hereinafter.

[0072] The composite image generation unit F7 performs image processing for generating a composite image such as the floor transparent image CP. The composite image generation unit F7 projects data of a plurality of camera images onto a virtual projection surface TS corresponding to the periphery of the subject vehicle, and generates a composite image indicating the periphery of the subject vehicle seen from the virtual viewpoint VP using the data on the projection surface.

[0073] As Figure 6 As conceptually illustrated in FIG. 6, the projection plane TS is a virtual three-dimensional plane corresponding to the periphery region of the subject vehicle. A central region of the projection surface TS is defined as a vehicle region RO, which is the position of the subject vehicle. The vehicle region RO is set as a rectangular flat surface portion, for example. The vehicle region RO is set to include a region overlapping the subject vehicle in a plan view, in other words, a region that cannot be directly imaged by the cameras 2. For example, the projection surface TS is formed as a plane along the horizontal direction of the vehicle in the vicinity of the vehicle region RO, and has a bowl-shaped shape having a downwardly convex curved surface in which the inclination (gradient) increases as the distance from the vehicle region RO increases. Which portion of the camera images is projected on each portion of the projection surface TS is associated with corresponding information such as table data. The projection of the camera images on the projection surface TS corresponds to the texture mapping of the camera images on the projection surface TS.

[0074] The shape of the projection surface TS can be appropriately changed. For example, the entire region of the projection surface TS can be set as a horizontal plane. Further, the projection surface TS can have a shape of a curved surface region that is convex downward from the edge portion of the vehicle region RO, in other words, a shape that does not have a planar region surrounding the vehicle region RO.

[0075] The operation of the composite image generation unit F7 is controlled by the display control unit F6. For example, the virtual viewpoint VP for generating the composite image is controlled by the display control unit F6. The composite image generation unit F7 includes a camera image synthesis unit F71 and a vehicle image addition unit F72 as subfunctions. The camera image synthesis unit F71 projects each camera image onto the above-mentioned projection surface TS. The vehicle image addition unit F72 arranges the image of the vehicle V at a predetermined position on the projection surface TS on which the camera image is projected. Details of the camera image synthesis unit F71 and the vehicle image addition unit F72 will be described later.

[0076] The display image generation unit F8 generates a display image DP for display on the display 3. The display image generation unit F8 generates the display image DP including the composite image and the camera image by using the composite image generated by the composite image generation unit F7 and the camera image acquired by the image acquisition unit Fl. The combination of images included in the display image DP is determined by the display control unit F6 in accordance with the traveling direction of the subject vehicle and the user operation to the touch panel 4, and the like. That is, the operation of the display image generation unit F8 is controlled by the display control unit F6.

[0077] The image output unit F9 converts the display image DP generated by the display image generation unit F8 into a video signal of a predetermined signal format, outputs the signal to the display 3, and displays the display image DP on the display 3. As a result, the composite image showing the surroundings of the subject vehicle from the virtual viewpoint VP is displayed on the display 3.

[0078] (Method for generating composite image)

[0079] In the following, the operation when the composite image generation unit F7 generates the underlay transparent image CP will be described. When the underlay display condition described later is satisfied, the camera image synthesis unit F71 projects data (value of each pixel) included in each camera image input from the image acquisition unit Fl onto the projection surface TS in the virtual three-dimensional space. The projection position of each camera image with respect to the projection surface TS is associated in advance with corresponding information such as table data.

[0080] For example, the camera image synthesis unit F71 projects the image data of the front camera 2F onto the front region PF of the projection surface TS, and projects the image data of the rear camera 2B onto the rear region PB of the projection surface TS. Furthermore, the camera image synthesis unit F71 projects the image data of the left camera 2L onto the left region PL of the projection surface TS, and projects the image data of the right camera 2R onto the right region PR of the projection surface TS. Regarding the overlapping region, which is the area captured by the two cameras in an overlapping manner, a method of mixing the images captured by the two cameras at a predetermined ratio or a method of connecting the images captured by the two cameras with a predetermined boundary line can be used.

[0081] Furthermore, based on instructions from the display control unit F6, an image of the portion where the vehicle region R0 is estimated to be located is extracted from past image data stored in the image memory M1, and the extracted image is projected after performing rotation corrections, etc. The image projected onto the vehicle region R0 is hereby referred to as the under-position image, because the image projected onto the vehicle region R0 is the image of the area currently located directly below the subject vehicle, i.e., the under-position image.

[0082] For example, such as Figure 7 As shown, when the subject vehicle moves forward, the underlying image is generated using an image captured by the front camera 2F, which is located 3m behind the current position. When the subject vehicle moves backward, the underlying image is generated using an image captured by the rear camera, which is located 3m in front of the current position. That is, the underlying image is generated using an image when the subject vehicle is at a predetermined distance from the current position on the opposite side of the direction of travel. When the subject vehicle moves forward, the underlying image can be generated using front camera images captured at multiple time points from when the vehicle is 3m behind the current position to the current position. The same method can be applied when the vehicle moves backward. The image data projected onto the vehicle region R0 (in other words, the underlying portion) can be updated as needed as the vehicle V moves.

[0083] Next, under the control of the display control unit F6, the composite image generation unit F7 sets a virtual viewpoint VP in the three-dimensional space including the projection surface TS. The composite image generation unit F7 can set a virtual viewpoint VP pointing in any field of view at any viewpoint position in the three-dimensional space.

[0084] Then, the vehicle image adding unit F72 arranges various image elements on the virtual three-dimensional space including the projection surface TS on which the camera image is projected, with respect to the subject vehicle. When the virtual viewpoint VP is arranged at the vehicle interior outside, such as the bird's-eye view viewpoint VPb, the vehicle image adding unit F72 adds a vehicle boundary line Lvc indicating a range of the road surface on which the vehicle body is present, a tire boundary line Lt indicating a range of the road surface on which the tire is present, and the like. The vehicle boundary line Lvc is a line showing an outline of an area formed by projecting the vehicle body perpendicularly to the road surface, and corresponds to a line showing an external shape of the vehicle V from the bird's-eye view. Similarly, the tire boundary line Lt is a line showing an outline of an area formed by projecting the tire perpendicularly to the road surface (i.e., orthogonal projection), and corresponds to a line showing an external shape of the tire from the bird's-eye view. The vehicle image adding unit F72 of the present embodiment adds a predetermined decorative image Ef to the vehicle boundary line Lvc. The display state of each of the vehicle boundary line Lvc, the decorative image Ef, the tire boundary line Lt, and the like is controlled by the display control unit F6. Details of the vehicle boundary line Lvc, the decorative image Ef, and the tire boundary line Lt will be described later.

[0085] Further, when the virtual viewpoint VP is arranged at the vehicle interior, such as the driver's viewpoint VPd, the synthetic image generating unit F7 arranges a 3D tire model, a 3D vehicle body model, and a 3D interior model, in addition to the vehicle boundary line Lvc and the tire boundary line Lt, at predetermined positions in the three-dimensional space. The 3D tire model is a 3D model of the tire and is set as a whole to be semi-transparent. The 3D tire model can be composed only of an opaque or semi-transparent outline line. That is, a portion other than the outline line of the 3D tire model can be set to be colorless and transparent.

[0086] Semi-transparent here is not limited to a state in which the transparency is 50%, and can include, for example, a state in which the transparency is 80%. In other words, the expression "semi-transparent" can include a level that indicates the presence thereof obscurely. The transparency is a parameter indicating that the higher the value, the more transparent. In the present disclosure, a state in which the transparency is 100% means a state of being completely transparent. The transparency or opacity of an image element is a concept corresponding to an alpha value of a pixel. Generally, the degree of transparency becomes higher as the alpha value increases.

[0087] The 3D tire model is a 3D model of the tire and is set as a whole to be semi-transparent. The 3D tire model can be composed only of an opaque or semi-transparent outline line. That is, a portion other than the outline line of the 3D tire model can be set to be colorless and transparent. The 3D interior model is a 3D model of, for example, an instrument panel, a steering wheel, an A-pillar, and the like.

[0088] The 3D tire model, 3D body model, 3D interior model, and the like are 3D models that show constituent members of the subject vehicle, and can be referred to as constituent member models. The 3D models arranged in three-dimensional space for generating the composite image can be changed as appropriate, and for example, the arrangement of the 3D tire model, 3D body model, 3D interior model, and the like can be omitted.

[0089] In addition, the vehicle image adding unit F72 can draw a scale line indicating a point separated by a predetermined distance from the end of the vehicle or a vehicle width line indicating the vehicle width. Further, the vehicle image adding unit F72 can draw a planned trajectory line as another image element related to the vehicle V, which is a line indicating a planned travel trajectory according to the steering angle of the vehicle V. The vehicle width line can also be used as the planned trajectory line. The vehicle boundary line Lvc, the tire boundary line Lt, the planned travel trajectory line, and the like can be referred to as a vehicle information image because they are images that show information about the vehicle V. In addition, the planned trajectory line, the scale line, the vehicle width line, and the like can also be used as information for guiding the driving operation of the driver. Therefore, the display line such as the planned trajectory line can also be referred to as a guide line. The guide line can also include the vehicle boundary line Lvc and the tire boundary line Lt.

[0090] Then, the composite image generating unit F7 renders various image elements existing in the three-dimensional space including the projection plane TS according to the virtual viewpoint VP, and cuts out the image elements included in the predetermined angle of view when viewed from the virtual viewpoint VP. As a result, the composite image generating unit F7 generates a composite image that shows the underbody region of the subject vehicle and the region around the subject vehicle as seen from the virtual viewpoint VP. That is, the composite image generating unit F7 generates a composite image in which the underbody of the vehicle body is transparent as an underbody transparent image CP. As described above, as an example, the composite image is generated in the order of projecting the camera image onto the projection surface TS and then adding the vehicle image. However, the respective processes can be performed in parallel.

[0091] In addition, various methods can be employed as the method for generating the composite image. For example, the road surface image generated based on the camera image, the vehicle boundary line Lvc, the tire boundary line Lt, and the image of the three-dimensional model of the member constituting the vehicle can be arranged in different layers and processed individually. For example, the composite image can be generated in conjunction with a camera image layer that is a layer including an image of a projection surface TS on which a camera image is projected, a boundary line layer that is a layer on which a vehicle boundary line, a tire boundary line, and the like are arranged, and a 3D model layer that includes an image of a three-dimensional model. In this case, the camera image layer can be the lowest layer (background side), and the boundary line layer and the 3D model layer can be superimposed on the camera image layer. Of course, the layer structure can also be changed as appropriate. When each image element is processed individually by using a plurality of layers, the image of each layer is finally integrated to become the composite image.

[0092] (Vehicle boundary line Lvc and decorative image Ef)

[0093] The image generation ECU 1 of the present embodiment generates an image in which a decorative image Ef such as a gradient or a shadow is added to the vehicle boundary line Lvc. Here, the vehicle boundary line Lvc and the decorative image Ef will be described with reference to Figure 8 Figure 8 is a diagram showing an example of a driver's point of view image CPd generated by the composite image generation unit F7 of the present embodiment. Figure 9 is Figure 8 is an enlarged view of the vicinity of the left front wheel.

[0094] In addition to the vehicle boundary line Lvc and the tire boundary line Lt, Figure 8 and Figure 9 The example shown in FIG. 8 further includes a tire model image Ptm, a body model image Pbm, and the like. When a 3D tire model, a 3D body model, and the like corresponding to the front wheels are viewed from the driver's point of view VPd, the various model images are computer graphics (CG) or computer generated imagery (CGI). Of course, the tire model image Ptm and the body model image Pbm are arbitrary elements, and can be omitted by user settings or the like. For example, based on the settings of the designer or the user of the image generation ECU 1, the tire model image Ptm and the body model image Pbm can be hidden in the driver's point of view image CPd.

[0095] ​The vehicle boundary line Lvc is a line image drawn on the vehicle region R0. The vehicle boundary line Lvc is a line indicating the range on the road surface where the vehicle V exists. The thickness (in other words, the width Wx) of the vehicle boundary line Lvc is set to a thickness corresponding to, for example, 4 cm on the road surface. The length in the real space can be converted into the length in the virtual space based on the ratio of the size of the vehicle body in the virtual space including the projection plane TS to the size of the vehicle body in the real space. The thickness of the vehicle boundary line Lvc displayed on the display 3 as the floor transparent image CP can be adjusted according to the distance from the virtual viewpoint VP.

[0096] The thickness of the vehicle boundary line Lvc in the real space can be equal to 3 cm or 2 cm. Further, the vehicle boundary line Lvc can have a thickness equal to 6 cm or 8 cm in the real space. The visibility of the vehicle boundary line Lvc improves as the vehicle boundary line Lvc becomes thicker. However, when the vehicle boundary line Lvc becomes thicker, the portion of the camera image hidden by the line increases. That is, when displayed as a composite image, the area of the road surface that the user can visually recognize based on the composite image becomes smaller. Therefore, preferably, the thickness of the vehicle boundary line Lvc in the real space is set to a value corresponding to less than 5 cm.

[0097] The color of the vehicle boundary line Lvc can be appropriately changed and, for example, set to blue so that the vehicle boundary line Lvc can be easily distinguished from the actual color of the road surface. Dark colors, for example, black, are easily assimilated by asphalt, and light colors, for example, white, are easily assimilated by gravel roads and lane markings given on the road surface. In consideration of various road surface tones, the color of the vehicle boundary line Lvc is set to enhance the visibility of the vehicle boundary line Lvc to the driver.

[0098] The decorative image Ef is a decorative image element for enhancing the visibility of the vehicle boundary line Lvc. In other words, the decorative image Ef is an auxiliary image element for highlighting the vehicle boundary line Lvc. For example, the decorative image Ef is formed in a bar shape having a predetermined width. As shown in Figure 10 , the decorative image Ef is arranged so that one edge thereof is along the vehicle boundary line Lvc. Figure 10 is a schematic view for illustrating Figure 8 and Figure 9 the configuration of the vehicle boundary line Lvc and the decorative image Ef. As shown in Figure 11 , the decorative image Ef can be arranged to extend outward from the vehicle boundary line Lvc in a manner inclined by about 45 degrees with respect to the vehicle horizontal plane. The decorative image Ef can extend from the outer edge of the vehicle boundary line Lvc toward the vehicle height direction (that is, directly above).

[0099] An extension angle δ - an angle formed by the decorative image Ef with respect to a vehicle horizontal plane - can be appropriately set in a range of 0 degrees to 90 degrees. That is, the decorative image Ef is set to extend from the vehicle boundary line Lvc toward the outside or extend directly above the vehicle boundary line Lvc. Between edges of the decorative image Ef, an edge opposite to the connecting edge Efa connected to the vehicle boundary line Lvc is defined as an open edge Efb. When the extension angle δ is set to 1 degree or more, the open edge Efb floats from the road surface. The decorative image Ef can face the inside of the vehicle boundary line Lvc, and the extension angle δ can be set to 91 degrees or more, for example, 120 degrees.

[0100] A color of the decorative image Ef can be the same as a color of the vehicle boundary line Lvc. The decorative image Ef is set to have a lower transparency as it approaches the vehicle boundary line Lvc, and a higher transparency as it moves away from the vehicle boundary line Lvc. That is, the decorative image Ef is a band-shaped image having a gradation that becomes transparent as the distance from the vehicle boundary line Lvc increases. In Figure 10 In the gradation, a dot pattern is shown. The transparency decreases as the density of the dot pattern increases. The transparency increases as the density of the dot pattern decreases. For example, the transparency of a portion of the decorative image Ef connected to the vehicle boundary line Lvc is set to about 50%, and an end portion distal therefrom is set to 100%, and the like. The degree of change in the transparency according to the distance from the vehicle boundary line Lvc, which forms the gradation, can be in the form of a linear function or a quadratic function. The degree of change in the hue can be appropriately changed.

[0101] An extension length H - a length in a direction away from the vehicle boundary line Lvc - of the decorative image Ef is set to be the same as or longer than a width Wx of the vehicle boundary line Lvc. For example, the extension length H in the real space is set to be about 5 cm to 8 cm. Further, the extension length H can be set to a length obtained by multiplying the width Wx of the vehicle boundary line Lvc by a predetermined extension coefficient. The extension coefficient can be, for example, 1.2 or 1.5. Further, the extension length H can be set to be shorter than the width Wx of the vehicle boundary line Lvc. For example, the extension length H can be set to be 0.8 times the width Wx of the vehicle boundary line Lvc. The extension length H can also be understood as a width of the decorative image Ef itself. The decorative image Ef that provides the gradation and extends radially from the above-described vehicle boundary line Lvc is an image that extends upward or outward from the vehicle boundary line Lvc, such as a curtain or an aurora.

[0102] According to the aspect in which the decorative image Ef is added to the bottom transparent image CP of the vehicle boundary line Lvc as described above, the visibility of the vehicle boundary line Lvc can be enhanced. Further, since the decorative image Ef is set to be semi-transparent, the possibility that the road surface cannot be seen due to the decorative image Ef can be reduced. The vehicle boundary line Lvc and the decorative image Ef can be collectively interpreted as a vehicle outline image Pvc. The vehicle boundary line Lvc in the vehicle outline image Pvc can be referred to as a main body or a main line. Further, the decorative image Ef in the vehicle outline image Pvc can be referred to as a decorative portion.

[0103] (tire boundary line Lt)

[0104] The tire boundary line Lt is a line that indicates a range of the road surface in which the tire (in other words, the wheel) of the vehicle V exists. The tire boundary line Lt is set to be thinner than the vehicle boundary line Lvc. For example, the thickness of the tire boundary line Lt is set to be about half of the vehicle boundary line Lvc. Specifically, the thickness of the tire boundary line Lt is set to a thickness corresponding to, for example, 2 cm on the road surface. By setting the tire boundary line Lt to be thinner, the driver can easily see the road surface directly below the tire and the road surface in the vicinity of the tire.

[0105] Further, the color of the tire boundary line Lt is set to a color different from the color of the vehicle boundary line Lvc. For example, the color of the tire boundary line Lt can be set to yellow or orange. Of course, the color of the tire boundary line Lt is set to enhance the visibility of the driver in consideration of various road surface colors as in the case of the vehicle boundary line Lvc.

[0106] For example, the tire boundary line Lt can have a color (so-called complementary color) located on the opposite side of the color of the vehicle boundary line Lvc in the color wheel or a color adjacent thereto. By using the color of the tire boundary line Lt as the complementary color to the vehicle boundary line Lvc, the tire boundary line Lt formed to be narrow can be highlighted. The complementary color of blue is orange, and the colors adjacent to orange include yellow and red.

[0107] On the other hand, from another perspective, the color of the tire boundary line (Lt) can be similar to the color of the vehicle boundary line (Lvc). For example, when the vehicle boundary line (Lvc) is blue, the tire boundary line (Lt) can be, for example, light blue or dark blue. Since both the vehicle boundary line (Lvc) and the tire boundary line (Lt) are line images indicating information about the vehicle's current position, using similar colors can enhance a sense of unity. Furthermore, as a composite image, objects of attention, such as obstacles surrounding the vehicle, can be highlighted by surrounding them with boxes in colors such as red or orange. Based on this configuration, the presence of lines of various colors in the image raises concerns about causing visual annoyance to the user. Therefore, by using similar colors for the vehicle boundary line (Lvc) and the tire boundary line (Lt), the user can distinguish between lines indicating information about the vehicle and lines indicating information about surrounding objects.

[0108] The above example illustrates a scenario where the virtual viewpoint VP is set to the driver's viewpoint VPd; however, the vehicle boundary line Lvc and tire boundary line Lt are also displayed when the bird's-eye viewpoint VPb is set. In the bird's-eye view image, the decorative image Ef of the vehicle boundary line Lvc can be displayed as if attached to the road surface.

[0109] Reference Figure 12 The flowchart shown describes the process of displaying a bottom-transparent image, which is a series of processes performed by the image generation ECU 1 when displaying the bottom-transparent image CP. When predetermined bottom-display conditions are met, the process begins... Figure 12 The flowchart shown illustrates this. The bottom display condition can be a condition for displaying the bottom transparent image CP. For example, when the user presses operation button 5, the image generation ECU 1 determines that the bottom display condition is met. Additionally, when a predetermined user operation for displaying the bottom transparent image CP is performed via touch panel 4, the image generation ECU 1 determines that the bottom display condition is met. That is, the process can begin based on the fact that the operation receiving unit F2 has acquired a signal indicating that an operation for displaying the composite image has been performed.

[0110] The conditions constituting the bottom-view display can include gear shift position, vehicle speed, etc. For example, setting the gear shift position to a predetermined range and the vehicle speed being less than a predetermined low-speed threshold can be included in the bottom-view display conditions. For example, image generation conditions can include setting the gear shift position to a low range and setting the gear shift position to a rearward position. The low-speed threshold can be, for example, 10 km / h, 15 km / h, 20 km / h, etc. In addition, when the operation receiving unit F2 receives a viewpoint switching operation via the touch panel 4, etc., while the front camera image or the rear camera image is displayed on the display 3, it is determined that the bottom-view display conditions are met.

[0111] Further, the execution of the processing flow illustrated above can be repeated at a predetermined cycle (for example, a 1 / 30 second cycle) until a predetermined display termination condition is satisfied. The display termination condition can be, for example, a case where the shift position is set to a range other than a predetermined range, a case where the vehicle speed exceeds a predetermined threshold, a case where the operation button 5 is pressed again, and the like. In addition, when an operation for terminating the display of the underlay transparent image CP is performed via the touch panel 4, the image generation ECU 1 can determine that the display termination condition is satisfied. Figure 12

[0112] Here, assume, as an example, that the underlay transparent image display processing includes S1 to S8. Of course, the number of steps and the processing order constituting the underlay transparent image display processing can be appropriately changed.

[0113] First, in S1, the vehicle state acquisition unit F3 acquires information indicating the state of the subject vehicle, such as the shift position and the vehicle speed, and the processing proceeds to S2. Such processing can be referred to as a vehicle state acquisition step.

[0114] In S2, the image acquisition unit Fl acquires the four camera images obtained by the four cameras 2, and the processing proceeds to S3. Such processing can be referred to as an image acquisition step. Further, S2 includes S21 as a more detailed step. In S21, based on the information acquired in S1, it is determined whether the vehicle V has traveled a predetermined storage distance since the last time an image captured by the cameras 2 on the traveling direction side was stored. When the moving distance from the previous storage of the camera image is equal to or longer than the storage distance, the image captured by the cameras 2 on the traveling direction side is stored in the image storage Ml.

[0115] That is, S21 corresponds to a step of storing the image captured by the cameras 2 on the traveling direction side in the image storage Ml each time the vehicle V travels the storage distance. When the state in which the image generation condition is not satisfied changes to a state in which the image generation condition is satisfied, the image acquisition unit Fl stores at least the image captured by the cameras 2 on the traveling direction side as a first storage processing. Such processing can be referred to as an image storage step.

[0116] In S3, as described above, the camera image synthesis unit F71 maps the captured image of each camera 2 to a predetermined position on the projection surface TS. Further, the camera image synthesis unit F71 maps the image of the road surface region corresponding to the current vehicle position to the vehicle region RO by using the past captured image data stored in the image storage Ml. After this processing is completed, the processing proceeds to S4. When the image that can be projected onto the vehicle region RO has not yet been accumulated, for example, immediately after the start of this flow, an opaque image such as black can be arranged in the vehicle region RO as a bottom image. Such processing can be referred to as an image synthesis step. ​

[0117] In S4, the vehicle image adding unit F72 arranges the vehicle boundary line Lvc and the tire boundary line Lt at predetermined positions in the vehicle region RO, and the processing proceeds to S5. In S4, a 3D tire model, a 3D vehicle body model, or the like can be arranged at predetermined positions above the vehicle region RO. Further, in S4, a planned trajectory line, a scale line, a vehicle width line, or the like can be drawn as another image element. In S4, since information indicating the current state of the vehicle is added to the composite image, it can be referred to as a vehicle information adding step.

[0118] In S5, the composite image generating unit F7 sets a virtual viewpoint VP for generating the composite image under the control of the display control unit F6. For example, when the composite image generating unit F7 generates the floor transparent image CP for the first time after the running electric power is turned on, the default setting of the virtual viewpoint VP can be a position and a line-of-sight direction which are preset by a designer or a user. The default setting can be, for example, a driver's viewpoint VPd whose line-of-sight direction is directed obliquely forward and obliquely downward. In addition, the setting of the virtual viewpoint VP set in S5 can be the position and the direction of the virtual viewpoint VP at the time when the floor transparent image CP was last displayed. In this case, as a preparation process, the display control unit F6 stores the setting data of the virtual viewpoint VP at the time when the floor transparent image CP was last displayed in the RAM 12 or the storage device 13.

[0119] Further, the position and the line-of-sight direction of the virtual viewpoint VP can be determined in accordance with the running direction of the subject vehicle. For example, when the running direction of the subject vehicle is a forward direction, the driver's viewpoint VPd whose line-of-sight direction is directed obliquely forward and obliquely downward as described above is adopted as the virtual viewpoint VP. On the other hand, when the running direction of the subject vehicle is a backward direction, for example, the driver's viewpoint VPd whose line-of-sight direction is directed obliquely backward and obliquely downward can be adopted as the virtual viewpoint VP. In addition, the line-of-sight direction of the virtual viewpoint VP can be adjusted in accordance with the direction of the steering angle. Further, the composite image generating unit F7 can cause the display control unit F6 to acquire the position and the direction of the virtual viewpoint VP specified by the user via the touch panel 4 or the like, and cause the display control unit F6 to set the virtual viewpoint VP of the specified position and direction.

[0120] Here, as an example, it is assumed that the driver's viewpoint VPd whose line-of-sight direction is directed obliquely forward and obliquely downward is applied as the virtual viewpoint VP. Such a process can be referred to as a visual viewpoint setting step. When S5 is completed, the processing proceeds to S6.

[0121] In S6, the synthesized image generation unit F7 generates a bottom transparent image CP viewed from the virtual viewpoint VP set in S6. Here, as an example, the synthesized image generation unit F7 generates a driver viewpoint image CPd showing the ground surface in the vicinity of the front wheels as the bottom transparent image CP by transmitting the bottom of the vehicle and the vehicle body. Data of the bottom transparent image CP generated by the synthesized image generation unit F7 is output to the display image generation unit F8. Such processing can be referred to as a synthesized image generation step. When the processing in S6 is completed, the processing proceeds to S7.

[0122] In S7, the display image generation unit F8 generates a display image DP including the bottom transparent image CP generated by the synthesized image generation unit F7. For example, as shown in Figure 13 the display image generation unit F8 generates an image including the driver viewpoint image CPd, the right side camera image SR, the left side camera image SL, and the viewpoint switching switch image SW as the display image DP. Specifically, the driver viewpoint image CPd is arranged above a center region of the display image DP, and the viewpoint switching switch image SW is arranged below the center region. The right side camera image SR is arranged to the right of the driver viewpoint image CPd, and the left side camera image SL is arranged to the left of the driver viewpoint image CPd. According to such a layout of the display image DP, the driver can be visually notified of the left and right situations while maintaining good visibility of the driver viewpoint image CPd for the user. That is, by viewing the display image DP, the driver can recognize the ground surface state in the vicinity of the front wheels and the state of the side of the vehicle.

[0123] The viewpoint switching switch image SW included in the display image DP is an image that functions as a switch for switching the display content to the display image DP when touched by the user. Whether or not the user touches the viewpoint switching switch image SW can be determined based on the touch position signal output from the touch panel 4. When a touch operation of the viewpoint switching switch image SW by the user is detected, the display image generation unit F8 changes the image to be displayed in the image center region from the driver viewpoint image CPd to the front camera image, for example, in accordance with an instruction from the display control unit F6. The viewpoint switching switch image SW can switch the virtual viewpoint VP from the driver viewpoint VPd to the bird's-eye perspective VPb. In this case, for example, the display image generation unit F8 generates an image including a bird's-eye perspective image of the bottom transmission as the display image DP. Data of the display image DP generated by the display image generation unit F8 is output to the image output unit F9. S7 can be referred to as a display image generation step.

[0124] In S8, the image output unit F9 converts digital data of the display image DP generated by the display image generation unit F8 into a signal in a predetermined signal format, and outputs the digital data to the display 3. As a result, the display image DP including the driver's point-of-view image CPd is displayed on the display 3. S8 can be referred to as a display image output step.

[0125] The above control is an example, and the display image generation unit F8 can select a camera image in accordance with the traveling direction of the subject vehicle, and generate the display image DP by using the camera image under the control of the display control unit F6. For example, when the traveling direction is the rearward direction, a rear camera image can be arranged in the center region of the display image DP. Further, the display control unit F6 can change the combination and layout of images to be displayed on the display image DP, the display range of the camera image, and the like, based on an operation signal received by the operation reception unit F2.

[0126] (EFFECTS)

[0127] According to the above configuration, when a component of the subject vehicle V (for example, an instrument panel and a vehicle body bottom) is transmitted, a floor transparent image CP showing the periphery of the vehicle is generated. The floor transparent image CP corresponds to an image in which the vehicle boundary line Lvc and the tire boundary line Lt are superimposed on an image of an object existing right below and around the vehicle. By confirming such a floor transparent image CP, the user can confirm the state of the surrounding environment of the subject vehicle from the viewpoint of the inside of the vehicle, and can intuitively grasp the state of the surrounding environment of the subject vehicle.

[0128] Further, the user can easily recognize the distance from the vehicle body to a three-dimensional object existing in the vicinity of the vehicle, such as a rock, a curb, or a guard, based on the vehicle boundary line Lvc included in the floor transparent image CP. Therefore, it is possible to reduce the risk of the vehicle body inadvertently coming into contact with the three-dimensional object. Further, according to the configuration in which the tire boundary line Lt is included in the floor transparent image CP, the user can easily recognize the positions of the vehicle body and the tires with respect to lane marks, curbs, rocks, and the like located below the vehicle body. As a result, for example, during off-road driving, detailed driving operations, such as placing the tires on a target rock, can be performed. In addition, even during driving on a road, it becomes easier to perform detailed driving operations, such as moving the vehicle body toward a road edge so that the tires do not hit a curb.

[0129] In addition, the image generation ECU 1 of the present embodiment displays the vehicle boundary line Lvc in which the decorative image Ef such as a gradient is added to the vehicle boundary line Lvc. According to this configuration, the visibility of the vehicle boundary line Lvc can be enhanced. Further, when the decorative image Ef such as a gradient is added to the vehicle boundary line Lvc, the other display lines are displayed without the decoration such as a gradient (in other words, the effect). According to such a configuration, it is possible to suppress the possibility that the vehicle boundary line Lvc is mistaken for another display line based on the presence or absence of the decoration. The other display lines here can include the tire boundary line Lt, the planned trajectory line, the scale line, the vehicle width line, and the like.

[0130] The line can be thickened without adding the decoration to make the vehicle boundary line Lvc stand out. However, if the vehicle boundary line Lvc becomes thicker, the portion of the road surface hidden by the vehicle boundary line Lvc increases. Further, if the vehicle boundary line Lvc becomes thicker, the range in which the vehicle actually exists can not be clear. If the boundary line is too thick, it is difficult for the user to determine where the end of the actual vehicle V is on the boundary line, that is, it is difficult for the user to determine whether the end of the vehicle V is located at the outer edge, the center, or the inner edge of the boundary line. On the other hand, according to the above-described configuration, the thickness of the vehicle boundary line Lvc is kept thin, and it can become apparent. That is, according to the above-described configuration, it is possible to make the vehicle boundary line Lvc stand out without excessively hiding the camera image. In addition, it is possible to reduce the possibility that the range in which the vehicle actually exists becomes unclear.

[0131] The effect of the present embodiment has been described by employing the case in which the driver's viewpoint VPd in which the line-of-sight direction is directed forward is applied as the virtual viewpoint VP of the composite image. The same effect can be obtained when the line-of-sight direction is directed to another direction such as backward, laterally, or diagonally laterally. Further, the same effect can be obtained when the virtual viewpoint VP is arranged at an arbitrary position in the vehicle interior other than the driver's viewpoint VPd. In addition, the same effect can be expected when the virtual viewpoint VP is provided on the outer surface of the vehicle V or in a region near the vehicle outside the vehicle interior. The region near the vehicle here refers to a region within, for example, 0.2 m from the outer surface of the vehicle. The outer surface can include a rear surface portion, a front end portion, and a roof in addition to the left and right side surface portions. A door panel, a fender, a pillar, and the like can be included in the side surface portions.

[0132] While the embodiments of the present disclosure have been described above, the present disclosure is not limited to the embodiments described above, and various modifications to be described below are included in the technical scope of the present disclosure, and can be implemented by various modifications without departing from the spirit described below. For example, the various modifications to be described below can be appropriately combined to be implemented without causing technical inconsistency. Note that members having the same function as that described in the above-described embodiments are denoted by the same reference numerals, and the description of the same members will be omitted. When only a part of the configuration is described, the configuration described in the above-described embodiments can be applied to the other parts.

[0133] (decorative image Ef)

[0134] The decorative image Ef serves as a gradient image, however the decorative image Ef can be an image having a vertical stripe pattern as shown in Figure 14 . For example, the vertical stripe pattern is formed by alternately arranging colored translucent portions (hereinafter referred to as colored portions E1) and colorless E2 portions which are colorless and translucent portions along the vehicle boundary line Lvc. The vertical stripe pattern can include a mottled pattern. The colored portions E1 correspond to the first portions, and the colorless portions E2 correspond to the second portions. The second portions can be relatively transparent compared to the first portions, and can not be colorless and transparent.

[0135] The width W1 of the colored portions E1 can be set to a value of about 1 to 4 times the width Wx of the vehicle boundary line Lvc. For example, the width W1 of the colored portions E1 is set to three times the width Wx of the vehicle boundary line Lvc. The width W2 of the colorless portions E2 is set to a size similar to the width W1 of the colored portions E1. The width W2 of the colorless portions E2 can be greater than the width W1 of the colored portions E1. The width W2 of the colorless portions E2 can be set to twice the width W1 of the colored portions E1. The width W2 of the colorless portions E2 defines the pitch between the colored portions E1. The area in which the camera image is hidden by the decorative image Ef becomes smaller as the width W2 of the colorless portions E2 increases, so that the visibility of the road surface to the user can be improved.

[0136] Further, as shown in Figure 15 , the colored portions E1 constituting the stripe pattern can have a gradient in which the transparency is set to be lower as closer to the vehicle boundary line Lvc, and the transparency is set to be higher as farther from the vehicle boundary line Lvc. For example, the transparency of the portion of the colored portions E1 connected to the vehicle boundary line Lvc is set to about 50%, and the end portion on the distal side is set to 100% or the like. Figure 16 and Figure 17 application Figure 15The pattern shown schematically is a transparent image CP placed at the bottom of the driver's viewpoint VPd, in the case of a decorative image Ef.

[0137] According to the configuration in which the decorative image Ef has a striped pattern—where the striped pattern has a gradient that becomes transparent with respect to the distance from the vehicle boundary line Lvc—the portion of the camera image hidden by the decorative image Ef is further reduced compared to the above-described embodiment.

[0138] Additionally, the decorative image Ef can have a shape in which the extension length H varies depending on the position. For example, as Figure 18 , Figure 19 and Figure 20 As shown, the opening edge portion Efb of the decorative image Ef can be set to a wavy shape. The wavy shape can include a so-called fan shape in which semicircles are connected. Furthermore, the opening edge portion Efb of the decorative image Ef can have a triangular wavy shape. When the decorative image Ef has a wavy shape, the maximum value Hmx of the real-world extension length H can be, for example, equal to 5 cm. Furthermore, the minimum value Hmn of the real-world extension length H can be 0 or equal to 2 cm.

[0139] In addition, such as Figure 21 As shown, even when the decorative image Ef has a wavy shape, the colored portion can have lower transparency as it gets closer to the vehicle boundary line Lvc, and higher transparency as it gets further away from the vehicle boundary line Lvc. Additionally, the aforementioned striped pattern can be applied between the connecting edge portion Efa and the opening edge portion Efb.

[0140] In addition, such as Figure 21 As shown, when the decorative image Ef has a wavy shape, the height of each location can change continuously over time. Based on this display control, the decorative image Ef moves over time, and the vehicle boundary line Lvc can become more prominent. Additionally, the decorative image Ef can be set to have an effect where the colored portions undulate like flames over time.

[0141] In addition, such as Figure 14 , Figure 15 , Figure 18 and Figure 21 As shown, when the decorative image Ef has a shape with different heights depending on its position, the decorative image Ef can vibrate as the vehicle moves. Specifically, in the decorative image Ef, the position of the root portion—the portion near the vehicle boundary line Lvc—can remain unchanged, but the vertex portions can translate in the opposite direction to the direction of travel, and the images can be continuously connected between them. Figure 22 This shows that when the decorative image Ef has such Figure 15The vertical stripe pattern shown is an example in which the inclination angle of the colored portion E1 changes according to the moving state of the vehicle.

[0142] As Figure 22 shown, the inclination of the colored portion E1 changes according to the traveling direction of the vehicle V and whether the vehicle V is stopped. According to this configuration, the user can easily intuitively recognize the moving state of the vehicle based on the display mode of the decorative image Ef. The inclination angle of the pattern of the decorative image Ef can be adjusted according to the traveling speed. The higher the vehicle travels, the larger the inclination angle can be set. According to such a configuration, the user can recognize the traveling speed of the vehicle according to the degree of inclination of the pattern of the decorative image Ef. Further, the inclination angle of the colored portion E1 can be adjusted according to the steering angle.

[0143] In addition, the synthesized image generation unit F7 can change the color of the vehicle boundary line Lvc according to the detected distance to the obstacle under the control of the display control unit F6. For example, when the distance to the obstacle is equal to or greater than a predetermined first distance, the color of the vehicle boundary line Lvc is set to a default color, such as blue or green. On the other hand, if the distance to the obstacle is less than the first distance and equal to or greater than a predetermined second distance, the color can be changed to yellow. Further, when the distance to the obstacle is less than the second distance, the color of the vehicle boundary line Lvc can be changed to red. The first distance can be, for example, 1.0 m or 0.8 m. The second distance can be a value shorter than the first distance, and can be, for example, 0.3 m or 0.4 m. The first distance and the second distance can be dynamically adjusted according to the traveling speed. The color of the decorative image Ef can be changed in connection with the color of the vehicle boundary line Lvc. For example, when the vehicle boundary line Lvc becomes yellow, the decorative image Ef also becomes a yellow-based semi-transparent image.

[0144] Further, when the distance to the obstacle is less than the first distance, the display control unit F6 can cause the vehicle boundary line Lvc to blink at a time interval corresponding to the distance to the obstacle. In this case, the display control unit F6 can also cause the decorative image Ef to blink together with (in other words, in synchronization with) the blinking of the vehicle boundary line Lvc.

[0145] The display control unit F6 can change the display of the portion of the decorative image Ef corresponding to the direction in which the obstacle exists locally. Examples of elements that constitute the display mode include size, shape, color, and transparency. For example, the display control unit F6 can cause the portion of the decorative image Ef close to the obstacle to be longer or to blink more than other portions. Further, the decorative image Ef can vibrate in the direction in which the obstacle exists. This configuration makes it easy for the user to recognize in which direction the obstacle exists for the vehicle.

[0146] Additionally, the display of the decorative image Ef can be switched based on the distance to the obstacle. For example, when the distance to the obstacle is a second distance or greater, the display control unit F6 displays the decorative image Ef. Conversely, when the distance to the obstacle is less than the second distance, the decorative image Ef can be hidden. By hiding the decorative image Ef, the interval and positional relationship between the vehicle boundary line Lvc and the obstacle can be easily identified when the distance to the obstacle is less than the second distance.

[0147] Furthermore, in the above embodiment, the tire boundary line Lt is not decorated with any embellishments such as gradients, but the present invention is not limited thereto. Decorative images can also be added to the tire boundary line Lt. The color of the decorative image added to the tire boundary line Lt can be the same as the color of the tire boundary line Lt. When a decorative image is added to the tire boundary line Lt, preferably, the colors of the tire boundary line Lt and the vehicle boundary line Lvc can be set to different colors so that the tire boundary line Lt can be easily distinguished from the vehicle boundary line Lvc.

[0148] (Variations on virtual viewpoints)

[0149] In the preceding text, the bird's-eye view VPb and the driver's viewpoint VPd have been shown as virtual viewpoints VP, but the combinations of positions and orientations that can be set as virtual viewpoints VP are not limited to the examples above. Virtual viewpoints VP can be positioned in various locations outside the vehicle's interior. For example, as... Figure 23 As shown, the image generation ECU 1 can set either a rearward bird's-eye view VPb1 or a forward bird's-eye view VPb2 as a virtual viewpoint VP. The rearward bird's-eye view VPb1 corresponds to a virtual viewpoint VP that looks down on the vehicle V and its rear from a point located above the roof of the vehicle V in front of the vehicle. Similarly, the forward bird's-eye view VPb2 corresponds to a virtual viewpoint VP that looks down on the vehicle V and its front from a point located above the roof of the vehicle V in rear of the vehicle.

[0150] Furthermore, the virtual viewpoint VP can be positioned at various locations within the vehicle. For example, the viewpoint VP can be located near the side mirrors, in the center of the headliner, etc. Additionally, the display control unit F6 can be set to position its viewpoint at a predetermined distance behind the eye, with its gaze directed diagonally backward and downward towards the interior rear viewpoint VPr. Based on this interior rear viewpoint VPr, the area near the rear wheels can be displayed more prominently as a bottom-mounted transparent image CP displayed when the vehicle is moving backward. As a result, this configuration makes it easier for the driver to identify the area near the rear wheels and rear bumper, both when the vehicle is moving backward and forward.

[0151] The display control unit F6 can change whether or not to display the decoration image Ef depending on the distance between the virtual viewpoint VP and the road surface or whether or not the virtual viewpoint VP exists inside the vehicle. For example, when the virtual viewpoint VP is set inside the vehicle, such as the driver's viewpoint Vpd, the decoration image Ef is added to the vehicle boundary line Lvc. When the virtual viewpoint VP is set at a position higher than the roof, such as the bird's-eye viewpoint VPb, the decoration image Ef can be hidden. In the bird's-eye viewpoint VPb, the viewpoint is away from the road surface, so even if the decoration image Ef is displayed, it is relatively small and less noticeable. In the bird's-eye viewpoint image, even if the fine image element such as the decoration image Ef is omitted, the impact on the user's convenience is small. Furthermore, by omitting the display of the decoration image Ef, it is possible to reduce the processing load of the processing unit 11. In this way, according to the configuration in which whether or not to display the decoration image Ef is switched depending on the position of the virtual viewpoint VP, in other words, depending on the display mode of the composite image, the visibility of the vehicle boundary line Lvc is improved, and it is possible to reduce the processing load of the processing unit 11.

[0152] (obstacle sensor)

[0153] In the above, a configuration in which the sonar 8 is used as a sensor for detecting an object existing around the vehicle (so-called obstacle sensor) has been shown, but the obstacle sensor can be a millimeter wave radar. Furthermore, the obstacle sensor can be a light detection and ranging / laser imaging detection and ranging (LiDAR). The image generation ECU 1 can be used by being connected to various obstacle sensors.

[0154] (display)

[0155] In the above-described embodiment, the display 3 is provided by the on-vehicle display, but the display destination of the image (for example, the underlay transparent image CP) generated by the image generation ECU 1 is not limited to the on-vehicle display. The display 3 can be a display provided in a mobile terminal such as a smartphone for remotely controlling the vehicle V. Furthermore, the display can be provided in a center for remotely controlling the vehicle V.

[0156] The apparatus, system, and method thereof that have been described in the present disclosure can also be implemented by a dedicated computer that constitutes a processor programmed to execute one or more functions embodied by a computer program. Furthermore, the apparatus and method thereof that have been described in the present disclosure can also be implemented by a dedicated hardware logic circuit. Furthermore, the apparatus and method thereof that have been described in the present disclosure can also be implemented by one or more dedicated computers that constitute a combination of a processor for executing a computer program and one or more hardware logic circuits. Furthermore, the computer program can store instructions that are computer-readable and non-transitory tangible recording media to be executed by a computer. The method or function provided by the image generation ECU 1 and the like can be provided by software stored in a tangible memory device and a computer executing the software, software only, hardware only, or a combination of software and hardware. Some or all of the functions of the image generation ECU 1 can be configured as hardware. The configuration of implementing a certain function as hardware includes a configuration of implementing the function by use of one or more ICs and the like. The image generation ECU 1 can be implemented by use of an MPU, a GPU, or a dataflow processor (DFP) instead of a CPU. The image generation ECU 1 can be implemented by a combination of a CPU, an MPU, and a GPU. The image generation ECU 1 can be implemented as a system on chip (SoC). Furthermore, processing can be implemented by use of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and the like, for example. It should be noted that the various programs described above can be stored in a non-transitory tangible storage medium. Various storage media such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, and a secure digital (SD) card can be employed as the program storage medium.

Claims

1. A peripheral image generation device for a vehicle, the peripheral image generation device being connected to an obstacle sensor (8) configured to detect a three-dimensional object present in the vicinity of the vehicle, the peripheral image generation device comprising: an image acquisition unit (Fl) configured to acquire a plurality of camera images obtained from a plurality of cameras each of which captures a periphery of the vehicle; an image storage unit (Ml) configured to store a travel direction camera image that is a part or all of an image included in the plurality of camera images and captured at least in a travel direction of the vehicle; and a synthetic image generation unit (F7) configured to generate a bottom transparent image that is a synthetic image that transmits a bottom portion of the vehicle based on the plurality of camera images and the travel direction camera image, wherein the synthetic image generation unit generates an image as the bottom transparent image in which (i) a vehicle boundary line (Lvc) indicating a boundary of a vehicle body range is superimposed on an image showing a part or all of a ground surface located below the vehicle and (ii) a decoration having a predetermined pattern is added to the vehicle boundary line, the peripheral image generation device further comprises: an obstacle information acquisition unit (F4) configured to acquire a signal indicating a detection result of the obstacle sensor; and a display control unit (F6) configured to control display of a decoration provided as a decoration image (Ef) set on the vehicle boundary line and display of the vehicle boundary line, wherein the display control unit changes the display of the vehicle boundary line in response to the obstacle information acquisition unit acquiring a signal from the obstacle sensor indicating that an obstacle is present within a predetermined distance from the vehicle, and changes the display of the decoration image in association with the change in the display of the vehicle boundary line.

2. The peripheral image generation device according to claim 1, wherein the decoration image (Ef) has a gradation in which transparency gradually increases as a distance from the vehicle boundary line increases.

3. The peripheral image generation device according to claim 1 or 2, wherein the decoration image (Ef) has a first portion (El) whose transparency is relatively low and a second portion (E2) whose transparency is relatively higher than that of the first portion, and the decoration image has a striped pattern through an alternating arrangement of the first portion and the second portion.

4. The peripheral image generation device according to claim 1 or 2, wherein the decoration image (Ef) has a stripe shape with a predetermined width and extends from the vehicle boundary line toward the outside of the vehicle or extends directly above the vehicle.

5. The peripheral image generation device according to claim 1 or 2, wherein the decoration image (Ef) has an edge farther from the vehicle boundary line laterally and has a wave shape, and the decoration image has a striped pattern through an alternating arrangement of the first portion and the second portion. The synthetic image generation unit changes a height of each position in the decoration image relative to the vehicle boundary line over time.

6. The peripheral image generation apparatus according to claim 1 or 2, wherein The synthetic image generation unit generates the bottom transparent image based on (i) a virtual projection surface on which data of the plurality of camera images is projected and (ii) a virtual viewpoint whose position and direction are changeable based on an instruction operation from a user, The synthetic image generation unit is configured to set the virtual viewpoint to a vehicle interior viewpoint (VPd) that is set inside the vehicle and a bird's-eye viewpoint (VPb) that is set at a position outside the vehicle interior and higher than a roof, The peripheral image generation apparatus further includes: a display control unit (F6) configured to control display of a decoration that is provided as a decoration image (Ef) set on the vehicle boundary line, wherein The display control unit is configured to: display the decoration image when the virtual viewpoint is set to the vehicle interior viewpoint (VPd), not display the decoration image when the virtual viewpoint is set to the bird's-eye viewpoint (VPb).

7. The peripheral image generation apparatus according to claim 1 or 2, further comprising: a display control unit (F6) configured to control display of a decoration that is provided as a decoration image (Ef) and display of the vehicle boundary line, wherein The display control unit changes display of the decoration image in accordance with a moving state of the vehicle.

8. A peripheral image generation apparatus for a vehicle, comprising: an image acquisition unit (Fl) configured to acquire a plurality of camera images obtained from a plurality of cameras each of which captures a periphery of the vehicle; an image storage unit (Ml) configured to store a travel direction camera image that is a part or all of an image included in the plurality of camera images and captured at least in a travel direction of the vehicle; and a synthetic image generation unit (F7) configured to generate a bottom transparent image that is a synthetic image that transmits a bottom of the vehicle based on the plurality of camera images and the travel direction camera image, wherein The synthetic image generation unit generates an image as the bottom transparent image in which (i) a vehicle boundary line (Lvc) indicating a boundary of a body range is superimposed on an image showing a part or all of a ground surface located below the vehicle and (ii) a decoration having a predetermined pattern is added to the vehicle boundary line, The synthetic image generation unit generates the bottom transparent image based on (i) a virtual projection surface on which data of the plurality of camera images is projected and (ii) a virtual viewpoint whose position and direction are changeable based on an instruction operation from a user, The peripheral image generation apparatus further includes: a display control unit (F6) configured to control display of a decoration that is provided as a decoration image (Ef) set on the vehicle boundary line, wherein The synthetic image generation unit is configured to set the virtual viewpoint to a vehicle interior viewpoint (VPd) set inside the vehicle and an overhead viewpoint (VPb) set outside the vehicle and at a position higher than a roof of the vehicle, The display control unit is configured to: display the decorative image when the virtual viewpoint is set to the vehicle interior viewpoint (VPd), not display the decorative image when the virtual viewpoint is set to the overhead viewpoint (VPb).

9. A display control method of controlling display of an image for supporting a driving operation of a vehicle, the display control method comprising: acquiring (S2) a plurality of camera images obtained from a plurality of cameras each of which captures a periphery of the vehicle; storing (S21) a travel direction camera image included in the plurality of camera images and captured in a travel direction of the vehicle; and generating (S3, S4, S6) a floor transparent image that is a synthetic image that transmits a floor of the vehicle, based on the plurality of camera images and the travel direction camera image, wherein (i) a vehicle boundary line (Lvc) indicating a boundary of a vehicle body range is superimposed on an image showing a part or all of the floor located below the vehicle, and (ii) a decoration having a predetermined pattern is added to the image of the vehicle boundary line, acquiring a signal indicating a detection result of an obstacle sensor (8) configured to detect a three-dimensional object present in a vicinity of the vehicle from the obstacle sensor; and in response to the signal acquired from the obstacle sensor indicating that an obstacle is present within a predetermined distance from the vehicle, changing display of the vehicle boundary line, and changing display of the decoration provided as a decorative image (Ef) set on the vehicle boundary line in association with the change in display of the vehicle boundary line.

10. A display control method of controlling display of an image for supporting a driving operation of a vehicle, the display control method comprising: acquiring (S2) a plurality of camera images obtained from a plurality of cameras each of which captures a periphery of the vehicle; storing (S21) a travel direction camera image included in the plurality of camera images and captured in a travel direction of the vehicle; and generating (S3, S4, S6) a floor transparent image that is a synthetic image that transmits a floor of the vehicle, based on the plurality of camera images and the travel direction camera image, and based on (i) a virtual projection surface on which data of the plurality of camera images is projected and (ii) a virtual viewpoint whose position and direction are changeable based on an instruction operation from a user, wherein The bottom transparent image is one in which (i) a vehicle boundary line (Lvc) indicating a boundary of a range of a vehicle body is superimposed on an image showing a part or all of a ground surface located below the vehicle, and (ii) a decoration having a predetermined pattern is added to the image of the vehicle boundary line, and The virtual viewpoints that can be set include a vehicle interior viewpoint (VPd) set inside the vehicle and a bird's-eye viewpoint (VPb) set at a position outside the vehicle and higher than a roof, The display control method further includes: When the virtual viewpoint is set to the vehicle interior viewpoint (VPd), the decoration is displayed, the decoration being provided as a decoration image (Ef) set on the vehicle boundary line, When the virtual viewpoint is set to the bird's-eye viewpoint (VPb), the decoration image is not displayed.

Images