Image generating device and image generating method

The image generation device adjusts projection plane shapes based on road conditions to provide a realistic and comfortable off-road driving experience by avoiding distortion of surrounding objects.

JP7593101B2Active Publication Date: 2024-12-03DENSO CORP +1
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2020216417
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-12-25
Publication Date
2024-12-03
Estimated Expiration
2040-12-25

AI Technical Summary

Technical Problem

Existing image generation devices for vehicles assume a flat road surface, leading to distorted and uncomfortable viewing experiences when driving off-road due to flattening of surrounding objects like rocks.

Method used

An image generation device that adjusts the projection plane shape based on whether the vehicle is on-road or off-road, using different curved surfaces for off-road environments to maintain a realistic three-dimensional representation of surroundings.

Benefits of technology

Reduces user discomfort by accurately representing off-road terrain, enhancing safety and driving experience by minimizing distortion of surrounding objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593101000001
    Figure 0007593101000001
  • Figure 0007593101000002
    Figure 0007593101000002
  • Figure 0007593101000003
    Figure 0007593101000003
Patent Text Reader

Abstract

To provide an image generation device and an image generation method, which can generate a synthesis image which hardly causes a user to have a sense of incongruity as a synthesis image to be displayed at an off-road running time.SOLUTION: An image generation ECU includes an image synthesis unit and a running environment determination unit. The running environment determination unit is configured to determine whether or not the vehicle running environment corresponds to be off-road based on signals from an on-vehicle sensor etc. The image synthesis unit is configured to project the data of a plurality of camera images onto a virtual projection plane which corresponds to the periphery of the vehicle, so as to generate a synthesis image which indicates the vehicle periphery seen from a virtual view point, using data on the projection plane. When the running environment is determined to be on-road, the image synthesis unit generates a synthesis image using a normal projection plane in which the vehicle periphery is set to be substantially horizontal. On the other hand, when the running environment is determined to be off-road, the image synthesis unit generates a synthesis image using an off-road projection plane in which a planar region is not formed in the vehicle periphery.SELECTED DRAWING: Figure 14
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating a composite image showing the surroundings of a vehicle using images captured by multiple vehicle-mounted cameras. [Background technology]

[0002] Patent Documents 1 and 2 disclose an image display system that generates a composite image showing the periphery of the vehicle seen from a virtual viewpoint by synthesizing images captured by cameras installed at multiple locations on the vehicle, and displays the composite image. Examples of the composite image include an overhead image showing the periphery of the vehicle as seen from above the vehicle, and a driver's viewpoint image showing the periphery of the vehicle seen from the driver's viewpoint through an instrument panel or the like.

[0003] A composite image showing the surroundings of the vehicle as seen from a virtual viewpoint as described above is generated by projecting data from multiple captured images onto a virtual projection surface corresponding to the surroundings of the vehicle and using the data on the projection surface.

[0004] As the shape of the projection surface, a horizontal plane, a bowl shape, etc. have been proposed. The bowl shape here refers to a shape having a curved surface that is approximately horizontal near the vehicle and that inclines more as it moves away from the vehicle position. A bowl-shaped curved surface corresponds to a paraboloid of revolution formed by rotating a parabola. The bowl shape also includes a shape that combines a downwardly convex curved surface area around a flat plane area.

[0005] Patent Document 1 also discloses a configuration in which the projection surface is switched from a normal projection surface having a bowl shape in which the area near the vehicle is formed flat to a predetermined special projection surface when an obstacle is detected or based on a user's instruction to enlarge the display. The special projection surface in Patent Document 1 is a projection surface having a shape in which an upwardly convex curved surface area is added to the periphery of a rectangular plane corresponding to the vehicle area.

[0006] By using the image display system as described above, a user (typically a driver) can check the situation around the vehicle in almost real time. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 5911775 [Patent Document 2] Patent No. 6257978 Summary of the Invention [Problem to be solved by the invention]

[0008] The image generating devices considered so far are configured on the premise that the vehicle is located on a road paved with asphalt, concrete, brick, cobblestones, etc. (i.e., on-road). In other words, the premise is that the ground around the vehicle is flat. Therefore, the projection surface corresponding to the area near the vehicle is formed to be substantially flat. The area near the vehicle here refers to a range within 2m to 3m from the vehicle, for example.

[0009] However, the developers of the present disclosure have studied a configuration for displaying a synthetic image such as a driver's viewpoint image even during off-road driving, and have discovered the following: In a synthetic image using a projection plane in which the surroundings of the vehicle are set to be substantially flat, images of rocks and the like present around the vehicle are crushed to a flat surface, which may result in an image that feels unnatural to the user.

[0010] The present disclosure has been made based on the above circumstances, and its purpose is to provide an image generation device and an image generation method capable of generating a composite image that is displayed during off-road driving and that does not cause a user to feel uncomfortable. [Means for solving the problem]

[0011] An example of an image generating device for achieving this purpose is an image generating device used in a vehicle, and includes an image acquisition unit (F1) that acquires multiple camera images obtained by multiple cameras that capture the surroundings of the vehicle, an image synthesis unit (F7) that projects data of the multiple camera images onto a virtual projection plane corresponding to the surroundings of the vehicle and generates a synthetic image showing the surroundings of the vehicle as seen from a virtual viewpoint using the data on the projection plane, and a driving environment determination unit (F62) that determines whether the driving environment of the vehicle is off-road or on-road based on a signal from another device mounted on the vehicle, and the image synthesis unit is configured to change the shape of the projection plane used to generate the synthetic image depending on whether the driving environment has been determined to be off-road by the driving environment determination unit, and the image synthesis unit determines whether the driving environment is off-road or on-road based on a signal from another device mounted on the vehicle. Therefore, when the driving environment is determined to be on-road, a synthetic image is generated using an on-road projection surface having a predetermined shape as the projection surface, and when the driving environment determination unit determines that the driving environment is off-road, a synthetic image is generated using an off-road projection surface having a shape different from that of the on-road projection surface as the projection surface, the on-road projection surface has a flat on-road planar area (R1) arranged around a vehicle area (R0) which is an area where the vehicle is located, and a curved on-road curved area (R2) arranged around the on-road planar area, the off-road projection surface has a curved off-road curved area (R2a) arranged outside the vehicle area, and the off-road curved area (R2a) has a shape different from that of the on-road curved area (R2). The on-road curved surface area is a parabola expressed by Z=a1×x^2 when the distance from the curved surface start position, which is the position where the on-road curved surface area starts outside the vehicle area, to the outside of the vehicle area is x and the height is Z, and the off-road curved surface area is a parabola expressed by Z=a2×x^2 when the distance from the curved surface start position, which is the position where the off-road curved surface area starts outside the vehicle area, to the outside of the vehicle area is x and the height is Z, and the coefficient a2 is set to a value larger than the coefficient a1. .

[0012] According to the above configuration, when the driving environment is determined to be off-road, a composite image is generated using a projection surface having a different shape from that when the vehicle is determined to be on-road. In other words, a projection surface having a shape that assumes that the driving environment of the vehicle is off-road is applied. Therefore, it is possible to generate a composite image that is less likely to give a sense of incongruity to the user as a composite image to be displayed during off-road driving.

[0013] Further, an image generation method for achieving the above object is an image generation method for generating an image for assisting driving operation of a vehicle, the image generation method comprising the steps of: acquiring a plurality of camera images obtained by a plurality of cameras photographing the periphery of the vehicle (S1); projecting data of the plurality of camera images onto a virtual projection plane corresponding to the periphery of the vehicle, and generating a composite image showing the periphery of the vehicle as seen from a virtual viewpoint using the data on the projection plane (S8); determining whether the driving environment of the vehicle is off-road or on-road based on a signal from another device mounted on the vehicle (S3); and if it is determined that the driving environment is on-road, projecting an on-road projection plane having a predetermined shape as the projection plane (S4). generating a synthetic image using a projection surface; and, when it is determined that the driving environment is off-road, generating a synthetic image using an off-road projection surface having a shape different from that of the on-road projection surface as the projection surface, wherein the on-road projection surface has a flat on-road planar area (R1) arranged around a vehicle area (R0) which is an area where the vehicle is located, and a curved on-road curved area (R2) arranged around the on-road planar area, and the off-road projection surface has a curved off-road curved area (R2a) arranged outside the vehicle area, and the off-road curved area (R2a) has a shape different from that of the on-road curved area (R2). The on-road curved surface area is a parabola expressed by Z=a1×x^2 when the distance from the curved surface start position, which is the position where the on-road curved surface area starts outside the vehicle area, to the outside of the vehicle area is x and the height is Z, and the off-road curved surface area is a parabola expressed by Z=a2×x^2 when the distance from the curved surface start position, which is the position where the off-road curved surface area starts outside the vehicle area, to the outside of the vehicle area is x and the height is Z, and the coefficient a2 is set to a value larger than the coefficient a1. .

[0014] The above method corresponds to the image generating device described above. According to the above method, when the vehicle is traveling in an off-road environment, a projection surface that assumes the traveling environment is off-road and has a different shape from the projection surface applied when the vehicle is on-road is applied. Therefore, it is possible to generate a composite image that is less likely to give a sense of incongruity to the user as a composite image to be displayed when traveling off-road.

[0015] In addition, the symbols in parentheses in the claims indicate a correspondence with the specific means described in the embodiments described below as one aspect, and do not limit the technical scope of the present disclosure. [Brief description of the drawings]

[0016] [Figure 1] FIG. 2 is a block diagram showing an overall view of the peripheral display system Sys. [Diagram 2] 2 is a diagram showing an example of the installation positions and imaging ranges of the cameras 2. FIG. [Diagram 3] FIG. 2 is a diagram showing an example of the installation position and imaging range of each sonar 8. [Figure 4] FIG. 2 is a block diagram for explaining the function of an image generation ECU 1. [Diagram 5] 11 is a diagram showing an example of a camera image acquired by an image acquisition unit F1. FIG. [Figure 6] FIG. 2 is a diagram for explaining a projection plane TS. [Figure 7] FIG. 2 is a diagram for explaining a driver's viewpoint VPd. [Figure 8] FIG. 2 is a diagram for explaining the shape of a normal projection surface TS1. [Figure 9] FIG. 2 is a diagram for explaining the shape of a normal projection surface TS1. [Figure 10] 11 is a diagram for explaining the shape of an off-road projection surface TS2. FIG. [Figure 11] 11 is a diagram for explaining the shape of an off-road projection surface TS2. FIG. [Figure 12] FIG. 2 is a diagram showing an example of a driver's viewpoint image CPd. [Figure 13] FIG. 13 is a diagram showing an example of a bird's-eye view image CPb. [Figure 14] 13 is a flowchart of a composite image display process. [Figure 15] FIG. 13 is a diagram showing an example of the layout of a display image DP. [Figure 16] FIG. 13 is a diagram for explaining how rock Rk appears when a normal projection plane TS1 is used. [Figure 17] 13 is a diagram for explaining how a rock Rk appears when an off-road projection surface TS2 is used. FIG. [Figure 18] 1 is a diagram of a driver's viewpoint image CPd generated using a normal projection plane TS1. [Figure 19]13 is a diagram of a driver's viewpoint image CPd generated using an off-road projection surface TS2. FIG. [Figure 20] 13A and 13B are diagrams showing modified examples of the off-road projection surface TS2. [Figure 21] 13A and 13B are diagrams showing modified examples of the off-road projection surface TS2. [Figure 22] 13A and 13B are diagrams showing modified examples of the off-road projection surface TS2. [Diagram 23] 13 is a diagram showing a display example of a notification image NP indicating that the off-road display mode is selected. FIG. [Figure 24] 1 is a diagram showing an example of a setting position of a virtual viewpoint VP other than a driver's viewpoint VPd in ​​a vehicle interior. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the periphery display system Sys according to the present disclosure will be described with reference to the drawings. The periphery display system Sys displays an image of the periphery of a vehicle equipped with the system on a display. Hereinafter, the vehicle V equipped with the periphery display system Sys will also be referred to as the host vehicle.

[0018] <Introduction> The vehicle of this embodiment is, as an example, a four-wheel drive vehicle that is assumed to travel not only on maintained roads (i.e., on-road) but also off-road, and has a normal mode suitable for on-road travel and an off-road mode as driving modes. The vehicle may also have a first off-road mode suitable for travel on slippery road surfaces such as muddy roads and sandy roads, and a second off-road mode suitable for travel on road surfaces with large irregularities such as mogul roads and rocky roads as off-road modes. Each driving mode has a different method of controlling the distribution of driving force to each wheel. Note that off-road here mainly refers to ground surfaces with large irregularities such as rocky roads. Of course, off-road can also be interpreted as ground surfaces other than on-road, that is, ground surfaces that are not maintained in general.

[0019] In the following description, the front-rear, left-right, and up-down directions are defined with respect to the host vehicle. Specifically, the front-rear direction corresponds to the longitudinal direction of the host vehicle. The left-right direction corresponds to the width direction of the host vehicle. The up-down direction corresponds to the vehicle height direction. From another perspective, the up-down direction corresponds to a direction perpendicular to a plane parallel to the front-rear and left-right directions. In the present disclosure, the plane perpendicular to the vehicle height direction is also referred to as the vehicle horizontal plane. In addition, the direction perpendicular to the vehicle height direction, including the front-rear and left-right directions, is also referred to as the vehicle horizontal direction. The vehicle horizontal direction corresponds to the direction away from the host vehicle.

[0020] In addition, "parallel" in the present disclosure is not limited to a completely parallel state. It may be tilted by several degrees to about 20 degrees. In other words, it may include a state in which it is roughly parallel (a so-called substantially parallel state). The expression "vertical" in the present disclosure is also not limited to a completely vertical state, but may include a state in which it is tilted by several degrees to about 20 degrees.

[0021] <Overall configuration> Fig. 1 is a diagram showing an example of a schematic configuration of a peripheral display system Sys according to the present disclosure. As shown in Fig. 1, the peripheral display system Sys includes an image generation ECU 1, multiple cameras 2, a display 3, a touch panel 4, an operation button 5, a vehicle state sensor 6, a sonar ECU 7, multiple sonars 8, and an integrated ECU 9. Note that ECU in the names of the components is an abbreviation for Electronic Control Unit, and means an electronic control device.

[0022] The image generation ECU 1 is communicatively connected to each of a plurality of cameras 2, a display 3, a touch panel 4, an operation button 5, a vehicle state sensor 6, a sonar ECU 7, and an integrated ECU 9. A plurality of sonars 8 are connected to the sonar ECU 7. The above-mentioned various devices or sensors and the image generation ECU 1 may be individually connected by dedicated lines, or may be connected via an in-vehicle network, which is a communication network built in the vehicle. For example, the camera 2 and the image generation ECU 1 may be directly connected by a dedicated video signal line.

[0023] The image generation ECU 1 is an ECU that generates a composite image CP of the vehicle surrounding area viewed from an arbitrary viewpoint based on image data input from each camera 2 as an image for assisting driving operation of the vehicle V, and displays the composite image CP on a display 3. The image generation ECU 1 is realized using a computer. That is, the image generation ECU 1 includes a processing unit 11, a RAM 12, a storage 13, an I / O 14, and a bus line connecting these components.

[0024] The processing unit 11 is hardware (in other words, a calculation core) for arithmetic processing coupled to the RAM 12. The processing unit 11 is, for example, a CPU. The processing unit 11 accesses the RAM 12 to execute various processes for realizing the functions of each functional unit described later. The RAM 12 is a volatile storage medium.

[0025] The storage 13 includes a non-volatile storage medium such as a flash memory. The storage 13 stores an image generation program Pg as firmware and various data used by the image synthesis unit F7 to generate a synthetic image. The data used to generate such a synthetic image includes a plurality of projection surface data Dt having different parameters that define the surface shape, etc. The execution of the image generation program Pg by the processing unit 11 corresponds to the execution of an image generation method that corresponds to the image generation program Pg. The I / O 14 is a circuit module for communicating with other devices. The I / O 14 is realized using analog circuit elements, ICs, etc. Details of the image generation ECU 1 will be described separately later.

[0026] The camera 2 is an in-vehicle camera that captures images of the surroundings of the vehicle and outputs the captured image data to the image generation ECU 1. Each camera 2 is equipped with at least a lens and an image sensor, and electronically acquires images showing the surroundings of the vehicle. The multiple cameras 2 are attached to the vehicle at predetermined positions and in predetermined orientations so as to capture different ranges. The surroundings display system Sys of this embodiment is equipped with a front camera 2F, a rear camera 2B, a left side camera 2L, and a right side camera 2R as the cameras 2, as shown in FIG. 2. These four cameras 2 are arranged at different positions on the vehicle and capture images of different directions around the vehicle. Specifically, the cameras 2 are as follows.

[0027] The front camera 2F is a camera that captures an image of the front of the vehicle at a predetermined angle of view. The front camera 2F is attached to the front end of the vehicle, such as the front grille, with its optical axis 2Fa facing the front direction of the vehicle. The rear camera 2B is a camera that captures an image of the rear of the vehicle at a predetermined angle of view. The rear camera 2B is disposed at a predetermined position on the rear of the body, such as near the rear license plate or near the rear window, with its optical axis 2Ba facing the rear of the vehicle. The left side camera 2L is a camera that captures an image of the left side of the vehicle. The left side camera 2L is attached to the left side mirror with its optical axis 2La facing the left side of the vehicle. The right side camera 2R is a camera that captures an image of the right side of the vehicle. The right side camera 2R is attached to the right side mirror with its optical axis 2Ra facing the right side of the vehicle.

[0028] The lenses of these cameras 2 are wide-angle lenses such as fisheye lenses, and each camera 2 has an angle of view θ of 180 degrees or more. Therefore, by using four cameras 2, it is possible to capture the entire surroundings of the vehicle. The mounting positions of each camera 2 described above can be changed as appropriate. The front camera 2F may be mounted on the rearview mirror or the upper end of the windshield. The right side camera 2R and the left side camera 2L may be disposed near the base of the A-pillar or the B-pillar. The surroundings display system Sys may include a camera 2 mounted on the roof. Some or all of the cameras 2 may be cameras that are retrofitted, for example, on the roof, on the dashboard, or near the window frame.

[0029] Data indicating the mounting position and attitude of each camera 2 on the vehicle V (hereinafter, mounting position data) is stored in the storage 13. The mounting position of each camera 2 may be expressed, for example, as a point on a three-dimensional coordinate system (hereinafter, vehicle three-dimensional coordinate system) centered on an arbitrary position on the vehicle V. The X-axis forming the vehicle three-dimensional coordinate system may be, for example, an axis parallel to the left-right direction of the vehicle. The Y-axis may be an axis parallel to the front-rear direction. The Z-axis may be an axis parallel to the vehicle height direction. The right direction of the vehicle corresponds to the X-axis positive direction, the front of the vehicle corresponds to the Y-axis positive direction, and the top of the vehicle corresponds to the Z-axis positive direction. The center of the vehicle three-dimensional coordinate system may be, for example, the center of the rear wheel axle.

[0030] The display 3 is a device that includes a thin display panel such as a liquid crystal display, and displays various information and images. The display 3 corresponds to a display device. The display 3 is arranged on an instrument panel of the vehicle so that the user can view the screen. The display 3 may be integrated with the image generation ECU 1 by being arranged in the same housing as the image generation ECU 1. Of course, the display 3 may be a device separate from the image generation ECU 1. The display 3 includes a touch panel 4 stacked on the display panel, and is configured to be able to accept user operations. The touch panel 4 is, for example, a capacitive touch panel, and outputs a signal indicating the user's touch position. The user here mainly refers to a passenger in the driver's seat (a so-called driver). In addition to the driver, the user may include a passenger in the passenger seat.

[0031] The operation button 5 is an operation member that accepts a user's operation. The peripheral display system Sys may include a display changeover switch 51 and a driving mode switch 52 as the operation button 5. The display changeover switch 51 is a switch for displaying the composite image CP generated by the image generation ECU 1 on the display 3 and for changing the virtual viewpoint of the composite image CP to be displayed. The display changeover switch 51 is provided, for example, on the steering wheel of the vehicle and mainly accepts operations from the driver. The user can perform various operations on the peripheral display system Sys via the display changeover switch 51 and the touch panel 4 of the display 3. When the user operates either the display changeover switch 51 or the touch panel 4, an operation signal indicating the content of the operation is input to the image generation ECU 1. The display changeover switch 51 can also be called a multi-information switch. The display changeover switch 51 may be arranged on an instrument panel.

[0032] The driving mode switch 52 is a button for the driver to switch the driving mode of the vehicle V. The driving mode switch 52 may be provided, for example, on a center console or an instrument panel. Buttons corresponding to each driving mode may be arranged as the driving mode switch 52. The driving mode switch 52 may be configured as a rotary switch. The operating member for switching the driving mode may be a dial type. The function of the driving mode switch 52 may be provided by a shift lever. The driving mode switch 52 and the shift lever may correspond to an input device for switching the driving mode to the off-road mode.

[0033] The driving mode switch 52 outputs a signal indicating the driving mode set by the user to the image generation ECU 1. The output signal of the driving mode switch 52 is also input to the integrated ECU 9 that controls the multiple ECUs constituting the drive system of the vehicle. The integrated ECU 9 corresponds to, for example, a domain ECU of a powertrain system. In response to an input signal from the driving mode switch 52, the integrated ECU 9 switches the driving mode of the vehicle V and controls the behavior of the powertrain ECU, chassis ECU, etc. in an integrated manner. In one aspect, such an integrated ECU 9 can be understood as an ECU that manages the driving mode of the vehicle. Note that the power unit that is the object of control of the powertrain ECU is not limited to an engine and may be a motor. The powertrain ECU may have a function of controlling the torque distribution of the front and rear wheels and further independently controlling the torque of the left and right rear wheels. For example, the powertrain ECU may have a function as a TVD (Torque Vectoring Differential) ECU.

[0034] The vehicle state sensor 6 is a sensor that detects a state quantity related to the driving control of the host vehicle. The vehicle state sensor 6 includes a shift position sensor, a vehicle speed sensor, a steering angle sensor, an acceleration sensor, and the like. The shift position sensor is a sensor that detects the position of a shift lever. The vehicle speed sensor is a sensor that detects the driving speed of the host vehicle. The steering angle sensor is a sensor that detects the rotation angle of the steering wheel (so-called steering angle). The acceleration sensor is a sensor that detects the acceleration acting on the host vehicle in at least one of the vehicle front-rear direction, lateral direction, and up-down direction. Here, it is assumed that a three-axis acceleration sensor is adopted as the acceleration sensor. The detection value of the acceleration sensor can be used as a material for determining the vehicle attitude with respect to the horizontal plane. Note that the type of sensor used by the surroundings display system Sys as the vehicle state sensor 6 may be appropriately designed, and it is not necessary to have all of the above-mentioned sensors. In addition, a vehicle height sensor and the like can be included in the vehicle state sensor 6. Each sensor outputs data indicating the current value of the physical state quantity to be detected (i.e., the detection result) to the image generation ECU 1.

[0035] The sonar ECU 7 is an ECU that controls the operation of the sonars 8. The sonars 8 detect objects present around the vehicle by transmitting ultrasonic waves as search waves and receiving the waves reflected by the objects. The sonars 8 can also detect the distance of the object based on the time it takes for the ultrasonic waves to return after being transmitted. Distance information of the object detected by each sonar 8 is input to the image generation ECU 1 via the sonar ECU 7, for example.

[0036] In this embodiment, as an example, eight sonars 8 are provided as shown in FIG. 3. Specifically, four sonars 8A to 8D are distributed from the right corner to the left corner of the front bumper. Four sonars 8E to 8H are distributed from the right corner to the left corner of the rear bumper of the vehicle. With such an arrangement of the sonars 8, the sonar ECU 7 can detect objects present in front of or behind the vehicle. Note that the arrangement of the sonars 8 is merely an example, and is not limited to the example shown in FIG. 3. The sonars 8 may be attached to a side sill, a fender, or a door panel so as to emit ultrasonic waves toward the sides of the vehicle.

[0037] In addition, the sonar ECU 7 identifies the relative position of an object present around the vehicle by combining the detection results of each sonar 8. For example, the direction of a detected object is derived based on the distance to the same object detected by each of two or more sonars 8, and the relative direction of the object with respect to the vehicle is derived. When the sonar 8 detects an object, the sonar ECU 7 inputs data indicating the direction and distance of the object as a detection result to the image generation ECU 1. The sonar ECU 7 may also be configured to output the height of the detected object, the reflection intensity, the received waveform, and the like to the image generation ECU 1 as information indirectly indicating whether the surroundings of the vehicle are an off-road environment. The sonar ECU 7 may also be configured to determine whether the surrounding environment is an off-road environment such as a rocky area based on the received waveform, and output the determination result to the image generation ECU 1. The sonar ECU 7 may be integrated with the image generation ECU 1.

[0038] <Image Generation ECU Configuration> As shown in FIG. 4, the image generation ECU 1 has, as functional units, an image acquisition unit F1, an operation reception unit F2, a vehicle state acquisition unit F3, an obstacle information acquisition unit F4, an image recognition unit F5, a display control unit F6, an image synthesis unit F7, a display image generation unit F8, and an image output unit F9.

[0039] The image acquisition unit F1 acquires camera images, which are images generated by the four cameras 2. When each of the multiple cameras 2 captures an image, four camera images SF, SB, SL, and SR are acquired, which respectively show the front, rear, left side, and right side of the vehicle, as shown in FIG. 5. These four camera images SF, SB, SL, and SR contain data of the entire periphery of the vehicle. The camera image SF is an image generated by the front camera 2F, and the camera image SB is an image generated by the rear camera 2B. The camera image SL is an image generated by the left side camera 2L, and the camera image SR is an image generated by the right side camera 2R. The image acquisition unit F1 has a function of converting an image signal input from the camera 2 into digital image data in a predetermined data format. The image acquisition unit F1 performs a predetermined image processing on the acquired camera images, and inputs the processed camera images to the image synthesis unit F7 and the display image generation unit F8.

[0040] The operation reception unit F2 receives an operation signal output from the operation button 5 and the touch panel 4 when the user performs an operation. In response to this, the operation reception unit F2 receives an instruction operation from the user regarding the display of the composite image CP and the camera image. The operation reception unit F2 inputs data corresponding to the received operation signal to the display control unit F6.

[0041] The vehicle state acquisition unit F3 is configured to acquire information indicating the state of the vehicle from other devices provided in the vehicle separately from the image generation ECU 1, such as the vehicle state sensor 6. The vehicle state acquisition unit F3 acquires the currently set shift position from, for example, a shift position sensor. The vehicle state acquisition unit F3 may acquire the driving mode from the integrated ECU 9. In addition, the vehicle state acquisition unit F3 acquires the acceleration in each detection axial direction, the vehicle speed, the steering angle, and the like.

[0042] The obstacle information acquisition unit F4 acquires information about three-dimensional objects existing around the vehicle from the sonar ECU 7. That is, it acquires the size, height, relative position, etc. of the detected three-dimensional object. In addition, if the sonar ECU 7 is equipped with a classifier that identifies the type of detected object by analyzing the feature amount of the signal waveform of the received reflected wave, it also acquires the identification result of the detected object by the sonar ECU 7.

[0043] The image recognition unit F5 is configured to detect the position and type of a predetermined detection object by analyzing an image input from the camera 2. The image recognition unit F5 has a function as a classifier that identifies the type of object based on, for example, a feature vector of the image. The image recognition unit F5 identifies the object using, for example, a CNN (Convolutional Neural Network) or DNN (Deep Neural Network) technology that applies deep learning. Detection objects include pedestrians, other vehicles, road markings such as lane markings that may be added to paved roads, road edges, etc. Detection objects may also include traffic signs such as direction signs, and three-dimensional structures attached to roads such as guardrails and utility poles.

[0044] In addition, the image recognition unit F5 may be configured to detect rocks. The recognition result of the image recognition unit F5 is output to the display control unit F6, etc. In addition, the image recognition unit F5 may be configured to output a recognition result indicating whether the driving environment is on-road or off-road to the display control unit F6 based on a pattern (e.g., an edge detection result) of an image area assumed to be the ground.

[0045] The display control unit F6 is configured to comprehensively control the entire image generation ECU 1. For example, the display control unit F6 controls the image synthesis unit F7 and the display image generation unit F8 based on information input from the operation reception unit F2 and the vehicle state acquisition unit F3 to generate a synthesis image CP and a display image DP according to the running state of the host vehicle and the user's settings.

[0046] The display control unit F6 includes, as sub-functional blocks, a traveling direction acquisition unit F61, a driving environment determination unit F62, a projection plane control unit F63, and a viewpoint control unit F64. The traveling direction acquisition unit F61 determines whether the traveling direction of the host vehicle is forward or backward, based on, for example, a signal from a shift position sensor or the rotation direction of the tires.

[0047] The driving environment determination unit F62 is configured to determine whether the driving position of the vehicle (in other words, the driving environment) is on-road or off-road. Such a driving environment determination unit F62 can also be called a road surface type determination unit that determines the road surface type. For example, the driving environment determination unit F62 may determine whether the driving environment is off-road based on an input signal from the driving mode switch 52. For example, the driving environment determination unit F62 determines that the driving environment is off-road based on a signal indicating that the driving mode is set to the off-road mode being input from the driving mode switch 52. On the other hand, the driving environment determination unit F62 may determine that the driving environment is the normal mode based on a signal indicating that the driving mode is set to the normal mode being input from the driving mode switch 52. Also, the driving environment determination unit F62 may determine whether the driving environment is off-road based on information on the driving mode acquired by the vehicle state acquisition unit F3 from the integrated ECU 9.

[0048] Furthermore, the driving environment determination unit F62 may determine whether the environment is off-road based on the recognition result of the image recognition unit F5. For example, the driving environment may be determined to be off-road based on the fact that a predetermined number or more objects having characteristics indicating rocks are detected by the image recognition process, or based on the fact that rocks are detected on the front, rear, left and right sides of the vehicle. Furthermore, the driving environment may be determined to be off-road based on the fact that multiple obstacles are continuously detected. The obstacles here refer to natural three-dimensional objects such as rocks and steps on the ground, rather than artificial three-dimensional objects such as guardrails and other vehicles. Note that since the shapes of natural three-dimensional objects such as rocks are diverse, it is difficult to determine the type of the detected object as a natural three-dimensional object by image recognition. On the other hand, if it is an artificial object, it is relatively easy to identify the type. In consideration of such circumstances, the driving environment determination unit F62 may determine whether the driving environment is off-road by regarding a three-dimensional object whose type could not be identified by the image recognition process as a natural three-dimensional object. The driving environment determination unit F62 may also determine that the driving environment is off-road based on the condition that the image recognition unit F5 has not detected any elements indicating a paved road on any of the front, rear, left and right sides of the vehicle. Elements that indicate that the road is paved include, for example, lane markings and road edges.

[0049] The driving environment determination unit F62 may also determine that the driving environment is off-road based on the fact that the above obstacles are detected continuously or on the front, rear, left and right sides of the vehicle by the sonar 8. The driving environment determination unit F62 may determine whether the driving environment is off-road by sensor fusion of image recognition processing and sonar.

[0050] In addition, the driving environment determination unit F62 may determine whether the driving environment is off-road based on the detection pattern of the acceleration sensor. For example, the driving environment may be determined to be off-road when the state in which the acceleration in the vertical direction fluctuates with the amplitude of a predetermined threshold continues for a certain distance (for example, 3 m). Similarly, the driving environment may be determined to be off-road when the vibration sensor detects vibration of a predetermined threshold or more as the vehicle travels continues for a certain distance. The driving environment determination unit F62 may determine that the driving environment is off-road based on the fact that any of a plurality of tires is spinning. The driving environment determination unit F62 may determine whether the driving environment is off-road using the position information of the vehicle identified by the GNSS (Global Navigation Satellite System) and map data. Various types of information and determination conditions that are materials for determining that the driving environment is off-road can be adopted.

[0051] The driving environment determination unit F62 can determine that the driving environment is on-road when the conditions for determining that the driving environment is off-road are no longer satisfied. For example, the driving environment determination unit F62 can determine that the driving environment is on-road when a dividing line, a road edge, a guard rail, a road sign, or the like is detected by image recognition processing. The driving environment may also be determined to be on-road when the amplitude of the vertical acceleration accompanying driving is less than a predetermined value. A variety of conditions can be adopted for determining that the driving environment is on-road.

[0052] The projection plane control unit F63 is configured to switch the projection plane TS used in the image synthesis process. As conceptually shown in FIG. 6, the projection plane TS is a virtual three-dimensional plane corresponding to the peripheral area of ​​the vehicle. The central area of ​​the projection plane TS is determined as the vehicle area R0, which is the position of the vehicle. The image generation ECU1 of this embodiment is configured to selectively use the normal projection plane and the off-road projection plane. The normal projection plane TS1 is the projection plane TS that is applied when the vehicle is located on-road, and the off-road projection plane TS2 is the projection plane TS that is applied when the vehicle is located off-road. The shapes of each of the multiple projection planes TS are determined by multiple projection plane data Dt that are pre-stored in the driving environment determination unit F62. The details of the projection plane TS will be described separately later.

[0053] In addition, in one aspect, switching the projection surface when generating the composite image CP corresponds to switching the display mode of the composite image CP. For convenience, the mode in which the composite image CP is displayed using the off-road projection surface is referred to as the off-road display mode. Also, the display mode in which the composite image CP is displayed using the normal projection surface is referred to as the normal display mode. The normal projection surface TS1 is the projection surface TS that is applied when the host vehicle is located on the on-road, and therefore can be called the on-road projection surface. Also, the normal display mode can be called the on-road display mode.

[0054] The viewpoint control unit F64 sets the position and line of sight of a virtual viewpoint VP when generating a composite image CP, which will be described later, based on at least one of the determination result of the traveling direction acquisition unit F61, the signal from the touch panel 4, and the signal from the display changeover switch 51. For example, a bird's-eye viewpoint VPb, a driver's viewpoint VPd, etc. can be adopted as a pattern of the virtual viewpoint VP that can be set. The bird's-eye viewpoint VPb is a setting pattern of a virtual viewpoint VP with a viewpoint position directly above the vehicle and a line of sight direction directly below, and can be applied when generating a bird's-eye image CPb that is an image of the vehicle and its surroundings viewed from directly above the vehicle. Note that the viewpoint position of the bird's-eye viewpoint VPb is not limited to directly above the vehicle, but may be a position shifted from directly above the vehicle to the rear side, the front side, or the side. The bird's-eye viewpoint VPb corresponds to an example of a vehicle exterior viewpoint in which the virtual viewpoint VP is arranged outside the vehicle cabin.

[0055] As shown in FIG. 7, the driver's viewpoint VPd is a setting pattern of a virtual viewpoint VP in which the viewpoint position is set at the assumed position of the driver's eyes in the vehicle cabin. The line of sight of the driver's viewpoint VPd can be set, for example, in a forward diagonally downward direction so as to include the vicinity of the front wheels. The diagonally downward direction can be, for example, a direction directed downward by about 20 to 30 degrees from the horizontal plane of the vehicle. Note that the line of sight of the driver's viewpoint VPd may be configured to be changeable to any direction based on a user operation (e.g., swipe) on the touch panel 4 while setting the forward diagonally downward direction as the default direction.

[0056] The iris set for each vehicle model can be used as the assumed position of the driver's eyes. The iris is a virtual spatial region defined for each vehicle model, and is set in a virtual ellipsoid shape based on the eye range that statistically represents the spatial distribution of the occupant's eye point (see JISD0021:1998). For example, the assumed position of the driver's eyes is located near the headrest of the driver's seat. The driver's viewpoint VPd corresponds to an example of an in-vehicle viewpoint in which the virtual viewpoint VP is placed inside the vehicle cabin. The position of the driver's viewpoint VPd may be located at a position shifted from the assumed position of the driver's eyes. For example, the driver's viewpoint VPd may be located at a position shifted by a predetermined amount toward the passenger seat side from the assumed position of the driver's eyes, for example, at a position halfway between the driver's seat and the passenger seat.

[0057] Switching the viewpoint when generating the composite image CP corresponds to switching the display mode of the composite image CP. For convenience, the state in which the bird's-eye viewpoint VPb is adopted is called the bird's-eye viewpoint mode. The state in which the driver's viewpoint VPd is adopted is called the driver's viewpoint mode.

[0058] The image synthesis unit F7 is configured to perform image processing for generating a synthetic image CP. The image synthesis unit F7 projects data of a plurality of camera images onto a virtual projection surface corresponding to the periphery of the vehicle, and generates a synthetic image CP showing the periphery of the vehicle as seen from a virtual viewpoint VP using the data on the projection surface. The operation of the image synthesis unit F7 is controlled by the display control unit F6. For example, the projection surface and the virtual viewpoint VP used for generating the synthetic image CP are controlled by the display control unit F6. More specifically, the image synthesis unit F7 selectively uses one of a plurality of projection surfaces each having a different shape under the control of the display control unit F6 to generate the synthetic image CP. The method of generating the synthetic image CP will be described in detail later.

[0059] The display image generating unit F8 generates a display image DP to be displayed on the display 3. The display image generating unit F8 generates a display image DP including the composite image CP and the camera image by using the composite image CP generated by the image composition unit F7 and the camera image acquired by the image acquisition unit F1. The combination of images included in the display image DP is determined by the display control unit F6 according to the traveling direction of the host vehicle and a user operation on the touch panel 4 or the like. That is, the operation of the display image generating unit F8 is controlled by the display control unit F6. The operation of the display image generating unit F8 and the display control unit F6 related to the generation of the display image DP will be described separately later.

[0060] The image output unit F9 converts the display image DP generated by the display image generation unit F8 into a video signal in a predetermined signal format, outputs the display image DP to the display 3, and displays the display image DP on the display 3. As a result, a composite image CP showing the surroundings of the host vehicle as seen from the virtual viewpoint VP is displayed on the display 3.

[0061] <Regarding normal projection surfaces> Here, the normal projection plane TS1 used to generate the composite image CP will be described. The normal projection plane TS1 has a bowl shape with a downwardly convex curved surface. FIG. 8 is a diagram showing a cross section of the normal projection plane TS1 along the left-right direction of the vehicle, and FIG. 9 is a diagram showing a cross section of the normal projection plane TS1 along the front-rear direction of the vehicle. As shown in the diagram, the normal projection plane TS1 is roughly formed as a plane along the horizontal direction of the vehicle in the vicinity of the vehicle region R0, and has a shape whose inclination (gradient) increases as it moves away from the vehicle region R0. In this specification, the "inclination" of the projection plane TS represents the vertical length relative to a unit length in the horizontal direction of the vehicle. The "inclination" of each position of the projection plane TS can also be said to be the inclination of the tangent to that position.

[0062] The normal projection surface TS1 can be divided into a planar vehicle region R0 corresponding to the road surface region on which the vehicle is located, a planar region R1 that is a plane continuous with the vehicle region R0, and a curved region R2 that is a downwardly convex curved surface away from the vehicle region R0. The planar region R1 corresponds to the on-road planar region, and the curved region R2 corresponds to the on-road curved region.

[0063] Such a normal projection plane TS1 can be said to have a planar area R1 that is relatively small in distance from the vehicle area R0 and has a relatively small inclination, and a curved area R2 that is relatively large in distance from the vehicle area R0 and has a relatively large inclination. Note that the vehicle area R0 corresponds to an area that overlaps with the host vehicle in a top view.

[0064] The planar region R1 is disposed adjacent to the vehicle region R0 so as to surround the vehicle region R0. The curved region R2 is disposed outside the planar region R1. From another point of view, such a configuration of the normal projection surface TS1 corresponds to a configuration in which the planar region R1 is interposed between the vehicle region R0 and the curved region R2. The planar region R1 extends from the edge of the vehicle region R0 to a point at least a predetermined minimum planar distance Dmin or more in the horizontal direction of the vehicle. In other words, the planar region R1 is set within at least the minimum planar distance Dmin from the vehicle region R0. The minimum planar distance Dmin can be, for example, 0.3 m, 0.5 m, or 1 m. The planar formation distance D1, which is the distance from the edge of the vehicle region R0 to the boundary between the planar region R1 and the curved region R2, is set to be equal to or greater than the minimum planar distance Dmin. The planar formation distance D1 can be, for example, 1.5 m. The size of the planar formation distance D1 may be different in the front-rear direction and the left-right direction. For example, the plane forming distance D1 in the front-rear direction may be set to be approximately 0.25 to 0.5 m larger than the plane forming distance D1 in the left-right direction.

[0065] The curved area R2 is formed outside the planar area R1, i.e., on the periphery of the normal projection surface TS1, and has a shape with a gradually increasing inclination. For example, the curved area R2 can be shaped like a quadratic curve, in other words, a parabola. Therefore, the shape of the curved area R2 can be defined by the coefficient parameters of the quadratic curve. For example, the cross-sectional shape of the curved area R2 provided on the normal projection surface TS1 can be expressed by the following formula (2) using coefficient a1, where the outer edge of the planar area R1 is the origin, the vehicle horizontal direction is the x-axis, and the vehicle height direction is the z-axis. The positive direction of the x-axis is the direction away from the vehicle area R0.

[0066] z = a1 x^2 …(1) The coefficient a1 is a parameter that defines the degree of inclination of the curved region R2, and can also be called the inclination coefficient. The magnitude of a1 can be appropriately designed. That is, the larger the coefficient a1, the greater the inclination of the curved region R2. Of course, the model of the curved region R2 is not limited to a quadratic curve, and may be a circular arc, a logarithmic function, or an exponential function.

[0067] In this example, the camera image data is not projected onto the vehicle area R0, but onto an area outside the vehicle area R0. In other words, the vehicle area R0 corresponds to a non-projection area onto which the camera image is not projected. Hereinafter, the area onto which the camera image data is projected (the area outside the vehicle area R0) on various projection surfaces TS is also referred to as the "projection target area." On a normal projection surface TS1, the planar area R1 and the curved area R2 correspond to the projection target area.

[0068] The normal projection surface TS1 includes a front area PF, which is an area corresponding to the front of the vehicle, a rear area PB, which is an area corresponding to the rear, a left area PL, which is an area corresponding to the left side, and a right area PR, which is an area corresponding to the right side. The front area PF is an area onto which the image SF of the front camera 2F is projected, and the rear area PB is an area onto which the image SB of the rear camera 2B is projected. The left area PL is an area onto which the image SL of the left side camera 2L is projected, and the right area PR is an area onto which the image SR of the right side camera 2R is projected. Each area PF, PB, PL, and PR of the normal projection surface TS1 includes a planar area R1. Each position in the projection target area of ​​the normal projection surface TS is associated with one of the four camera images SF, SB, SL, and SR by correspondence information such as table data.

[0069] <About off-road projection screen> Next, the off-road projection surface TS2 will be described with reference to Figures 10 and 11. Like the normal projection surface TS1, the off-road projection surface TS2 is a projection surface TS for generating a composite image CP, and corresponds to the surface of a three-dimensional model onto which each camera image is projected (in other words, texture mapping).

[0070] FIG. 10 is a diagram showing a cross section of the off-road projection surface TS2 along the left-right direction of the vehicle, and FIG. 11 is a diagram showing a cross section of the off-road projection surface TS2 along the front-rear direction of the vehicle. As shown in FIG. 10 and FIG. 11, the off-road projection surface TS2 is generally bowl-shaped with a downwardly convex curved surface. Such an off-road projection surface TS2 can be divided into a vehicle area R0 where the vehicle is located and a curved surface area R2a that is a downwardly convex curved surface surrounding the vehicle area R0. In other words, the off-road projection surface TS2 of this embodiment does not have a flat area between the vehicle area R0 and the curved surface area R2a. The curved surface area R2a corresponds to the off-road curved surface area.

[0071] The curved area R2a is formed so as to join with the edge of the vehicle area R0. In the off-road projection surface TS2 of this embodiment, only the curved area R2a corresponds to the projection target area. The curved area R2a includes an area as a front area PF facing the front of the vehicle, an area as a rear area PB facing the rear part of the vehicle, an area as a left side area PL facing the left side, and an area as a right side area PR facing the right side.

[0072] The curved surface region R2a has a smaller inclination closer to the vehicle region R0 and a larger inclination farther away from the vehicle region R0. In other words, the curved surface region R2a has a shape in which the inclination gradually increases toward the horizontal direction of the vehicle. For example, the cross-sectional shape of the curved surface region R2a can be expressed by the following equation (2) where the edge of the vehicle region R0 is the origin, the horizontal direction of the vehicle is the x-axis, and the vehicle height direction is the z-axis.

[0073] z = a2 x^2 …(2) a2 is a coefficient that determines the magnitude of the inclination of the curved area R2a. The coefficient a2 can also be called the inclination coefficient. The larger the coefficient a2, the larger the inclination of the curved area R2a. The coefficient a2 is set to a value larger than the above-mentioned coefficient a1. In other words, the curved area R2a of the off-road projection surface TS2 is formed with a steeper inclination than the curved area R2 constituting the normal projection surface TS1. For example, the inclination of the curved area R2a of the off-road projection surface TS2 is set to 1.5 times or more the inclination of the curved area R2 constituting the normal projection surface TS1. For example, it is set to satisfy the relationship a2≧1.5·a1.

[0074] The inclination of the curved area R2a of the off-road projection surface TS2 may be different between the areas PL, PR facing the sides of the vehicle and the areas PF, PB facing the front or rear of the vehicle. For example, the forward area PF and the rear area PB may be set to have a larger inclination than the left and right side areas. The coefficient a2 in the front direction and the rear direction of the vehicle may be set to 1.2 times or more the coefficient a2 in the right direction and the left direction of the vehicle. The inclination is gradually changed between the parts of the curved area R2a with different inclinations.

[0075] In this embodiment, the curved region R2a is formed in a downwardly convex parabolic shape, but this is not limiting. The model of the curved region R2a is not limited to a quadratic curve, and may be a circular arc, a logarithmic function, or an exponential function.

[0076] <How to generate a composite image> Here, a method is described in which the image synthesis unit F7 generates a synthetic image CP that shows the surroundings of the vehicle as seen from the virtual viewpoint VP. Note that the method of generating the synthetic image CP is described here by taking as an example a case in which a normal projection plane TS1 is used as the projection plane TS. The same can be applied to a case in which an off-road projection plane TS2 is used as the projection plane TS.

[0077] When generating a composite image CP, the image synthesis unit F7 first projects the data (values ​​of each pixel) contained in the four camera images SF, SB, SL, and SR input from the image acquisition unit F1 onto a normal projection surface TS1 in a virtual three-dimensional space. The projection positions of the camera images on the normal projection surface TS1 are associated in advance by correspondence information such as table data. The image synthesis unit F7 projects the data of the four camera images SF, SB, SL, and SR onto the corresponding areas of the normal projection surface TS1.

[0078] Specifically, the image synthesis unit F7 projects the data of the camera image SF of the front camera 2F onto the forward region PF of the normal projection surface TS1. The image synthesis unit F7 also projects the data of the camera image SB of the rear camera 2B onto the rear region PB of the normal projection surface TS1. Furthermore, the image synthesis unit F7 projects the data of the camera image SL of the left side camera 2L onto the left side region PL of the normal projection surface TS1, and projects the data of the camera image SR of the right side camera 2R onto the right side region PR.

[0079] In addition, the portions of the camera images that protrude beyond the projection target area may be deleted. For overlapping areas, which are areas captured by two cameras in an overlapping manner, a method of blending the images captured by the two cameras at a predetermined ratio or a method of joining the images captured by the two cameras at a predetermined boundary line may be adopted.

[0080] When the data of the corresponding camera image is projected onto each portion of the normal projection surface TS1, the image synthesis unit F7 then generates a host vehicle image Pv, which is a 3D model of the host vehicle. The data for drawing the host vehicle image Pv may be stored in advance as drawing data in the storage 13. This host vehicle image Pv is placed on the vehicle area R0.

[0081] Next, the image synthesis unit F7 sets a virtual viewpoint VP in a three-dimensional space including the normal projection surface TS1 under the control of the display control unit F6. The image synthesis unit F7 can set the virtual viewpoint VP facing an arbitrary viewing direction at an arbitrary viewpoint position in the three-dimensional space. Then, the image synthesis unit F7 cuts out data projected onto an area included in a predetermined viewing angle as viewed from the set virtual viewpoint VP on the normal projection surface TS1 as an image. In addition, the image synthesis unit F7 performs rendering on the host vehicle image Pv according to the set virtual viewpoint VP, and superimposes the resulting two-dimensional host vehicle image Pv on the cut-out image. As a result, the image synthesis unit F7 generates a synthetic image CP showing the host vehicle and the area around the host vehicle as viewed from the virtual viewpoint VP.

[0082] For example, when the position of the virtual viewpoint VP is set to the driver viewpoint VPd, the image synthesis unit F7 generates a driver viewpoint image CPd showing the front of the vehicle while transmitting the instrument panel, etc., as shown in FIG. 12. The driver viewpoint image CPd corresponds to an image in which a component image Pve showing the component of the host vehicle seen from the driver viewpoint VPd is superimposed on an image of a subject existing in front of the vehicle seen from the driver. By checking such a driver viewpoint image CPd, the user can check the surroundings of the host vehicle from a viewpoint in the vehicle cabin, and can intuitively grasp the surroundings of the host vehicle. In addition, the user can intuitively grasp which direction the driver viewpoint image CPd shows around the host vehicle based on the component image Pve included in the driver viewpoint image CPd.

[0083] The component image Pve includes, for example, a floor image Pvf, a tire image Pvt, and an interior image Pvi. The floor image Pvf is an image showing an area where the bottom of the vehicle body is present, and is set to be non-transparent, for example. The tire image Pvt is an image drawn using a 3D model of a tire, and is set to be opaque or semi-transparent. The tire image Pvt may be an image of only an opaque or semi-transparent outline. In the tire image Pvt, the inside of the outline may be set to be colorless and transparent. In addition, the composite image CP may include a frame image showing the road surface area where the tire is present, instead of or in parallel with the tire image Pvt. The interior image Pvi is an image showing an instrument panel, a steering wheel, an A-pillar, and the like. The interior image Pvi is also set to be transparent or semi-transparent in areas other than the outline. In addition, the outline of the interior image Pvi may be set to be semi-transparent. In addition, the interior image Pvi may not be displayed. 12 shows a state in which the steering wheel, A-pillar, and headlights are displayed, but images showing the positions of these members do not have to be displayed. The elements included in the component member image Pve can be changed as appropriate.

[0084] As described above, in the composite image CP, components that are unlikely to be displayed in order for the driver to recognize the positional relationship between the vehicle V and the surrounding objects are hidden or displayed only as outlines. In other words, the composite image CP is adjusted to include as few component images Pve as possible. This configuration can prevent the visibility of the surroundings of the vehicle from being impaired.

[0085] In addition, when the position of the virtual viewpoint VP is set to a bird's-eye view VPb, the image synthesis unit F7 generates a bird's-eye view image CPb, which is a synthetic image CP that overlooks the vehicle and its surrounding area from above, as shown in Figure 13.

[0086] The initial state of the position of the virtual viewpoint VP and the line of sight direction in the synthetic image CP is determined based on at least one of the traveling direction of the vehicle, the user's presetting, and the steering angle. In addition, the position of the virtual viewpoint VP and the line of sight direction in the synthetic image CP can be changed based on the user's operation on the touch panel or the like. The image synthesis unit F7 of this embodiment is configured to mainly adopt the driver's viewpoint VPd. That is, the image synthesis unit F7 is configured to generate a driver's viewpoint image CPd, which is a synthetic image CP from the driver's line of sight with the vehicle configuration transparent, as the synthetic image CP to be generated first when a display start condition described later is satisfied.

[0087] <Operation flow> The flow of the composite image display process, which is a series of processes executed by the image generation ECU 1 when displaying the composite image CP, will be described with reference to the flowchart shown in FIG. 14. The flowchart shown in FIG. 14 is started when a predetermined display start condition is satisfied. The display start condition can be understood as a condition for displaying the composite image CP, in other words, a condition for starting this flow. For example, the image generation ECU 1 determines that the display start condition is satisfied when the display changeover switch 51 is pressed by the user. In addition, the image generation ECU 1 may be configured to determine that the display start condition is satisfied when it detects that a predetermined user operation for displaying the composite image CP has been performed via the touch panel 4. In other words, this flow can be started based on the operation acceptance unit F2 acquiring a signal indicating that an operation for displaying the composite image CP has been performed.

[0088] The items constituting the display start condition may include a shift position, a vehicle speed, etc. For example, the display start condition may include at least one of the following: the shift position is set to a predetermined range, and the vehicle speed is less than a predetermined threshold. In addition, this flow may be started when a viewpoint switching operation is received while the front camera image SF or the rear camera image SB is being displayed.

[0089] 14 may be repeatedly executed at a predetermined cycle (e.g., every 1 / 30 seconds) until a predetermined display cancellation condition is satisfied. The display cancellation condition may be, for example, when the display changeover switch 51 is pressed again, when the shift position is set to a position other than a predetermined range, when the vehicle speed is equal to or greater than a predetermined threshold, etc. Additionally, the image generation ECU 1 may be configured to determine that the display cancellation condition is satisfied when it detects that an operation for terminating the display of the composite image CP has been performed via the touch panel 4.

[0090] Here, as an example, the composite image display process includes steps S1 to S10. Of course, the number of steps constituting the composite image display process and the processing order can be changed as appropriate.

[0091] First, in step S1, the image acquisition unit F1 acquires four camera images SF, SB, SL, and SR obtained by the four cameras 2, respectively, and then proceeds to step S2. Step S1 can be called an image acquisition step. In step S2, the vehicle state acquisition unit F3 acquires information indicating the state of the host vehicle, such as the shift position and vehicle speed, and then proceeds to step S3. Step S2 can be called a vehicle state acquisition step.

[0092] In step S3, the driving environment determination unit F62 determines whether or not the driving environment corresponds to off-road, for example, based on a signal from the driving mode switch 52. Of course, the method of determining that the driving environment is off-road can employ the various methods described above. Step S3 can be called a driving environment determination step. Here, if the driving environment determination unit F62 determines that the driving environment is off-road, a positive determination is made in step S4 and the process proceeds to step S5. On the other hand, if the driving environment is determined to be on-road, a negative determination is made in step S4 and step S6 is executed.

[0093] In step S5, the display control unit F6 as the projection plane control unit F63 outputs a signal to the image synthesis unit F7 to instruct the image synthesis unit F7 to use the off-road projection plane TS2 as the projection plane TS used to generate the synthetic image CP. Based on the instruction from the display control unit F6, the image synthesis unit F7 reads out the projection plane data Dt corresponding to the off-road projection plane TS2 from the storage 13 and acquires the shape of the off-road projection plane TS2. This causes the off-road projection plane TS2 to be applied to the image synthesis process. Such step S5 can be called an off-road projection plane application step in which the image synthesis unit F7 selects the off-road projection plane TS2 as the projection plane TS under the control of the display control unit F6. When the processing in step S5 is completed, the process proceeds to step S7.

[0094] In step S6, the display control unit F6 as the projection plane control unit F63 outputs a signal to the image synthesis unit F7 to instruct it to use the normal projection plane TS1 as the projection plane TS used to generate the composite image CP. This results in the normal projection plane TS1 being applied to the image synthesis process. Such step S6 can be called a normal projection plane application step in which the image synthesis unit F7 selects the normal projection plane TS1 as the projection plane TS under the control of the display control unit F6. When the process in step S6 is completed, the process proceeds to step S7. Such a series of processes from step S4 to S6 can be called a projection plane selection step in which the projection plane TS used to generate the composite image CP is switched based on the determination result of the driving environment determination unit F62.

[0095] In step S7, the viewpoint control unit F64 determines a virtual viewpoint VP for generating a composite image CP. When the driving power is turned on and the composite image CP is displayed for the first time, the setting of the virtual viewpoint VP can be a position and a line of sight direction preset as a default setting by a designer or a user. The default setting can be, for example, a driver's viewpoint VPd with a line of sight direction directed diagonally downward and forward. In addition, the setting of the virtual viewpoint VP read out in step S7 can be the position and direction of the virtual viewpoint VP displayed by the driver last time. In this case, the display control unit F6 is configured to store the setting data of the virtual viewpoint VP when the composite image CP was displayed last time in the storage 13 or the like as a preparation process.

[0096] The position and line of sight of the virtual viewpoint VP may be determined according to the traveling direction of the vehicle. For example, when the traveling direction of the vehicle acquired by the traveling direction acquisition unit F61 is forward, the driver's viewpoint VPd with the line of sight directed diagonally downward forward as described above is adopted as the virtual viewpoint VP. On the other hand, when the traveling direction of the vehicle acquired by the traveling direction acquisition unit F61 is backward, for example, the driver's viewpoint VPd with the line of sight directed diagonally downward backward may be adopted as the virtual viewpoint VP for generating the synthetic image CP. In addition, the line of sight of the virtual viewpoint VP may be adjusted to a direction according to the steering angle. The viewpoint control unit F64 may also acquire the position and direction of the virtual viewpoint VP designated by the user via the touch panel 4 or the like, and set the virtual viewpoint VP at the designated position and direction.

[0097] The setting information of the virtual viewpoint VP determined in step S7 is output to the image synthesis unit F7. As an example, the driver's viewpoint VPd whose line of sight is directed diagonally downward and forward is applied as the virtual viewpoint VP. Such a step S7 can be called a virtual viewpoint setting step. When step S7 is completed, the process proceeds to step S8.

[0098] In step S8, the image synthesis unit F7 generates a synthetic image CP seen from the virtual viewpoint VP set in step S7, using the projection plane TS according to the determination result of the driving environment determination unit F62. That is, a driver's viewpoint image CPd showing the ground in front of the vehicle seen from the driver's viewpoint VPd is generated.

[0099] For example, when the normal projection surface TS1 is applied, the image synthesis unit F7 projects the data of each camera image on the projection target area including the planar area R1 of the normal projection surface TS1. Then, a driver's viewpoint image CPd is generated using the data on the normal projection surface TS1 on which the various camera images are projected. Also, when the off-road projection surface TS2 is applied, the image synthesis unit F7 projects the data of each camera image on the curved area R2a as the projection target area of ​​the off-road projection surface TS2. Then, a driver's viewpoint image CPd is generated using the data of the off-road projection surface TS2 on which the respective camera images are mapped. The data of the synthetic image CP generated by the image synthesis unit F7 is output to the display image generation unit F8. Such step S8 can be called a synthetic image generation step. When the process in step S8 is completed, the process proceeds to step S9.

[0100] In step S9, the display image generating unit F8 generates a display image DP to be displayed on the display 3 using the composite image CP generated by the image synthesis unit F7. For example, the display image generating unit F8 generates an image including a driver viewpoint image CPd as a composite image CP, a right side camera image SR, a left side camera image SL, and a viewpoint changeover switch image SW as the display image DP, as shown in FIG. 15. Specifically, the driver viewpoint image CPd is arranged on the upper side of the central area of ​​the display image DP, and the viewpoint changeover switch image SW is arranged below it. The right side camera image SR is arranged on the right side of the driver viewpoint image CPd, and the left side camera image SL is arranged on the left side of the driver viewpoint image CPd. According to such a layout of the display image DP, the driver can visually notify the left and right situations while maintaining good visibility of the driver viewpoint image CPd by the user. In other words, by looking at the display image DP, the driver can recognize the state of the front wheels and the ground near the front end of the vehicle, and can also recognize the situation on the side of the vehicle at the same time.

[0101] The viewpoint changeover switch image SW included in the display image DP is an image that functions as a switch for switching the display contents as the display image DP by being touched by the user. Whether or not the user has touched the viewpoint changeover switch image SW can be determined based on a touch position signal output from the touch panel 4. When the user's touch operation on the viewpoint changeover switch image SW is detected, for example, the display image generation unit F8 switches the image displayed in the image central area from the driver viewpoint image CPd to the front camera image SF in response to an instruction from the display control unit F6. The viewpoint changeover switch image SW may be one that switches the virtual viewpoint VP from the driver viewpoint VPd to the bird's-eye viewpoint VPb. In that case, for example, the display image generation unit F8 generates an image including the bird's-eye view image CPb as the display image DP. The data of the display image DP generated by the display image generation unit F8 is output to the image output unit F9. Step S9 can be called a display image generation step.

[0102] In S10, the image output unit F9 converts the digital data of the display image DP generated by the display image generation unit F8 into a signal in a predetermined signal format and outputs it to the display 3. As a result, the display image DP including the driver's viewpoint image CPd is displayed on the display 3. Step S10 can be called an image output step.

[0103] The above control mode is merely an example, and the display image generating unit F8 can select a camera image according to the traveling direction of the vehicle under the control of the display control unit F6 to use for generating the display image DP. For example, when the traveling direction is a reverse direction, the rear camera image SB may be arranged in the central area of ​​the display image DP. Furthermore, the display control unit F6 can change the combination and layout of images to be displayed in the display image DP, the display range of the camera image, and the like, based on the operation signal received by the operation receiving unit F2.

[0104] <Effect of switching projection surfaces> Here, the effect of switching the projection plane TS used for generating a composite image from the normal projection plane TS1 to the off-road projection plane TS2 when the driving environment is off-road will be described. Note that, as a premise, when the vehicle is off-road, there are a plurality of continuous three-dimensional objects such as rocks and terrain steps around the vehicle compared to when the vehicle is on-road. Also, when off-road, the ground itself is often not flat. Therefore, when the driving environment is off-road, it is important to display these rocks and terrain without losing the three-dimensional effect and realism.

[0105] 16 is a diagram for explaining the position where the image of a rock Rk existing near the vehicle is projected when the normal projection plane TS1 is used to generate the composite image CP. The rock Rk is a rock that exists relatively close to the vehicle, and represents a rock that exists within the planar region R1 of the normal projection plane TS1, that is, within the planar formation distance D1 from the vehicle. Note that the rock here refers to a stone having a size of, for example, 20 cm or more.

[0106] The image of the rock Rk obtained by photographing the rock Rk with a camera 2 (for example, the front camera 2F) is projected at a position where a straight line connecting the position of the camera 2 and the position of the rock Rk intersects with the normal projection plane TS1. Since the rock Rk is located near the vehicle, the image of the rock Rk is projected onto the planar region R1 of the normal projection plane TS1. On the normal projection plane TS1, X1 in the figure represents the range where the image of the rock Rk is projected. Also, the dashed line attached to the rock Rk represents the surface portion imaged by the front camera 2F. As shown in FIG. 16, the portion of the surface of the rock Rk that is imaged by the front camera 2F is located at a position higher than the horizontal plane where the planar region R1 is located. Therefore, the image of the rock Rk is projected in a manner stretched further back (in other words, outside) than the actual position in the planar region R1.

[0107] Therefore, in the synthetic image CP in which the normal projection surface TS1 is viewed from the driver's viewpoint VPd, the image of the rock Rk near the vehicle appears larger than the actual rock and is flat, losing the sense of three-dimensionality. This phenomenon also occurs with other three-dimensional objects such as rocks that exist in the planar region R1. In addition, in an off-road environment, the driver tends to focus on the part of the driver's viewpoint image CPd that is close to the vehicle V in order to place the tires on the targeted rocky area or to perform an operation to avoid a large rock that may damage the body. As a result, when the user views the driver's viewpoint image CPd using the normal projection surface TS1, the user may feel that the surroundings of the vehicle are flat. In addition, the user is likely to have the impression that the three-dimensional objects are displayed in a crushed manner.

[0108] Of course, when the vehicle is traveling on an on-road environment, that is, on a flat road, the visual impression given to the user by using the normal projection surface TS1 matches the actual ground shape. Therefore, when the vehicle is traveling on a on-road environment, the normal projection surface TS1 has a suitable shape as the projection surface TS for generating a synthetic image. On the other hand, as described above, in an off-road environment, the ground is rarely flat, and the visual impression is likely to not match the actual ground shape. As a result, there is a relatively high possibility that the configuration using the normal projection surface TS1 even when traveling off-road will give the user a sense of discomfort.

[0109] On the other hand, FIG. 17 is a diagram for explaining the position where the image of the rock Rk is projected when the off-road projection surface TS2 is used to generate the composite image CP. The image of the rock Rk captured by the camera 2 is projected at a position where a straight line connecting the position of the camera 2 and the position of the rock Rk intersects with the off-road projection surface TS2. Here, the off-road projection surface TS2 does not have a flat area R1 between the vehicle area R0 and the curved area R2a. The image of the rock Rk is projected onto the curved area R2a. That is, the image of the rock Rk is projected onto the projection position X2 on the curved area R2a.

[0110] In this way, when the off-road projection surface TS2 is used, the image of the rock Rk is projected at a projection position X2 closer to the vehicle area R0 than the projection position X1 where the image is projected when the normal projection surface TS1 is used. In addition, since the imaging surface of the rock Rk and the projection surface are relatively close to each other, the risk and the degree of stretching of the image of the rock Rk can be reduced. In other words, the risk that the image of the rock Rk is displayed distorted in the driver's viewpoint image CPd can be reduced. As a result, the impression that the driver gets when looking at the synthetic image CP can be made closer to the impression that the driver gets when looking directly at the rock Rk. In addition, it is possible to make it difficult for the driver to get the impression that the surroundings of the vehicle are flat or that three-dimensional objects are crushed.

[0111] Figures 18 and 19 show two driver's viewpoint images CPd generated in the same off-road environment. Figure 18 is a line diagram of the driver's viewpoint image CPd generated using the normal projection surface TS1, and Figure 19 is a line diagram of the driver's viewpoint image CPd generated using the off-road projection surface TS2. In Figures 18 and 19, in order to ensure visibility of the figures, the display of component images Pve other than the floor image Pvf, such as tire images, is omitted.

[0112] As can be seen by comparing the driver's viewpoint images CPd shown in Figures 18 and 19, compared to the driver's viewpoint image CPd using the normal projection surface TS1, rocks and other objects near the vehicle are represented three-dimensionally in the driver's viewpoint image CPd using the off-road projection surface TS2.

[0113] In this way, when the driving environment is determined to be off-road, the synthetic image CP is generated using the off-road projection plane TS2 that does not include the planar area R1 outside the vehicle area R0, thereby reducing the possibility that the user will feel uncomfortable. In addition, the user can be prevented from misunderstanding the road surface conditions around the vehicle, allowing the user to drive the vehicle safely.

[0114] In addition, when the driving environment is determined to be on-road, the normal projection surface TS1 is used to make the visual impression given to the user regarding the road surface shape consistent with the actual ground shape. In other words, by switching the projection surface TS depending on whether the driving environment is off-road or not, it is possible to reduce the risk of the user feeling uncomfortable in each scene.

[0115] In the above, the effect of this embodiment has been described by taking as an example a case where the driver's viewpoint VPd with the line of sight facing forward is used as the virtual viewpoint VP of the composite image CP, but the same effect can be obtained when the line of sight is directed in other directions, such as backward, side, or diagonally side. The same effect can also be obtained when the virtual viewpoint VP is placed at any position in the vehicle cabin other than the driver's viewpoint VPd. In addition, the same effect can be expected when the virtual viewpoint VP is set on the outer surface of the vehicle V or in the vehicle vicinity area outside the vehicle cabin. The vehicle vicinity area here refers to an area within 0.2 m from the outer surface of the vehicle, for example. The outer surface can include the left and right side surfaces, as well as the back surface, front end, and roof. The side surface can include door panels, fenders, pillars, etc.

[0116] Although the embodiment of the present disclosure has been described above, the present disclosure is not limited to the above-mentioned embodiment, and various modified examples described below are also included in the technical scope of the present disclosure. Furthermore, in addition to the following, various modifications can be implemented within the scope of the gist. For example, the following various modified examples can be implemented in appropriate combinations within the scope of no technical contradiction. Note that the same reference numerals are used for components having the same functions as those described in the above-mentioned embodiment, and their description is omitted. In addition, when only a part of the configuration is mentioned, the configuration of the embodiment described above can be applied to the other parts.

[0117] <Application of underfloor confirmation images> Although the above describes a mode in which the composite image CP including the opaque floor image Pvf is displayed as the driver's viewpoint image CPd, the present invention is not limited to this. For example, a front camera image SF showing the ground from directly below the front end of the vehicle to 3 m ahead when the vehicle is moving forward may be stored in the RAM 12 or the like, and the stored image data may be used to generate a composite image CP with the bottom of the vehicle body being transparent as the vehicle moves forward. The image data to be projected onto the underfloor portion may be updated as the vehicle V travels. The display control may be applied not only when the vehicle is moving forward, but also when the vehicle is moving backward. When moving backward, an image of the underfloor may be generated using the rear camera image SB.

[0118] <Additional information on the off-road projection surface TS2> In the above, the off-road projection surface TS2 does not have a planar area, but this is not limited to the above. The off-road projection surface TS2 may include a planar area R1a with a width of less than 0.3 m between the vehicle area R0 and the curved area R2a as shown in FIG. 20 and FIG. 21. The planar area R1a corresponds to the off-road planar area. If the width of the planar area R1a is less than 0.3 m, the display range in the driver's viewpoint image, etc. is small. In addition, since the camera 2 is positioned almost directly above the area that exists within 0.3 m from the vehicle body, the image is captured almost directly from above. As a result, even if the captured image of the area is projected onto a flat projection surface, the image is unlikely to be distorted. Therefore, if the planar area R1a is small, the driver is unlikely to get the impression that a three-dimensional object near the vehicle is crushed. In other words, even if the off-road projection surface TS2 includes a small planar area R1a on the front side of the curved area R2a, the same effect as the above embodiment can be obtained.

[0119] The configurations exemplified above, including the embodiment, correspond to a configuration in which the curved surface area of ​​the projection surface TS is located closer to the vehicle when the driving environment is determined to be off-road than when the driving environment is determined to be on-road. Specifically, when the driving environment is determined to be on-road, the curved surface start position is located 0.3 m or more away from the vehicle, whereas when the driving environment is determined to be off-road, the curved surface start position is located at a position less than 0.3 m away from the vehicle. The curved surface start position is the point where the curved surface areas R2 and R2a start, and can be interpreted as the rising position of the projection surface. The curved surface start position can be a point at which the inclination with respect to the plane to which the vehicle area R0 belongs is, for example, 3 to 5 degrees or more. The configuration in which the curved surface start position is located at a position less than 0.3 m away from the vehicle includes a configuration in which the curved surface start position is located at a position 0.0 m away from the vehicle and a configuration in which the curved surface start position is located inside the vehicle area.

[0120] <Using Obstacle Information> In a situation where the driving environment is determined to be off-road by the driving environment determination unit F62, the inclination and shape of the off-road projection surface TS2 may be dynamically adjusted according to the detection result of the sonar 8. For example, even if the driving environment is determined to be off-road, if no obstacles are detected in the surroundings, the inclination of the curved area R2a may be made smaller than the default value. In addition, in a case where the driving environment is determined to be off-road, if it is detected that a plurality of three-dimensional objects having a height equal to or greater than a predetermined threshold are present within a predetermined distance from the vehicle, the inclination of the curved area R2a (actually, the coefficient a2) may be made larger than the default value. Similarly, in a case where the number of detected three-dimensional objects is equal to or greater than a predetermined value, for example, if the unevenness of the ground around the vehicle is equal to or greater than a predetermined value, the inclination of the curved area R2a (actually, the coefficient a2) may be made larger than the default value. The inclination of the curved area R2a can be adjusted by the coefficient a2.

[0121] R2aH shown in FIG. 22 represents a curved area whose inclination is set larger than the default value, and R2aL represents a curved area whose inclination is set larger than the default value. For convenience, the projection surface TS including the curved area R2aL is also called the small inclination projection surface TS3. The projection surface TS including the curved area R2aH is also called the steep inclination projection surface TS4. In addition, the off-road projection surface TS2 located between the small inclination projection surface TS3 and the steep inclination projection surface TS4 in terms of the inclination of the curved area R2a is also called the intermediate projection surface. Note that the inclination of the curved area R2a of the off-road projection surface TS2 may be dynamically determined within a predetermined range according to the detection status of an obstacle by the sonar 8, rather than being adjusted based on a default value. In addition, the curved area R2a of the off-road projection surface TS2 is not limited to a downwardly convex curved shape. It may also be an upwardly convex curved shape.

[0122] <Obstacle sensor supplement> In the above, the sonar 8 is used as a sensor for detecting objects around the vehicle (so-called obstacle sensor), but the obstacle sensor may be a millimeter wave radar or a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging). The image generation ECU 1 can be connected to various obstacle sensors for use.

[0123] <Additional information on the driving environment judgment section> The driving environment determination unit F62 may be configured to determine the type of off-road by combining the above-mentioned various determination materials. Off-road can be broadly classified into, for example, three-dimensional roads with many bumps and dips, such as rocky roads and mogul roads, and slippery roads, such as muddy roads, sandy roads, grasslands, and snowy fields. A slippery road refers to a ground surface with relatively gentle changes in the gradient and bumps. For example, the driving environment determination unit F62 may be configured to determine the type of off-road as a three-dimensional road or a slippery road based on at least one of the image recognition results, the sonar detection results, and the set driving mode.

[0124] The projection plane control unit F63 may change the shape pattern of the projection plane to be applied according to the type of off-road determined by the driving environment determination unit F62. For example, when the driving environment is determined to be a three-dimensional road by the driving environment determination unit F62, the off-road projection plane TS2 for the three-dimensional road is applied, whereas when the driving environment is determined to be a three-dimensional road, the off-road projection plane TS2 for the slip road is applied. The off-road projection plane TS2 for the slip road may be, for example, a small inclined projection plane TS3. Also, the off-road projection plane TS2 for the three-dimensional road may be, for example, an intermediate projection plane or a steeply inclined projection plane TS4. The off-road projection plane TS2 for the three-dimensional road may have a larger inclination of the curved surface region R2a than the small inclined projection plane TS3.

[0125] According to the above-mentioned configuration, even in off-road driving, the projection plane TS is applied according to the type of the driving, so that the driver's viewpoint image CPd that is less likely to cause discomfort can be generated.

[0126] <Off-road display mode supplement> When the off-road display mode is selected, that is, when the off-road projection surface TS2 is applied as the projection surface TS, the display image generating unit F8 may place a notification image NP indicating that the off-road display mode is selected in a corner of the display image DP as shown in Fig. 23. This configuration can reduce the risk that the driver will be confused by the ground appearing differently from normal conditions.

[0127] In addition, whether or not to display the notification image NP may be changed depending on the material determined to be off-road. For example, when the driving environment is determined to be off-road based on the user's operation on the operation members such as the operation button 5, the touch panel 4, and the shift lever, the notification image NP is not displayed. On the other hand, when the driving environment is automatically determined to be off-road based on information other than the user's operation, such as the detection result of image recognition or sonar, the notification image NP may be displayed. When the mode is switched in response to the user's operation on the operation member, the user is aware that the vehicle V is operating in off-road mode, so that the annoyance can be reduced by omitting unnecessary notifications. On the other hand, when the projection surface TS is automatically switched, there is a possibility of malfunction, so that the risk of confusing the user by notifying that the mode is off-road display mode can be reduced.

[0128] <Variations of virtual viewpoints> In the above, the bird's-eye view point VPb and the driver's view point VPd are exemplified as the virtual view point VP, but the combination of the position and direction that can be set as the virtual view point VP is not limited to the above examples. As the virtual view point VP, a pattern in which the view point position is set to the left rear of the vehicle and the viewing direction is set to the front of the vehicle can be adopted. The virtual view point VP can be arranged at various positions outside the vehicle cabin. In addition, the virtual view point VP may be configured to be arranged at any position inside the vehicle cabin.

[0129] The virtual viewpoint VP may be configured to adopt a pattern in which the viewpoint position is located near the side mirror, or a setting pattern in which the viewpoint position is located at the center of the ceiling of the vehicle interior. The viewpoint control unit F64 may also be configured to set an interior rear viewpoint VPr in which the viewpoint position is a predetermined distance behind the iris and the line of sight is directed diagonally downward backward as shown in FIG. 24. According to such an interior rear viewpoint VPr, it is possible to generate and display an interior viewpoint rear image, which is a composite image CP including the vicinity of the rear wheels and the rear of the vehicle, as a composite image CP to be displayed when the vehicle is backing up. As a result, the driver can easily recognize the state of the vicinity of the rear wheels and the rear bumper when backing up, as well as when forwarding. Note that it is preferable that the interior viewpoint rear image also includes tire images corresponding to the rear wheels and a body boundary line, as in the driver viewpoint image CPd. According to this configuration, it is easy to recognize the positional relationship between each member constituting the vehicle and an object outside the vehicle interior.

[0130] <Additional remarks> The device, system, and method thereof described in the present disclosure may be realized by a dedicated computer comprising a processor programmed to execute one or more functions embodied by a computer program. The device and method described in the present disclosure may also be realized using a dedicated hardware logic circuit. The device and method described in the present disclosure may also be realized by one or more dedicated computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. The computer program may also be stored in a computer-readable non-transient tangible recording medium as instructions executed by a computer. That is, the means and / or functions provided by the processing unit 11 and the like can be provided by software recorded in a substantial memory device and a computer that executes the software, software only, hardware only, or a combination thereof. For example, some or all of the functions provided by the processing unit 11 may be realized as hardware. Aspects in which a certain function is realized as hardware include aspects in which it is realized using one or more ICs, etc. The processing unit 11 may be realized using an MPU, GPU, or DFP (Data Flow Processor) instead of a CPU. The processing unit 11 may be realized by combining a plurality of types of arithmetic processing devices, such as a CPU, an MPU, and a GPU. The processing unit 11 may be realized as a system-on-chip (SoC). Furthermore, the various processing units may be realized using a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The various programs may be stored in a non-transitive tangible storage medium. As a storage medium for the programs, various storage media, such as a hard-disk drive (HDD), a solid-state drive (SSD), a flash memory, and a secure digital (SD) card, may be adopted. The scope of the present disclosure includes not only the image generating device described above, but also various other forms such as a system including the image generating device as a component. For example, the scope of the present disclosure also includes a program for causing a computer to function as an image generating device, a non-transient tangible recording medium such as a semiconductor memory on which the program is recorded, and the like. [Explanation of symbols]

[0131] 1 Image generation ECU (image generation device), 2 Camera, 3 Display, 8 Sonar (obstacle sensor), 11 Processing unit, 52 Driving mode switch, CP Synthetic image, CPd Driver viewpoint image, DP Display image, NP Notification image, VP Virtual viewpoint, VPd Driver viewpoint, F1 Image acquisition unit, F2 Operation reception unit, F3 Vehicle state acquisition unit, F4 Obstacle information acquisition unit, F5 Image recognition unit, F6 Display control unit, F61 Travel direction acquisition unit, F62 Driving environment determination unit, F63 Projection surface control unit, F64 Viewpoint control unit, F7 Image synthesis unit, F8 Display image generation unit, F9 Image output unit, S1 Image acquisition step, S3 Driving environment determination step, S5 Off-road projection surface application step, S6 Normal projection surface application step, S8 Synthetic image generation step, R1 On-road flat area, R2 On-road curved area, R1a Off-road flat area, R2a off-road curved area

Claims

1. 1. An image generating device for use in a vehicle, comprising: An image acquisition unit (F1) that acquires a plurality of camera images obtained by a plurality of cameras that capture the surroundings of the vehicle; an image synthesis unit (F7) that projects data of the plurality of camera images onto a virtual projection plane corresponding to the periphery of the vehicle, and generates a synthetic image showing the periphery of the vehicle as seen from a virtual viewpoint using the data on the projection plane; A driving environment determination unit (F62) that determines whether the driving environment of the vehicle is off-road or on-road based on a signal from another device mounted on the vehicle, the image synthesis unit is configured to change a shape of the projection surface used for generating the synthetic image depending on whether the driving environment is determined to be off-road by the driving environment determination unit, The image synthesis unit includes: When the driving environment is determined to be on-road by the driving environment determination unit, the synthetic image is generated using an on-road projection surface having a predetermined shape as the projection surface; When the driving environment is determined to be off-road by the driving environment determination unit, the synthetic image is generated using an off-road projection surface having a shape different from that of the on-road projection surface as the projection surface, The on-road projection surface has a flat on-road planar area (R1) arranged around a vehicle area (R0) in which the vehicle is located, and a curved on-road curved area (R2) arranged around the on-road planar area, The off-road projection surface has a curved off-road curved area (R2a) arranged outside the vehicle area, The off-road curved surface area (R2a) has a different shape from the on-road curved surface area (R2), The on-road curved surface area has a parabolic shape expressed by Z = a1 × x^2, where x is a distance from a curved surface start position, which is a position where the on-road curved surface area starts outside the vehicle area, toward the outside of the vehicle area, and Z is a height of the on-road curved surface area. The off-road curved area has a parabolic shape expressed by Z = a2 × x^2, where x is a distance from a curved surface start position, which is a position where the off-road curved area starts outside the vehicle area, toward the outside of the vehicle area, and Z is a height of the off-road curved area. An image generating device in which coefficient a2 is set to a value greater than coefficient a1.

2. 2. The image generating device according to claim 1, An image generating device in which the off-road curved area (R2a) on the off-road projection surface is arranged outside the vehicle area so as to connect to an edge of the vehicle area.

3. 2. The image generating device according to claim 1, The off-road projection surface has a planar off-road planar area (R1a) arranged around the vehicle area (R0), The off-road curved surface area (R2a) is disposed outside the off-road flat surface area, The off-road planar area is set smaller than the on-road planar area.

4. 4. The image generating device according to claim 3, In the on-road projection surface, a width of the on-road flat area formed between the vehicle area and the on-road curved area is set to 0.3 m or more, An image generating device, wherein a width of the off-road flat area formed between the vehicle area and the off-road curved area on the off-road projection surface is set to less than 0.3 m.

5. 5. An image generating device used in connection with an obstacle sensor (8) for detecting a three-dimensional object present around the vehicle according to claim 1, an obstacle information acquisition unit (F4) for acquiring a signal indicating a detection result of the obstacle sensor; The image generating device, wherein the driving environment determination unit is configured to determine that the driving environment is off-road based on the obstacle sensor detecting a plurality of three-dimensional objects around the vehicle.

6. 6. An image generating device according to claim 1, An image recognition unit (F5) that detects a predetermined detection target by analyzing the camera image, The image generating device is configured such that the driving environment determination unit determines that the driving environment is off-road based on at least one of the following: a plurality of rocks are detected by the image recognition unit; lane markings are not detected by the image recognition unit; and a road edge is not detected by the image recognition unit.

7. 7. The image generating device according to claim 1, which is used in a vehicle having an off-road mode as a driving mode for driving on an off-road road, The image generating device is configured so that the driving environment determination unit determines whether or not the driving environment is off-road based on a signal from an input device mounted on the vehicle for switching the driving mode to the off-road mode.

8. 8. An image generating device according to claim 1, The image synthesis unit is configured to selectively use a plurality of projection surfaces each having a different shape as the off-road projection surface. the driving environment determination unit is configured to, when determining that the driving environment is off-road, determine a type of the off-road based on a signal from the other device; The image generating device is configured such that the image synthesis unit changes the projection surface used to generate the synthetic image in accordance with the type of off-road environment determined by the driving environment determination unit.

9. 9. An image generating device according to claim 1, The image generating device is configured to, when the image synthesis unit generates the synthetic image using the projection surface for off-road use, display on a display device a notification image (NP) indicating that an image generated using off-road settings is being displayed.

10. An image generation method for generating an image for assisting a driving operation of a vehicle, comprising: Obtaining a plurality of camera images obtained by a plurality of cameras photographing the surroundings of the vehicle (S1); projecting the data of the plurality of camera images onto a virtual projection plane corresponding to the periphery of the vehicle, and generating a composite image showing the periphery of the vehicle as seen from a virtual viewpoint using the data on the projection plane (S8); Determining whether the vehicle is traveling in an off-road or on-road environment based on a signal from another device mounted on the vehicle (S3); When the driving environment is determined to be on-road, generating the synthetic image using an on-road projection surface having a predetermined shape as the projection surface; When the driving environment is determined to be off-road, generating the synthetic image using an off-road projection surface having a shape different from that of the on-road projection surface as the projection surface, The on-road projection surface has a flat on-road planar area (R1) arranged around a vehicle area (R0) in which the vehicle is located, and a curved on-road curved area (R2) arranged around the on-road planar area, The off-road projection surface has a curved off-road curved area (R2a) arranged outside the vehicle area, The off-road curved surface area (R2a) has a different shape from the on-road curved surface area (R2), The on-road curved surface area has a parabolic shape expressed by Z = a1 × x^2, where x is a distance from a curved surface start position, which is a position where the on-road curved surface area starts outside the vehicle area, toward the outside of the vehicle area, and Z is a height of the on-road curved surface area. The off-road curved area has a parabolic shape expressed by Z = a2 × x^2, where x is a distance from a curved surface start position, which is a position where the off-road curved area starts outside the vehicle area, toward the outside of the vehicle area, and Z is a height of the off-road curved area. An image generating method in which coefficient a2 is set to a value greater than coefficient a1.

Citation Information

Patent Citations

  • Switching regulator

    JP1984011775A

  • Centrifugal dehydrator for cheese

    JP1987057978A

  • Vehicle image processing device, vehicle image processing program, and vehicle image processing method

    JP2009171537A

  • Video display system, video synthesis device and video synthesis method

    JP2015015527A

  • Periphery monitoring device

    JP2018016250A