Peripheral image display device and display control method

Abstract

The present disclosure relates to a peripheral image display device and a display control method. A synthetic image forming unit forms an in-vehicle viewpoint image (PD) that shows the periphery of the vehicle in a manner that transparently passes through a portion or all of the vehicle and is taken from an in-vehicle viewpoint (Vd) in the vehicle compartment. The synthetic image forming unit also uses pre-stored data (Dt) for drawing images of internal components to form an internal added image (PDi) including an image of the internal components viewed from the in-vehicle viewpoint. When the display control unit converts the display screen of the display to the in-vehicle viewpoint image in response to the user's manipulation of the vehicle, the internal added image is temporarily displayed before the in-vehicle viewpoint image is displayed, and an animation that gradually increases the transparency of the image of the internal components is displayed.

Description

Technical Field

[0001] The present disclosure relates to a peripheral image display device and a display control method for forming and displaying a peripheral image showing an area of ​​at least a part of the periphery of a vehicle based on an image from an on-vehicle camera device. Background Art

[0002] Japanese Patent No. 6257978 discloses a technology for displaying images captured by camera devices placed at a plurality of positions on a vehicle or a composite image provided by synthesizing these captured images as a peripheral image showing the periphery of the vehicle. Such a composite image as the peripheral image includes: a top-view image showing the periphery of the vehicle in a manner of looking down on the vehicle from above, a driver's viewpoint image showing the periphery of the vehicle from the driver's viewpoint in a manner of transparently passing through the instrument panel, and the like.

[0003] By adopting a structure configured to display peripheral images taken from various viewpoints, such as the device disclosed in Japanese Patent No. 6257978, it is possible to enable a user to view a picture at a desired viewpoint, thereby improving usability. On the other hand, since peripheral images taken from increased viewpoints can be displayed, the following problem arises. That is, when the content displayed on the display has been converted into a peripheral image, the user finds it difficult to intuitively understand from which viewpoint the peripheral image being displayed was taken. Summary of the invention

[0004] The purpose of the present disclosure is to provide a peripheral image display device and a display control method so that a user can easily identify in which direction the peripheral image being displayed was taken, wherein there is a structure for displaying a composite image showing the vehicle periphery viewed from a virtual viewpoint located within the vehicle cabin.

[0005] According to a first aspect of the present disclosure, a peripheral image display device for a vehicle includes a synthetic image forming unit configured to form a peripheral image, which is a synthetic image showing an area of ​​the periphery of the vehicle viewed from an arbitrary virtual viewpoint using multiple camera images captured by multiple camera devices configured to capture the periphery of the vehicle. The peripheral image display device also includes a display control unit configured to perform processing for displaying the peripheral image formed by the synthetic image forming unit on a display. The synthetic image forming unit is configured to form an in-vehicle viewpoint image showing the periphery of the vehicle in a manner that transparently passes through a part or all of the vehicle, the in-vehicle viewpoint image being a peripheral image captured from an in-vehicle viewpoint positioned in the vehicle compartment as a virtual viewpoint. The synthetic image forming unit is also configured to form an interior addition image using data for drawing an image of an interior component, the interior addition image being a synthetic image including an image of an interior component viewed from an in-vehicle viewpoint, the data being stored in a predetermined storage medium. The display control unit is configured to: when the display screen of the display is changed to the in-vehicle viewpoint image in response to the user's manipulation of the vehicle, temporarily display the interior additional image before displaying the in-vehicle viewpoint image, and display an animation that gradually increases the transparency of the image of the interior component.

[0006] With the above-mentioned structure, when the screen is changed to the in-vehicle viewpoint image, an image including an image of an interior component is temporarily displayed, instead of suddenly displaying a peripheral image transparently showing an interior component such as an instrument panel. Then, the image of the interior component is gradually made transparent, so that the in-vehicle viewpoint image is finally displayed. By temporarily displaying the interior component as described above, the user can easily recognize in which direction the peripheral image being displayed was taken.

[0007] According to a second aspect of the present disclosure, a display control method is executed by at least one processor to control the display of a peripheral image, which is a composite image showing an area of ​​the periphery of the vehicle viewed from an arbitrary virtual viewpoint by using a plurality of camera images captured by a plurality of camera devices configured to capture the periphery of the vehicle. The display control method includes forming an in-vehicle viewpoint image as a peripheral image showing the periphery of the vehicle in a manner that transparently passes through a part or all of the vehicle, the in-vehicle viewpoint image being a peripheral image captured from an in-vehicle viewpoint positioned in the vehicle compartment as a virtual viewpoint. The display control method also includes: forming an interior addition image by using data, the interior addition image being a composite image including an image of an interior component viewed from an in-vehicle viewpoint, the data being used to draw the image of the interior component and being stored in a predetermined storage medium. The display control method also includes: displaying the formed composite image on a display. The display control method also includes: when the display screen of the display is converted to the in-vehicle viewpoint image in response to the user's manipulation of the vehicle, before displaying the in-vehicle viewpoint image, temporarily displaying the interior addition image, and thereafter displaying an animation that gradually increases the transparency of the image of the interior component in the interior addition image.

[0008] The above-mentioned display control method is a method corresponding to the above-mentioned peripheral image display device, and provides the same advantages through the same effects as those of the above-mentioned peripheral image display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the accompanying drawings:

[0010] Figure 1 is a block diagram showing an overview of the peripheral display system Sys.

[0011] Figure 2 It is a diagram showing the positions where each camera device is installed and the range captured thereby.

[0012] Figure 3 This is a block diagram for explaining the functions of the image forming ECU.

[0013] Figure 4 is a view used to illustrate the projection surface.

[0014] Figure 5 A view used to illustrate a virtual viewpoint.

[0015] Figure 6 is a flowchart illustrating an example of a process for forming a composite image.

[0016] Figure 7 is a view for explaining image data for forming an under-floor image.

[0017] Figure 8 is a view showing an example of the structure of a composite image.

[0018] Fig. 9 Detailed Description of the Invention FIG. 1 is a flowchart showing the operation of the image forming ECU for displaying a front in-vehicle image.

[0019] Fig.10 is a diagram showing an example of a moving path of a virtual viewpoint to a front in-vehicle viewpoint.

[0020] Fig.11 : is a view showing an example of transition of a display screen during a viewpoint movement phase.

[0021] Fig.12 is a view showing an example of transition of a display screen during a vehicle body transparent stage.

[0022] Fig.13 : is a view showing an example of a final front in-vehicle viewpoint image.

[0023] Fig.14 : is a diagram showing an example of a moving path of a virtual viewpoint to a rear in-vehicle viewpoint.

[0024] Fig.15 is a view for explaining an aspect of controlling the transparency of a top portion image just before a virtual viewpoint enters a vehicle compartment.

[0025] Fig.16 is a view for explaining an aspect of controlling the transparency of a top portion image just before a virtual viewpoint enters a vehicle compartment.

[0026] Fig.17 is a view showing an example of displaying an image indicating a display range at a final in-vehicle viewpoint during the viewpoint movement phase. DETAILED DESCRIPTION

[0027] Hereinafter, with reference to the accompanying drawings, an embodiment of a surrounding display system Sys according to the present disclosure will be described. The surrounding display system Sys displays an image of the surroundings of a vehicle incorporating the system on a display. Hereinafter, a vehicle Hv incorporating the surrounding display system Sys will also be referred to as the present vehicle.

[0028] <Introduction>

[0029] The vehicle according to the present embodiment is a four-wheel drive vehicle, which is estimated to be driven not only on a road having a road surface paved with asphalt or the like (i.e., on-road), but also on an off-road. The vehicle has a normal mode and an off-road mode as driving modes. The normal mode is a driving mode suitable for driving on roads. The off-road mode is a driving mode suitable for off-road driving. In each driving mode, the distribution of driving force to the front wheels, rear wheels, left wheels and right wheels is controlled in different ways. In this case, the term "off-road" mainly refers to a ground surface with large bumps, such as a rocky road. Of course, "off-road" can be understood to refer to a ground surface other than a road, i.e., a general unpaved ground surface. The present disclosure can also be applied to vehicles that are estimated not to be driven off-road. The vehicle can be a vehicle having an engine as a driving force source or an electric vehicle or a hybrid vehicle including a motor as a driving force source.

[0030] In the following description, the front, rear, left, right, top and bottom directions are each defined relative to the vehicle. More specifically, the front and rear directions correspond to the longitudinal direction of the vehicle. The left and right directions correspond to the lateral direction of the vehicle. The top and bottom directions correspond to the height direction of the vehicle. From another perspective, the top and bottom directions correspond to directions perpendicular to planes parallel to the front and rear directions and the left and right directions. In the present disclosure, the plane perpendicular to the height direction of the vehicle is also referred to as the vehicle horizontal plane. The directions perpendicular to the height direction of the vehicle (including the front, rear, left and right directions) are also referred to as the vehicle horizontal directions. The vehicle horizontal direction corresponds to the direction away from the vehicle.

[0031] In addition, in the present disclosure, "parallel" is not limited to a completely parallel state. "Parallel" may mean a state of being inclined by several degrees to about 20 degrees. That is, "parallel" may include a state of being roughly parallel (a so-called substantially parallel state). In the present disclosure, the expression "perpendicular" is not limited to a completely perpendicular state and may include a state of being inclined by several degrees to about 20 degrees.

[0032] <Description of the entire structure>

[0033] Figure 1 1 is a view showing an example of a schematic structure of a peripheral display system Sys according to the present disclosure. Figure 1 As shown, the peripheral display system Sys includes an image forming ECU 1, a plurality of cameras 2, a display 3, a touch panel 4, a manipulation button 5, and a vehicle condition sensor 6. In the naming of the components, the term "ECU" is an abbreviation of Electronic Control Unit, which means an electronic control device.

[0034] The image forming ECU 1 is connected to a plurality of camera devices 2, a display 3, a touch panel 4, a manipulation button 5, and a vehicle condition sensor 6 so as to communicate with each of these components. The image forming ECU 1 can be connected to the various devices or sensors mentioned above individually through a dedicated line or through an in-vehicle network as a communication network constructed in the vehicle. For example, the image forming ECU 1 can be directly connected to the camera device 2 through a dedicated picture signal line.

[0035] As follows, the image forming ECU 1 is an ECU. That is, the ECU forms a composite image of the vehicle's surrounding area viewed from an arbitrary viewpoint based on the image data input from each camera device 2 as an image for assisting the manipulation of the driving vehicle Hv. In addition, the ECU displays the composite image on the display 3. The image forming ECU 1 corresponds to a peripheral image display device. The image forming ECU 1 is implemented using a computer. That is, the image forming ECU 1 includes a processing unit 11, a RAM 12, a storage device 13, an input / output (I / O) 14, and a bus for connecting these structures to each other.

[0036] The processing unit 11 is hardware for arithmetic processing (in other words, an arithmetic core), which is connected to the RAM 12. The processing unit 11 is, for example, a CPU. The processing unit 11 performs various processes for realizing the functions of various functional parts by accessing the RAM 12, which will be described later. The RAM 12 is a volatile storage medium.

[0037] The storage device 13 is configured to include a nonvolatile storage medium such as a flash memory. The storage device 13 stores an image forming program Pg as firmware, and various image drawing data Dt for enabling the synthetic image forming unit F5 to form a synthetic image. The image drawing data Dt includes data indicating the shape of the projection surface TS to be described later, 3D model data indicating the appearance of the vehicle, 3D model data about predetermined constituent members of the vehicle, and the like. As constituent members for which 3D model data are prepared, there are various internal members such as handles, instrument panels, columns, body panels, seats, and a center console. In addition, the storage device 13 also stores 3D model data about tires as image drawing data Dt. In addition, the execution of the image forming program Pg by the processing unit 11 corresponds to the execution of a display control method, which is a method corresponding to the image forming program Pg. The I / O 14 is a circuit module for communicating with other devices. The I / O 14 is implemented using analog circuit elements, ICs, and the like. The image forming ECU 1 will be described separately in more detail later.

[0038] The camera device 2 is a vehicle-mounted camera device configured to capture the periphery of the vehicle and output the captured image data to the image forming ECU 1. Each of the camera devices 2 includes at least a lens and an image pickup element, and electronically acquires an image showing the periphery of the vehicle. A plurality of camera devices 2 are installed at predetermined positions on the vehicle in predetermined postures so as to capture respective different ranges. Figure 2 As shown, the surrounding display system Sys according to the present embodiment includes a front camera 2F, a rear camera 2B, a left camera 2L, and a right camera 2R as the camera devices 2. These four camera devices 2 are placed at various positions in the vehicle and are configured to capture the surroundings of the vehicle in various directions. More specifically, this is achieved as follows.

[0039] The front camera 2F is a camera for photographing the front of the vehicle at a predetermined angle of view. The front camera 2F is installed at the front edge of the vehicle, such as the front grille, with its optical axis 2Fa oriented in the front direction of the vehicle. The rear camera 2B is a camera for photographing the rear of the vehicle at a predetermined angle of view. The rear camera 2B is installed at a predetermined position on the rear surface portion of the vehicle body, such as near the rear number plate or near the rear window, with its optical axis 2Ba oriented behind the vehicle. The left camera 2L is a camera for photographing the left side of the vehicle. The left camera 2L is installed on the left mirror with its optical axis 2La oriented to the left of the vehicle. The right camera 2R is a camera for photographing the right side of the vehicle. The right camera 2R is installed on the right mirror with its optical axis 2Ra oriented to the right of the vehicle. Each camera 2 can also be installed with its optical axis oriented downward by a predetermined amount, such as 10 degrees, relative to the horizontal direction of the vehicle.

[0040] As the lens in these camera devices 2, a wide-angle lens such as a fisheye lens is adopted, and each of the camera devices 2 has a viewing angle θ equal to or greater than 180 degrees. Therefore, by using four camera devices 2, the entire range around the vehicle can be photographed. In addition, the above-mentioned positions where each camera device 2 is installed can be appropriately changed. The front camera device 2F can also be installed to the upper end portion of the interior mirror or the front glass. The right camera device 2R and the left camera device 2L can also be placed near the root portion of the A-pillar and the B-pillar. The peripheral display system Sys can also include a camera device 2 installed on the top. For example, some or all of the camera devices 2 can also be cameras that are later attached to the top, on the dashboard, near the window frame, etc.

[0041] The storage device 13 stores data indicating the position and posture of each camera device 2 installed in the vehicle Hv (hereinafter, it will be referred to as "installation position data"). For example, the position where each camera device 2 is installed can be expressed as a point in a 3D coordinate system (hereinafter, referred to as a vehicle 3D coordinate system) centered at an arbitrary position in the vehicle Hv. The X-axis forming the vehicle 3D coordinate system can be, for example, an axis parallel to the left-right direction of the vehicle. In addition, the Y-axis can be an axis parallel to the front-rear direction. The Z-axis can be an axis parallel to the height direction of the vehicle. The right direction of the vehicle corresponds to the positive direction of the X-axis, the front direction of the vehicle corresponds to the positive direction of the Y-axis, and the upper direction of the vehicle corresponds to the positive direction of the Z-axis. The center of the vehicle 3D coordinate system can be, for example, the center of the rear wheel axle.

[0042] The display 3 is, for example, a device including a thin display panel such as a liquid crystal display panel, and various types of information and images are displayed thereon. The display 3 is placed in a dashboard or the like in the vehicle so that the user can view its screen. The display 3 may also be placed in the same housing as the housing of the image forming ECU 1, and thus may be integrated with the image forming ECU 1. Of course, the display 3 may also be a device separate from the image forming ECU 1. The display 3 includes a touch panel 4 stacked on the display panel, and thus is capable of receiving user manipulations thereon. The touch panel 4 is, for example, composed of a capacitive touch panel, and is configured to output a signal indicating the position touched by the user. Here, the term "user" mainly refers to an occupant in the driver's seat (the so-called driver). In addition to the driver, the term "user" may also include an occupant in the co-pilot seat.

[0043] The manipulation button 5 is a manipulation component that receives the user's manipulation of the content displayed on the display 3. The manipulation button 5 is a switch for displaying a camera image or a composite image formed by the image forming ECU 1 on the display 3 or for changing a virtual viewpoint of a composite image, etc. The manipulation button 5 is, for example, provided on the steering wheel of the vehicle, and mainly receives the driver's manipulation. The user can perform various manipulations on the peripheral display system Sys through the manipulation button 5 and the touch panel 4. If the user performs manipulation on the manipulation button 5 or the touch panel 4, a manipulation signal indicating the content of the manipulation is input to the image forming ECU 1. The manipulation button 5 may also be referred to as a multi-information switch. The manipulation button 5 may also be placed on the dashboard.

[0044] In addition, the surrounding display system Sys may further include a driving mode switch as a manipulation component for enabling the driver to change the driving mode of the vehicle Hv to an off-road mode. The manipulation component for changing the driving mode may be a dial type. The gear lever may be provided with the function of the driving mode switch.

[0045] The vehicle condition sensor 6 is a sensor for detecting the amount of the condition related to the driving control of the vehicle. The vehicle condition sensor 6 includes a gear position sensor, a vehicle speed sensor, a steering angle sensor, an acceleration sensor, and the like. The gear position sensor is a sensor for detecting the gear position. The vehicle speed sensor is a sensor for detecting the driving speed of the vehicle. The steering angle sensor is a sensor for detecting the rotation angle of the handle (the so-called steering angle). The acceleration sensor is a sensor for detecting the acceleration acting on the vehicle in at least one of the front and rear directions, the lateral direction, and the upper and lower directions of the vehicle. In this case, it is assumed that a three-axis acceleration sensor is adopted as the acceleration sensor. The value detected by the acceleration sensor can be used as a material for determining the posture of the vehicle relative to the horizontal plane. The type of sensor adopted as the vehicle condition sensor 6 in the surrounding display system Sys can be appropriately designed, and the surrounding display system Sys does not need to include all of the above-mentioned sensors. The vehicle condition sensor 6 may also include a vehicle height sensor, a gyro sensor, an azimuth sensor, and the like. Each sensor outputs data indicating the current value of the amount of the physical condition to be detected (i.e., the detection result) to the image forming ECU 1.

[0046] <About the structure of the image forming ECU>

[0047] like Figure 3 As shown, the image forming ECU 1 includes an image acquisition unit F1, a manipulation receiving unit F2, a vehicle condition acquisition unit F3, an image recognition unit F4, a synthetic image forming unit F5, a display control unit F6, and an image output unit F7 as functional parts. The image forming ECU 1 includes an image memory M1 as a memory for temporarily saving image data. The image memory M1 is implemented, for example, by using a part of a storage area included in the RAM 12. The image data provided from the respective camera devices 2 is stored in the image memory M1 in association with position information and orientation information about the vehicle Hv during shooting.

[0048] The image acquisition unit F1 acquires a camera image which is an image formed by each of the four cameras 2. The images from each camera 2 are combined with each other, thereby providing image data about the entire range around the host vehicle. The image acquisition unit F1 converts the image signal input from the camera 2 into digital image data in a predetermined data format, and then outputs it to the image recognition unit F4 and the composite image forming unit F5.

[0049] For example, each time the host vehicle moves a predetermined distance for storage, the image acquisition unit F1 stores the data of the image captured by each camera device 2 at that point in time in association with the position information and body posture information about the host vehicle acquired separately in the image memory M1. The distance for storage may be, for example, 0.1 m, 0.3 m, 0.5 m, etc. For example, during the forward travel of the vehicle, the image acquisition unit F1 only needs to store in the image memory M1 the image data captured by the front camera device 2F that is photographing at least the ground surface from just below the front edge of the vehicle to 3 m in front of it. During the rearward travel of the vehicle, the image acquisition unit F1 only needs to store in the image memory M1 the image data captured by the rear camera device 2B that is photographing at least the ground surface from just below the rear edge of the vehicle to 3 m in front of it.

[0050] As an aspect of storing the image data in the image memory M1, various aspects can be adopted. For example, the image acquisition unit F1 overwrites and stores new data in the oldest updated area in the image memory M1. In addition, the image acquisition unit F1 can store the image frames acquired in the most recent predetermined time period in an area different from the area of ​​the image data captured for each constant distance described above regardless of the moving distance. The image data stored in the image memory M1 can be referenced by the image recognition unit F4, the synthetic image forming unit F5, the display control unit F6, etc. The image acquisition unit F1 can perform image processing corresponding to the lens characteristics such as distortion correction, enlargement and reduction, and shearing on the image acquired from the camera 2 before storing it in the image memory M1.

[0051] The manipulation receiving unit F2 receives the manipulation signal output from the manipulation button 5 and the touch panel 4. The manipulation signal is a signal indicating the content of the manipulation performed by the user on the manipulation button 5 and the touch panel 4. Therefore, the manipulation receiving unit F2 receives the user's command manipulation regarding the display of the composite image and the camera image. The manipulation receiving unit F2 inputs data corresponding to the received manipulation signal to the display control unit F6.

[0052] The vehicle condition acquisition unit F3 is configured to acquire information indicating the condition of the vehicle from other devices (e.g., the vehicle condition sensor 6) provided in the vehicle. The vehicle condition acquisition unit F3 acquires the gear position, the acceleration in the direction of each detection axis, the vehicle speed, the steering angle, etc. from various sensors. In addition, the vehicle condition acquisition unit F3 can acquire the amount of various conditions from other ECUs. For example, the vehicle condition acquisition unit F3 can acquire information about the steering angle, etc. from the ECU constituting the steering system. In addition, the vehicle condition acquisition unit F3 can acquire the distance that the vehicle has moved relative to the previous predetermined time period, the amount of change in the orientation of the vehicle, etc. from the image recognition unit F4. That is, the vehicle condition acquisition unit F3 can acquire various types of information related to the condition of the vehicle from other ECUs and the image recognition unit F4 and from sensors. The vehicle condition acquisition unit F3 can be configured to calculate the vehicle moving distance, the amount of change in the orientation of the vehicle body, etc. by combining multiple types of information. The "orientation of the vehicle" can include not only the direction in which the vehicle is oriented (the so-called yaw angle), but also the pitch angle and the roll angle.

[0053] The image recognition unit F4 is configured to detect the position of a predetermined object to be detected, its type, etc. by analyzing the image input from the camera 2. The image recognition unit F4 has a function as an identifier for identifying the type of an object based on, for example, a feature vector in an image. The image recognition unit F4 uses, for example, CNN (convolutional neural network) technology and DNN (deep neural network) technology using deep learning to recognize objects. In addition to pedestrians and other vehicles, such objects to be detected also include road markings, such as dividing lines that can be set on paved roads, road edges, curbs, etc. The dividing lines can include lines as lane markings, and lines indicating parking boxes.

[0054] The image recognition unit F4 may also be configured to estimate the amount of movement of the vehicle and the amount of change in its orientation based on the amount of position change of feature points appropriately determined on the image between image frames. For example, the image recognition unit F4 is configured to estimate the current position, amount of movement, amount of change in the orientation of the vehicle body, yaw rate, etc. relative to a specific reference point using an optical flow method, which is a type of inter-frame difference method. Optical flow is information that indicates the movement of an object seen in image data by a vector. The result of object recognition performed by the image recognition unit F4 and the result of estimation of the position of the vehicle performed by the image recognition unit F4 are output to the vehicle condition acquisition unit F3, the display control unit F6, etc.

[0055] The synthetic image forming unit F5 is configured to perform, for example, image processing for forming a synthetic image. The synthetic image forming unit F5 projects the data of the plurality of camera images on a virtual projection surface TS corresponding to the periphery of the host vehicle. Furthermore, the synthetic image forming unit F5 forms a synthetic image showing the periphery of the host vehicle viewed from the virtual viewpoint VP using the data on the projection surface.

[0056] like Figure 4 Conceptually shown, the projection surface TS is a virtual three-dimensional surface corresponding to the peripheral area around the vehicle. The central area of ​​the projection surface TS is defined as the vehicle area R0 as the position of the vehicle. The vehicle area R0 is set to a rectangular flat portion, for example. The vehicle area R0 is set to include an area overlapping with the vehicle in a top view, that is, an area that cannot be directly photographed by the camera device 2. The projection surface TS has a bowl-like shape with a curved surface convex downward, wherein the projection surface TS has a flat surface around the vehicle area R0 along the horizontal direction of the vehicle, and also has a larger inclination (slope) at a greater distance from the vehicle area R0. According to corresponding information such as table data, the various parts of the projection surface TS are associated with corresponding parts of the camera image to be projected onto these various parts of the projection surface TS. Projecting the camera image onto the projection surface TS corresponds to performing texture mapping of the camera image onto the projection surface TS.

[0057] The shape of the projection surface TS can be changed as needed. For example, the projection surface TS can also have a horizontal surface over its entire area. In addition, the projection surface TS can also have a shape having a curved surface area convex downward from the edge portion of the vehicle area R0 without having a flat surface area.

[0058] The display control unit F6 controls the operation of the synthetic image forming unit F5. For example, the display control unit F6 controls the virtual viewpoint VP used to form the synthetic image. The synthetic image forming unit F5 includes a camera image synthesis unit F51 and a vehicle image providing unit F52 as sub-functions. The camera image synthesis unit F51 is configured to project each camera image on the above-mentioned projection surface TS. The vehicle image providing unit F52 is configured to place the image of the vehicle Hv at a predetermined position in the projection surface TS on which the camera image is projected. The camera image synthesis unit F51 and the vehicle image providing unit F52 will be described separately in more detail later.

[0059] The display control unit F6 comprehensively controls the entirety of the image forming ECU 1, and also controls the image displayed on the display 3. When a predetermined peripheral image display condition is satisfied, the display control unit F6 outputs at least one of the camera image and the synthetic image formed by the synthetic image forming unit F5 to the display 3. The camera image is displayed on the display 3 in a state of being subjected to distortion correction and the like for display. The display control unit F6 determines the combination of images displayed on the display 3 according to the traveling direction of the host vehicle and the manipulation of the touch panel 4 by the user and the like. The display control unit F6 displays an image corresponding to the operation mode designated by the user on the display 3. Thus, a peripheral image of an area including at least a part of the periphery of the vehicle is displayed on the display 3 as a photographic subject image.

[0060] In this specification, the term "peripheral image" refers to an image showing an area of ​​at least a part of the periphery of the vehicle, and conceptually includes a composite image and a camera image adjusted for display. Such a camera image adjusted for display forms a peripheral image captured from a viewpoint at the position of the lens of the camera 2 that has acquired the camera image. Such a composite image forms a peripheral image captured from a virtual viewpoint VP set at an arbitrary position of the periphery of the vehicle.

[0061] The display control unit F6 has a function of controlling the synthetic image forming unit F5 based on the information input from the manipulation receiving unit F2 and the vehicle condition acquiring unit F3, so that the synthetic image forming unit F5 forms a synthetic image according to the driving condition of the vehicle and the settings made by the user. For example, the display control unit F6 sets the position and sight line direction of the virtual viewpoint VP used to form the synthetic image based on at least one of the driving direction of the vehicle, the signal from the touch panel 4, and the signal from the manipulation button 5, which will be described later. As the settable modes of the virtual viewpoint VP, such as Figure 5 As shown, a rear bird's-eye view point Vbb, a front bird's-eye view point Vbf, a front in-vehicle view point Vdf, a rear in-vehicle view point Vdb, etc. may be used.

[0062] The rear bird's-eye viewpoint Vbb corresponds to a virtual viewpoint VP that looks down on the vehicle Hv and the rear thereof at a point located above the top portion of the vehicle Hv and in front of the vehicle. The front bird's-eye viewpoint Vbf corresponds to a virtual viewpoint VP that looks down on the vehicle Hv and the front thereof at a point located above the top portion of the vehicle Hv and in the rear of the vehicle. The front bird's-eye viewpoint Vbf corresponds to a first bird's-eye viewpoint, and the rear bird's-eye viewpoint Vbb corresponds to a second bird's-eye viewpoint.

[0063] The bird's-eye viewpoint Vb is not limited to the examples mentioned above, and a mode of a bird's-eye viewpoint Vb set at a position directly above the vehicle or laterally deviated from directly above the vehicle may also be adopted. Bird's-eye viewpoints Vb such as the rear bird's-eye viewpoint Vbb and the front bird's-eye viewpoint Vbf correspond to examples of vehicle exterior viewpoints, which are virtual viewpoints VP placed outside the vehicle compartment. In the case of forming a bird's-eye view image that is an image of the host vehicle and its surroundings viewed from above the vehicle, various bird's-eye viewpoints Vb may be adopted. The viewing angle of the bird's-eye viewpoint Vb is appropriately adjusted so as to include a predetermined range of the surroundings of the vehicle.

[0064] The front in-vehicle viewpoint Vdf is a configurable mode of the virtual viewpoint VP, in which the position of the viewpoint is set at the estimated position of the driver's eyes in the vehicle compartment. The line of sight of the front in-vehicle viewpoint Vdf can be set, for example, to the forward and obliquely downward direction so as to include the vicinity of the front wheel. The "obliquely downward direction" can be assumed to be, for example, a direction about 20 to 30 degrees downward relative to the horizontal plane of the vehicle. The front in-vehicle viewpoint Vdf can also be referred to as the driver's viewpoint.

[0065] Based on the user's manipulation (e.g., sliding) on ​​the touch panel 4, the sight line direction of the front in-vehicle viewpoint Vdf can be changed to any direction within a predetermined left and right angle range, and the forward and obliquely downward direction is defined as a default direction. The viewing angle of the front in-vehicle viewpoint Vdf can also be adjusted as needed to include a predetermined range near the front wheel. For example, the viewing angle of the front in-vehicle viewpoint Vdf can be changed to any direction within a predetermined left and right angle range. It is 100 degrees horizontally and 60 degrees vertically.

[0066] The estimated position of the driver's eyes is, for example, set near the headrest of the driver's seat. The estimated position of the driver's eyes can also be set at the center of the eye ellipse defined for each type of vehicle. Such an eye ellipse is a virtual space area defined for each type of vehicle. Based on the eye range of the special distribution of the occupant's eye points that is statically expressed, the eye ellipse is set to a virtual ellipse shape. The front in-car viewpoint Vdf corresponds to an example of the in-car viewpoint Vd, which is a virtual viewpoint VP placed in the car compartment. The position of the front in-car viewpoint Vdf can also be placed at a position that deviates from the estimated position of the driver's eyes. For example, the front in-car viewpoint Vdf can also be placed at a position that deviates from the estimated position of the driver's eyes toward the co-pilot seat by a predetermined amount, such as at a position between the driver's seat and the co-pilot seat.

[0067] The rear in-vehicle viewpoint Vdb is a virtual viewpoint VP whose position is at a predetermined distance behind the eye ellipse and whose sight line is directed rearward and obliquely downward. For example, during the rearward travel of the vehicle, when displaying the road surface near the rear wheels of the vehicle, the rear in-vehicle viewpoint Vdb can be adopted. With the rear in-vehicle viewpoint Vdb, the vicinity of the rear wheels can be displayed more greatly. As a result, similar to during the forward travel, during the rearward travel, the driver can easily recognize the state of the vicinity of the rear bumper and the rear wheels. The rear in-vehicle viewpoint Vdb also corresponds to the example of the in-vehicle viewpoint Vd. The position of the rear in-vehicle viewpoint Vdb itself can be the same as that of the front in-vehicle viewpoint Vdf. The front in-vehicle viewpoint Vdf corresponds to the first in-vehicle viewpoint, and the rear in-vehicle viewpoint Vdb corresponds to the second in-vehicle viewpoint.

[0068] In addition, the virtual viewpoint VP can be set at various positions in the vehicle compartment. For example, as the virtual viewpoint VP, a mode in which the viewpoint is placed near the side mirror or a setting mode in which the viewpoint is placed at the center of the ceiling portion in the vehicle compartment can be adopted.

[0069] In addition, for example, the display control unit F6 determines whether the vehicle is traveling forward or backward based on a signal from a shift position sensor or the rotation direction of a tire. The display control unit F6 can also determine whether the driving environment is off-road based on an input signal from a driving mode switch.

[0070] The image output unit F7 converts the image input from the display control unit F6 into an image signal in a predetermined signal format and then outputs the image signal to the display 3, thereby causing the display 3 to display the image. Thus, a synthetic image showing the periphery of the host vehicle viewed from the virtual viewpoint VP is displayed on the display 3.

[0071] <About Method for Forming Composite Image>

[0072] In the following, reference will be made to Figure 6 The flowchart shown in 1 describes an example of the operation of the composite image forming unit F5 for forming a composite image. In this case, the process for forming a composite image generally includes steps S101 to S106, but the process is not limited thereto. The number of steps included in the process for forming a composite image and the order of the processes therein may be changed appropriately. The process may be continuously performed while shooting is performed with each camera 2 or while a mode for displaying a composite image is set. Figure 6 For example, the process shown in FIG. 1 can be repeated at a cycle of 1 / 30 seconds. Figure 6In the case where a process is repeatedly executed, the process can be executed in such a manner that a portion of the result of the previous process that can be reused therein is reused, thereby appropriately changing the process to reduce the processing load.

[0073] First, in step S101, the camera image synthesis unit F51 acquires data (values ​​of respective pixels) included in respective camera images input from the image acquisition unit F1, and also projects it onto the projection surface TS in the virtual 3D space. The respective camera images and the positions where the respective camera images are projected onto the projection surface TS have been previously associated with each other by corresponding information such as table data. The corresponding information such as table data corresponds to data indicating at which positions on the projection surface TS the respective pixels included in the respective camera images are projected.

[0074] For example, the camera image synthesis unit F51 projects the image data from the front camera 2F onto the front area tsF in the projection surface TS, and also projects the image data from the rear camera 2B onto the rear area tsB ​​in the projection surface TS. The camera image synthesis unit F51 projects the image data from the left camera 2L onto the left area tsL in the projection surface TS, and also projects the image data from the right camera 2R onto the right area tsR. For the overlapping area which is an area redundantly photographed by two cameras, a method of mixing the images captured by the two cameras at a predetermined ratio, or a method of coupling the images captured by the two cameras to each other on a predetermined boundary line may be adopted.

[0075] Next, in step S102, the camera image synthesis unit F51 extracts an image of a portion where the estimated vehicle region R0 is located from the past image data stored in the image memory M1. In addition, the camera image synthesis unit F51 performs rotation correction and the like on the extracted image, and then projects it onto the vehicle region R0. Since the image to be projected onto the vehicle region R0 is an image of an area currently located directly below the host vehicle, that is, an image of an area below the floor, hereinafter, this image will also be referred to as an under-floor image.

[0076] For example, when the vehicle is moving forward, Figure 7As shown, an image under the floor is formed using an image captured by the front camera 2F at 3m behind the current position. When the vehicle is traveling rearward, an image under the floor is formed using an image captured by the rear camera 2F at 3m in front of the current position. That is, an image under the floor is formed using an image when the vehicle is located at a predetermined distance from the current position in a direction opposite to the traveling direction. When the vehicle is traveling forward, an image under the floor can be formed using front camera images captured at multiple time points from when the vehicle is located 3m behind the current position until when the vehicle reaches the current position. The same is applicable when the vehicle is traveling rearward. The image data projected on the vehicle area R0 (in other words, the portion under the floor) can be updated at any time as the vehicle Hv travels.

[0077] The synthetic image showing the projection surface TS viewed from the virtual viewpoint VP formed in step S101 or step S102 corresponds to an external world image, which is an image of the external world of the vehicle Hv viewed from the virtual viewpoint VP. The external world image corresponds to an image excluding the guide lines and the image of the vehicle Hv, which will be described later, in the synthetic image as the peripheral image.

[0078] In step S103, the vehicle image providing unit F52 places various types of CG (computer graphics) indicating the existence range of the vehicle, etc. at a predetermined position in the 3D space including the projection surface TS. CG can also be called CGI (computer generated image). For example, in step S103, the vehicle image providing unit F52 places an opaque floor image Pf indicating the area where the bottom part of the vehicle exists above the vehicle area R0. The floor image Pf can be placed in a manner that the floor image Pf is just superimposed on the image under the floor, and the floor image is an image that masks the image under the floor. In addition, in step S103, a vehicle boundary line Lvc indicating the road surface range where the vehicle body (actually, the bottom part of the vehicle) exists and a tire boundary line Lt indicating the road surface range where the tire exists are placed on the vehicle area R0. The vehicle boundary line Lvc is a line indicating the outline of the area of ​​the vertical projection of the vehicle body on the road surface, and the vehicle boundary line Lvc corresponds to a line indicating the external shape of the vehicle Hv from a bird's-eye view point. The tire boundary line Lt is also a line indicating the outline of the area of ​​the vertical projection (ie, orthogonal projection) of the tire on the road surface, which corresponds to a line indicating the outline of the tire from a bird's-eye view point.

[0079] The color of the vehicle boundary line Lvc can be appropriately changed, and can be set to blue, for example, so that the vehicle boundary line Lvc can be easily distinguished from the actual color of the road surface. The color of the vehicle boundary line Lvc is set in a manner to improve the visibility of the vehicle boundary line Lvc to the driver in consideration of the colors of various road surfaces. The color of the tire boundary line Lt is set to a color different from the color of the vehicle boundary line Lvc. For example, the color of the tire boundary line Lt can be set to yellow or orange.

[0080] In addition, in step S103, the vehicle image providing unit F52 can also draw an image of a rectangular shape indicating the ground contact portion of the tire or the contour line of the ground contact portion of the tire as a ground contact area image, as other image elements related to the vehicle Hv. The vehicle image providing unit F52 can also draw an expected track line according to the steering angle and the rotation angle of the vehicle Hv as a line indicating the expected driving track. The vehicle image providing unit F52 can also draw a scale line indicating a ground point at a predetermined distance from the end portion of the vehicle and a vehicle width line indicating the width of the vehicle. The vehicle boundary line Lvc and the tire boundary line Lt are images indicating information about the vehicle Hv, and therefore can also be referred to as a vehicle information image. The expected track line, the scale line and the vehicle width line can also be used as information to guide the driver's driving manipulation. Therefore, the display line such as the expected track line can also be referred to as a guide line or an auxiliary line. Such a guide line can include the vehicle boundary line Lvc and the tire boundary line Lt. This step S103 can also be referred to as a drawing step.

[0081] In step S104, the vehicle image providing unit F52 places various CGs indicating the appearance and interior components of the vehicle at predetermined positions in the 3D space including the projection surface TS. More specifically, the vehicle image providing unit F52 places various 3D models about the vehicle in a virtual 3D space including the projection surface TS, on which the camera image has been projected. For example, 3D models of interior components such as dashboards, handles, interior mirrors, seats, and pillars are placed at predetermined positions in the 3D space including the projection surface TS. In addition, 3D models indicating the appearance of the vehicle such as tires and a body are also placed. Such 3D models of vehicles and individual constituent components may also be referred to as objects or stereoscopic images. The display control unit F6 can change the transparency and placement / non-placement of individual 3D objects.

[0082] In step S105, the synthetic image forming unit F5 sets the virtual viewpoint VP in the 3D space including the projection surface TS by the control of the display control unit F6. The synthetic image forming unit F5 can set the virtual viewpoint VP oriented in any field of view direction at any viewpoint position in the 3D space. Step S105 can also be performed before step S101 or step S104.

[0083] In step S106, the synthetic image forming unit F5 performs rendering corresponding to the virtual viewpoint VP on various types of image elements existing in the 3D space including the projection surface TS, and also cuts off image elements included in a predetermined viewing angle when viewed from the virtual viewpoint VP. Thus, the synthetic image forming unit F5 forms a synthetic image as a peripheral image viewed from the virtual viewpoint VP.

[0084] For example, when the front bird's-eye viewpoint Vbf is set as the virtual viewpoint VP, a synthetic image showing the vehicle and the surrounding area around the vehicle is formed so that the vehicle is looked down from the sky above and from the oblique rear of the vehicle. When the front in-vehicle viewpoint Vdf is set as the virtual viewpoint VP, the synthetic image is formed as the front in-vehicle viewpoint image PDf so that the synthetic image shows the ground surface near the front wheels in a manner that transparently passes through the bottom part, the interior, and the body of the vehicle Hv. When the rear in-vehicle viewpoint Vdb is set as the virtual viewpoint VP, a rear in-vehicle viewpoint image PDb is formed, wherein the rear in-vehicle viewpoint image PDb is a synthetic image that shows the ground surface near the rear wheels in a manner that transparently passes through the bottom part of the vehicle. Such synthetic images that show the road surface by transparently passing through the bottom part of the vehicle body, such as the front in-vehicle viewpoint image PDf and the rear in-vehicle viewpoint image PDb, may be referred to as under-floor transparent images or under-floor recognition images.

[0085] Figure 6 S106A is shown, which indicates step S106 in the case of forming an in-vehicle viewpoint image PD such as a front in-vehicle viewpoint image PDf and a rear in-vehicle viewpoint image PDb. Step S106A can be called an in-vehicle viewpoint image forming step. The in-vehicle viewpoint image PD is a synthetic image that shows a road surface area in a blind spot behind an interior member when viewed from the in-vehicle viewpoint in a manner that transparently passes through the interior member. The in-vehicle viewpoint image PD can be a synthetic image including an opaque floor image Pf or an under-floor transparent image including a transparent floor image Pf.

[0086] The synthetic image forming unit F5 is configured to form a synthetic image showing the internal components in an opaque state or a semi-transparent state under the control of the display control unit F6. For example, the synthetic image forming unit F5 is configured to form a synthetic image showing the image of the internal components in an opaque or semi-transparent state in the in-vehicle viewpoint image PD as an internal added image PDi. Such an internal added image PDi corresponds to an image formed by superimposing the image of the internal components on the in-vehicle viewpoint image PD. The internal added image PDi includes a front internal added image PDif and a rear internal added image, the front internal added image PDif including an internal image taken from the front in-vehicle viewpoint Vdf, and the rear internal added image including an internal image taken from the rear in-vehicle viewpoint Vdb. For example, various types of internal added images PDi can be formed by adjusting the transparency of the 3D models of various internal components drawn in step S104.

[0087] The above-mentioned method for forming a composite image is an example, and when actually forming a composite image, it is not necessary to determine the values ​​of all pixels in the projection surface TS. For example, it is also possible to form a composite image using only the pixel values ​​in the desired range corresponding to the set virtual viewpoint VP in the images of multiple camera devices. The combination of 3D models placed therein can also be changed according to the position of the virtual viewpoint VP. For example, when the virtual viewpoint VP exists outside the vehicle compartment, the drawing of the 3D model of the internal components can be removed. When the virtual viewpoint VP is placed in the vehicle compartment, the drawing of the 3D model of the vehicle body, etc., which forms the outer surface of the vehicle, can be removed. When the front in-vehicle viewpoint Vdf with the line of sight directed to the front of the vehicle is set as the virtual viewpoint VP, the drawing of the image of the internal components behind the driver's seat, such as the rear seat, can also be removed. By removing the drawing of images outside the display range, the processing speed can be increased or the processing load can be reduced.

[0088] As an example, an aspect of forming a composite image in such a sequence that a camera image is projected onto the projection surface TS and various types of CGs about the vehicle Hv are thereafter added thereto has been described. However, the respective processes may also be performed in parallel. Various methods may be adopted as a method for forming a composite image. It is also possible to place an image of the outside world formed by a combination of a plurality of camera images and an image of a stereoscopic model of components constituting the vehicle in respective different layers, and to perform processing on them separately.

[0089] For example, Figure 8As shown, a composite image can be formed by combining the camera image layer L1, the floor image Pf layer L2, the guide line layer L3, and the vehicle image layer L4 with each other. The camera image layer L1 is a layer for drawing thereon an external world image, which is an image of the projection surface TS viewed from the virtual viewpoint VP, on which the image data stored in the image memory M1 and the camera image have been projected. The external world image includes an image under the floor. The floor image Pf layer L2 is a layer for placing an opaque floor image Pf thereon. The guide line layer L3 is a layer for drawing thereon a guide line such as a vehicle boundary line Lvc.

[0090] The vehicle image layer L4 is a layer on which the CG of the vehicle appearance or various constituent members viewed from the virtual viewpoint VP is drawn. The CG of the tire may be provided separately from the layer on which the CG of the internal components is placed. In this case, the layer on which the CG of the tire is drawn may be placed between the guide line layer L3 and the layer on which the CG of the internal components is placed. For convenience, the CG of the vehicle appearance and the CG of various constituent members are collectively designated as vehicle CG. The vehicle CG conceptually includes the CG of internal components such as an instrument panel, CG of tires, etc., and CG indicating the vehicle appearance. The CG of the internal components is also referred to as an internal image. The CG of the tire is also referred to as a tire image.

[0091] In the case of forming a composite image by integrating multiple layers as described above, the camera image layer L1 can be set as the lowermost (background side) layer, and the floor image Pf layer L2, the guide line layer L3 and the vehicle image layer L4 can be superimposed thereon in the order mentioned. Of course, the layer structure can also be changed as needed. In the case of using multiple layers to process each image element separately, the various layers are integrated to form an image, which ultimately forms a composite image. The transparency can be adjusted for each layer or for each image object. For example, by adjusting the transparency of the vehicle image layer L4, the front internal added image PDif can be changed to the front in-vehicle viewpoint image PDf.

[0092] <About transitions between operation modes>

[0093] The image forming ECU 1 has operation modes corresponding to the respective virtual viewpoints VP as operation modes. That is, the image forming ECU 1 has a front in-vehicle viewpoint mode, a rear in-vehicle viewpoint mode, a front bird's-eye viewpoint mode, and a rear bird's-eye viewpoint mode, etc. The front in-vehicle viewpoint mode is an operation mode for displaying the front in-vehicle viewpoint image PDf as a synthetic image shot from the front in-vehicle viewpoint Vdf. The rear in-vehicle viewpoint mode is an operation mode for displaying the rear in-vehicle viewpoint image as a synthetic image shot from the rear in-vehicle viewpoint Vdb. The front bird's-eye viewpoint mode is an operation mode for displaying the front bird's-eye viewpoint image PBf as a synthetic image shot from the front bird's-eye viewpoint Vbf. The rear bird's-eye viewpoint mode is an operation mode for displaying the rear bird's-eye viewpoint image as a synthetic image shot from the rear bird's-eye viewpoint Vbb. Operation modes for displaying synthetic images shot from in-vehicle viewpoints (such as the front in-vehicle viewpoint mode and the rear in-vehicle viewpoint mode) are referred to as in-vehicle viewpoint modes. Operation modes for displaying a synthetic image captured from a virtual viewpoint VP placed outside the vehicle compartment, such as a front bird's-eye viewpoint mode and a rear bird's-eye viewpoint mode, are also referred to as an exterior viewpoint mode. The bird's-eye viewpoint mode is included in the exterior viewpoint mode.

[0094] Image forming ECU 1 also has operation modes for displaying an image from each camera 2 as a main image on display 3, such as a front camera mode for displaying an image from front camera 2F, and a rear camera mode for displaying a rear camera image.

[0095] The image forming ECU 1 has a standby mode for not displaying the surrounding images in addition to the operation modes corresponding to the respective virtual viewpoints VP and the respective camera devices 2. When the image forming ECU 1 is in the standby mode, an image other than the surrounding images, such as a map image formed by the navigation system or an image formed by the audio system, is displayed on the display 3. The standby mode corresponds to a mode for not outputting a picture signal corresponding to the surrounding images to the display 3. For convenience, the operation mode other than the standby mode (i.e., the mode for displaying the surrounding images) is also referred to as the surrounding image display mode.

[0096] The transition between these operation modes is performed by control by the display control unit F6 according to the user's manipulation of the touch panel 4 and the manipulation button 5 and according to the driving condition of the vehicle. Based on the user's manipulation of the manipulation components such as the shift lever and the brake pedal, the driving condition of the vehicle itself is also changed. The user's manipulation of the vehicle includes the manipulation of various devices incorporated in the vehicle, for example, in addition to the manipulation of the touch panel 4 and the manipulation button 5, it also includes the manipulation of the shift lever and the brake pedal.

[0097] When a predetermined peripheral image display condition is satisfied, the image forming ECU 1 shifts to the last valid operation mode (so-called last mode) among the peripheral image display modes corresponding to the traveling direction of the host vehicle. The last mode information is stored in the RAM 12, the storage device 13, or the like.

[0098] For example, when the gear position is set to the forward driving position, the last valid operating mode is activated in the front camera mode, the front bird's-eye view mode, and the front in-vehicle view mode. In this case, the forward driving position is a position for transmitting torque from the drive source in the direction of driving the vehicle forward. Such forward driving positions include "D", "S", "B", "1", "2", "L" positions, etc. For example, when the gear position is the "D" position while satisfying the peripheral image display condition, the front in-vehicle view mode is set in the standby mode.

[0099] When the gear position is set to the rearward travel position, the last effective operation mode is activated in the rear camera mode, the rear bird's-eye view mode, and the rear in-vehicle view mode. The rearward travel position is a position for transmitting torque from the drive source in a direction for causing the vehicle to travel rearward. The rearward travel position corresponds to the so-called "R" position.

[0100] The surrounding image display conditions include: the gear position is set to a position other than the parking position and the neutral position, and the vehicle speed is lower than a predetermined low-speed threshold. The low-speed threshold may be, for example, 10 km / h, 15 km / h, 20 km / h, etc. The parking position is a position for locking the transmission, and corresponds to the so-called "P" position. The neutral position is a position for interrupting the torque transmission between the input shaft and the output shaft, in other words, a position for not transmitting the torque from the drive source to the drive wheels, etc. The neutral position corresponds to the so-called "N" position.

[0101] The peripheral image display condition may be that the above-mentioned conditions regarding the gear position and the vehicle speed are satisfied, and in addition, the manipulation receiving unit F2 detects that the user has performed a manipulation for displaying the peripheral image. For example, such a user's manipulation for displaying the peripheral image includes the user pressing the manipulation button 5, the user's touch manipulation on the area where the button for displaying the peripheral image is displayed, etc. The transition between the various image display modes can be performed by pressing the manipulation button 5, etc. If the peripheral image display condition is no longer satisfied, the operating mode of the image forming ECU 1 returns to the standby mode. The image forming ECU 1 is capable of transforming the display screen of the display 3 based on output signals from various manipulation components incorporated in the vehicle, the manipulation components including the touch panel 4, the gear lever, etc.

[0102] <Animation Display>

[0103] By using a structure configured to display peripheral images taken from various viewpoints as in the present disclosure, the user can view pictures at a desired viewpoint, which improves usability. On the other hand, since peripheral images taken from increased viewpoints can be displayed, the following problem arises. That is, when the content displayed on the display has been converted into a peripheral image, the user finds it difficult to intuitively understand from which viewpoint the peripheral image being displayed is taken. In particular, in the case of an in-vehicle viewpoint image PD such as a front in-vehicle viewpoint image PDf and a rear in-vehicle viewpoint image PDb, the user cannot easily recognize in which direction the peripheral image being displayed is taken.

[0104] To cope with this, the image forming ECU 1 according to the present embodiment is configured to display an animation including a scene showing a virtual viewpoint VP that continuously moves from a predetermined starting point to a target in-vehicle viewpoint when changing from an operation mode other than the in-vehicle viewpoint mode to the in-vehicle viewpoint mode. Such an operation mode other than the in-vehicle viewpoint mode is, for example, a standby mode or a bird's-eye viewpoint mode. In this case, "animation" can be rephrased as a moving image. In the following, "target viewpoint" or "final viewpoint" will specify a viewpoint in the final display screen.

[0105] Fig. 9 1 is a flowchart showing a series of processes executed by the image forming ECU 1 as display transition processing when displaying a composite image captured from the in-vehicle viewpoint. Fig. 9 The flow chart shown in FIG. 1 shows a flow chart of the vehicle interior viewpoint mode, for example, the front vehicle interior viewpoint mode or the rear vehicle interior viewpoint mode. The display transition process according to this embodiment includes, for example, steps S201 to S210. Of course, the number of steps constituting the display transition process and the processing order therein can be changed as needed.

[0106] In general, the display transition processing according to the present embodiment can be divided into a viewpoint movement stage Fs1, a vehicle body transparency stage Fs2, and a bottom part transparency stage Fs3. The viewpoint movement stage Fs1 is a stage for moving the virtual viewpoint to the last viewpoint from the starting position defined according to the last viewpoint. The vehicle body transparency stage Fs2 is a stage for gradually increasing the transparency of the vehicle CG such as the instrument panel to a predetermined level after the movement to the last viewpoint is completed. The bottom part transparency stage Fs3 is a stage for gradually rendering the floor image Pf transparent so that it is finally completely transparent.

[0107] Each stage is set to be completed within approximately 0.5 seconds to 1.5 seconds, for example, within 1 second. Of course, the length of each stage can be set by the user. On the display 3, the images displayed in each stage are displayed as a series of animations. In this case, the "stage" can also be rephrased as a scene. In addition, the "stage" can be programmatically considered as an integration of processing. Therefore, the viewpoint movement stage Fs1, the vehicle body transparency stage Fs2 and the bottom part transparency stage Fs3 can also be referred to as viewpoint movement processing, vehicle body transparency processing and bottom part transparency processing, respectively.

[0108] Hereinafter, the operation of the image forming ECU 1 will be described by exemplifying a case where the mode is shifted from the standby mode to the front in-vehicle viewpoint mode.

[0109] First, in step S201, the display control unit F6 sets the last viewpoint as the final virtual viewpoint VP according to the operation mode, and the process goes to step S202. For example, in the case of transitioning to the front in-vehicle viewpoint mode, the front in-vehicle viewpoint Vdf is set as the last viewpoint VL. This step S201 can also be called the last viewpoint setting step.

[0110] In step S202, the display control unit F6 sets the starting viewpoint VS in the animation according to the last viewpoint VL. For example, when the last viewpoint VL is the front in-vehicle viewpoint Vdf, the front bird's-eye viewpoint Vbf is set as the starting viewpoint VS. When the last viewpoint VL is the rear in-vehicle viewpoint Vdb, the rear bird's-eye viewpoint Vbb is set as the starting viewpoint VS. After this step S202 is completed, the process goes to step S203.

[0111] In step S203, the synthetic image forming unit F5 forms a synthetic image corresponding to the starting viewpoint VS, and the display control unit F6 causes the display 3 to display the synthetic image. For example, in the case of transitioning to the front in-vehicle viewpoint mode, a front bird's-eye viewpoint image is displayed at the start of the animation. Step S203 corresponds to a step of displaying a bird's-eye view image corresponding to the starting viewpoint VS. After this step S203 is completed, the process proceeds to step S204.

[0112] Step S204 is used, for example, Fig.10The process shown is to start the process of gradually and continuously moving the virtual viewpoint VP from the front bird's-eye viewpoint Vbf as the starting viewpoint VS toward the front in-vehicle viewpoint Vdf as the final viewpoint VL. Step S205 is a step of determining whether the virtual viewpoint has moved to reach the front in-vehicle viewpoint Vdf as the final viewpoint VL. When the virtual viewpoint VP reaches the final viewpoint VL ("Yes" in S205), a synthetic image is displayed (step S206), wherein the synthetic image includes an image of internal components such as an instrument panel viewed from the front in-vehicle viewpoint Vdf as the final viewpoint VL. Steps S201 to S206 correspond to the viewpoint movement stage Fs1 and the viewpoint movement process.

[0113] During the stage of moving the virtual viewpoint VP, the synthetic image forming unit F5 continuously forms synthetic images under the control of the display control unit F6, each synthetic image showing the virtual viewpoint VP moved by an amount corresponding to a single frame toward the viewpoint as the destination of the movement. Based on the position and sight line direction of the starting viewpoint VS, the position and sight line direction of the last viewpoint VL, and the number of frames of the synthetic image for the viewpoint movement scene, the viewpoint position and sight line direction of the midway viewpoint Vm, which is the virtual viewpoint VP during the movement of the viewpoint, can be obtained by linear interpolation. As described above, the synthetic image forming unit F5 continuously forms synthetic images showing the virtual viewpoint VP gradually approaching the last viewpoint VL from the starting viewpoint VS over time. The plurality of formed synthetic images are sequentially output to the display 3 by the display control unit F6. Therefore, the plurality of synthetic images showing the virtual viewpoint VP at different positions formed by the synthetic image forming unit F5 are displayed as a continuous animation showing the viewpoint moving from the starting viewpoint VS to the last viewpoint VL.

[0114] The moving speed of the virtual viewpoint VP can be made constant. The moving speed of the virtual viewpoint VP can also be set to increase with a constant acceleration. The moving speed of the virtual viewpoint VP can also be set to decrease as the virtual viewpoint VP gets closer to the last viewpoint VL. The moving speed of the virtual viewpoint VP can also be controlled by combining the above-mentioned aspects, for example, so that the speed increases with a constant acceleration, and thereafter, the speed decreases with a constant acceleration. The display control unit F6 can control the position of the virtual viewpoint VP and aspects of its movement.

[0115] During the stage for moving the virtual viewpoint VP within the vehicle compartment, the internal image of the instrument panel and the like is drawn and displayed in an opaque state or a semi-transparent state so that the user can view these structures. That is, in a scene showing the virtual viewpoint VP existing within the vehicle compartment, the image displayed therein is an image formed by superimposing the internal image captured from the virtual viewpoint VP on the image of the external world captured from the virtual viewpoint VP. Since the internal image is displayed in an opaque state or a semi-transparent state, the user can intuitively and easily recognize the position and direction of the viewpoint.

[0116] In this case, "semi-transparency" is not limited to a state with 50% transparency, and may also include a state with 30% transparency or a state with 80% transparency. The expression "semi-transparency" may also include a level that vaguely expresses the presence of a structure. Transparency is a parameter whose value is higher, which indicates a more transparent state. A state with 100% transparency corresponds to a completely transparent state, and a state with 0% transparency corresponds to a completely opaque state. The transparency or opacity of an image element conceptually corresponds to the alpha value of a pixel. For example, the alpha value can be set to 256 levels on a scale of 0 to 255. Typically, in many cases, such an alpha value is used so that the smaller its value, the higher the transparency. In such a case, making the transparency higher corresponds to making the alpha value smaller.

[0117] It is also possible to change the combination of the displayed interior components and the respective transparency of the individual interior components according to the position of the virtual viewpoint. For example, when the midway viewpoint exists behind the front seat in the vehicle compartment, an image of the backrest portion of the front seat placed in a semi-transparent state can be displayed. The image of the backrest portion is placed so as to be superimposed on the image of the instrument panel, etc. With this structure, it is expected that the user can intuitively recognize the position of the moving virtual viewpoint.

[0118] Fig.11 1 shows the transition of the display screen during the viewpoint movement phase in the case of transitioning to the front in-vehicle viewpoint mode. Fig.11 As shown in (A) in FIG. 1 , first, a front bird's-eye viewpoint image PBf is displayed as a start viewpoint VS. Thereafter, the viewpoint gradually descends from the front bird's-eye viewpoint Vbf toward the vehicle, then enters the vehicle compartment and reaches the front in-vehicle viewpoint Vdf. Fig.11 (B) shows, for example, Fig.10 Display image at the midway viewpoint Vmn in . Fig.11 (C) in FIG. 1 shows a display image immediately after the virtual viewpoint VP reaches the front in-vehicle viewpoint Vdf. Fig.11 In the example, "t" indicates the time that has elapsed since the start of the animation display.

[0119] like Fig.11 As shown in (A) in FIG. 1 , in the front bird's-eye view image PBf, a vehicle image Pv as a CG indicating the appearance of the vehicle and an image showing its surroundings are displayed. In the figure, "Pr" refers to an image of the top portion of the vehicle Hv in the vehicle image Pv. As the virtual viewpoint VP gets closer to the vehicle, the top portion image Pr is displayed larger on the display screen. Fig.11As shown in (C) in FIG. 1 , when the virtual viewpoint VP reaches the front in-vehicle viewpoint Vdf as the final viewpoint VL, the CG of the constituent components indicating that the viewpoint is oriented in front of the vehicle is displayed so as to be superimposed on the external world image. The CG of the constituent components indicating that the viewpoint is oriented in front of the vehicle are, for example, the instrument panel image Pi, the handle image Ph, the image of the seat surface and the backrest portion of the front seat, the image of the center console, and the like.

[0120] Next, as step S207, the synthetic image forming unit F5 starts a process of gradually increasing the transparency of the vehicle CG other than the floor image Pf at a predetermined speed under the control of the display control unit F6. That is, the synthetic image forming unit F5 starts to make the interior image transparent. More specifically, the transparency of the instrument panel image Pi, the handle image Ph, etc. is gradually increased, for example, from an opaque state to a completely transparent state.

[0121] At the timing when a predetermined period of time (e.g., 0.5 seconds) has passed since the start of making the interior image transparent, the transparency of the tire image Pt is also gradually increased from 0% to a predetermined target level at a predetermined speed. As described above, by providing a time difference between the start of making the interior image transparent and the start of making the tire image Pt transparent, an animation expression can be realized that displays the tire image Pt as the interior image transparent starts and gradually makes the tire image Pt transparent thereafter. As described above, with the structure that provides a time difference between the start of making the interior image transparent and the start of making the tire image Pt transparent, it can be expected to give the user a visual impression that a component closer to the in-vehicle viewpoint Vd as the last viewpoint VL disappears earlier. This can lead to an improvement in the attractiveness, that is, the marketability, of the product.

[0122] Step S208 is a step for determining whether the adjustment of the transparency of images other than the floor image Pf, such as the interior image, the tire image Pt, has been completed. Steps S207 to S208 correspond to the vehicle body transparency stage Fs2 and the vehicle body transparency processing. At the time point when step S208 is completed, the final transparency of the tire image Pt may be such that the tire image Pt is completely transparent. In addition, in order to enable the user to recognize the existence of the tire image Pt, the final transparency may be only about 60% to 80%. After the adjustment of the transparency of the components other than the bottom part of the vehicle is completed, the processing goes to step S209. At the time point when the vehicle body transparency stage is completed, the displayed image is an image that shows the periphery of the vehicle in a manner that transparently passes through the vehicle body, such as the engine room and the door panel, while hiding the part below the floor with the floor image Pf.

[0123] Fig.12 FIG. 2 shows the transition of the display screen during the vehicle body transparent stage when the vehicle body transparent stage is switched to the front vehicle interior viewpoint mode. Fig.12In the figure, the semi-transparent components are expressed by dotted lines. Fig.12 As shown in (A) to (B) in FIG. 1 , first, internal images such as the instrument panel image Pi and the handle image Ph are made transparent. Then, the tire image Pt, the floor image Pf, etc. are displayed. Fig.12 As shown in (B) in FIG. 1 , as the internal image is made transparent, a guide line, such as a vehicle boundary line Lvc placed closer to the background than the internal image, begins to be displayed.

[0124] Fig.12 (C) in FIG. 2 shows a display screen when the vehicle body transparent stage is completed. At the time point when the vehicle body transparent stage is completed, a peripheral image including a floor image Pf and a vehicle boundary line Lvc is displayed. In this case, as an example, it is assumed that at the time point when step S208 is completed, the final transparency of the tire image Pt is set to about 80%. This is to give the impression that the tire boundary line Lt to be displayed thereafter is a frame line indicating the tire position.

[0125] At the time point when the vehicle body transparent stage is completed, the tire image Pt may be formed only by an opaque outline or a semi-transparent outline. That is, the portion other than the tire outline may be made colorless and transparent. In addition, similar to the instrument panel image Pi, etc., the tire image Pt may also be made completely transparent. The constituent components displayed at the time point when the vehicle body transparent stage is completed may be appropriately changed. For example, based on the setting made by the user or the designer of the image forming ECU 1, the tire image Pt may not be displayed in the front in-vehicle viewpoint image PDf.

[0126] After the vehicle body transparency processing is completed ("Yes" in S208), as step S209, finally, the synthetic image forming unit F5 starts the processing of gradually increasing the transparency of the floor image Pf at a predetermined speed under the control of the display control unit F6. More specifically, the transparency of the floor image Pf is gradually increased from an opaque state to a completely transparent state. If the tire image Pt still exists at the start of step S209, the transparency of the tire image is also increased to 100% at a predetermined speed. Therefore, the under-floor image is gradually displayed, and finally the front in-vehicle viewpoint image PDf is displayed, in which all vehicle components including the bottom part of the vehicle are transparent.

[0127] Step S210 is a step of determining whether the adjustment of the transparency of the floor image Pf, etc. has been completed. If the process of making the floor image Pf, etc. transparent has been completed, step S210 results in an affirmative determination, and the display transition process ends. Steps S209 to S210 correspond to the bottom portion transparent stage Fs3 and the bottom portion transparent process. Fig.13As shown, the image finally displayed is an image formed by superimposing the vehicle boundary line Lvc, the tire boundary line Lt, etc. on the external world image including the area below the floor photographed from the front in-vehicle viewpoint Vdf as the front in-vehicle viewpoint image PDf. That is, a peripheral image is displayed, wherein the peripheral image shows the front of the vehicle and the vicinity of the front wheels viewed from the driver's viewpoint, etc. in a manner that transparently passes through the vehicle body and the bottom portion. Fig.13 “Lb” shown in is a parking frame line set on the road surface and is an example of a photographic object included in the camera image.

[0128] The case of transitioning to the front in-vehicle viewpoint mode has been described. On the other hand, in the case of transitioning to the rear viewpoint, during the viewpoint movement phase, for example, Fig.14 As shown, the viewpoint can gradually move from the rear bird's-eye viewpoint Vbb as the starting viewpoint VS toward the rear in-vehicle viewpoint Vdb as the final viewpoint VL. When the virtual viewpoint VP has reached the rear in-vehicle viewpoint Vdb, it is preferred to display the interior images of the rear seats, rear window, C-pillar, etc. in an opaque state. By displaying the interior image indicating that the viewpoint is oriented rearward as described above, the user can easily recognize the direction of the virtual viewpoint, etc.

[0129] After the virtual viewpoint VP reaches the rear in-vehicle viewpoint Vdb, in the transparent processing, the interior image of the rear seat, etc. is made transparent at a predetermined speed, and then a predetermined time difference is provided thereafter, and the transparency of the floor image Pf and the tire image Pt corresponding to the rear wheel is also gradually increased. As a result, the image displayed as the final rear in-vehicle viewpoint image PDb shows the ground surface near the rear wheel as the shooting object viewed from the rear in-vehicle viewpoint Vdb, and guide lines such as the vehicle boundary line Lvc and the tire boundary line Lt superimposed thereon.

[0130] <Function of the above-mentioned structure>

[0131] As described above, the surrounding display system Sys is configured to continuously form and display a plurality of synthetic images showing virtual viewpoints closer to the final viewpoint VL from the starting viewpoint VS set outside the vehicle compartment, while converting the content displayed on the display 3 into the in-vehicle viewpoint image PD, over time. That is, the surrounding display system Sys provides an animation expression showing that the position of the virtual viewpoint VP moves step by step from the starting viewpoint VS to the final viewpoint VL outside the vehicle compartment. In addition, immediately after the virtual viewpoint reaches the final viewpoint VL, the internal image taken from the final viewpoint VL is displayed, and then the internal image is gradually made transparent.

[0132] This enables the user to easily understand, with reference to the entire vehicle, from which viewpoint the peripheral image finally displayed on the display 3 is captured.

[0133] During the stage of making the vehicle image transparent, the floor image Pf in the various vehicle images is finally made transparent. In other words, the floor image Pf is displayed at a level sufficient for visual recognition for a predetermined period of time, and thereafter the floor image Pf is made not to be displayed. With this aspect, the user can be left with an impression of the range in which the vehicle body exists and the range in which the image under the floor based on the past camera image is displayed. This can reduce the risk of the user misidentifying the road surface condition under the floor and the range in which the vehicle body exists.

[0134] In addition, using the above-mentioned structure, the starting viewpoint VS is set at different positions depending on the last viewpoint VL. More specifically, when the last viewpoint VL is the front in-vehicle viewpoint Vdf, the front bird's-eye viewpoint Vbf is set as the starting viewpoint VS. When the last viewpoint VL is the rear in-vehicle viewpoint Vdb, the rear bird's-eye viewpoint Vbb is set as the starting viewpoint VS. That is, the position of the starting viewpoint VS directly above the center of the vehicle is set at the area outside the car on the opposite side of the last viewpoint VL. In addition, the virtual viewpoint VP moves from the starting viewpoint VS toward the last viewpoint VL. Using this structure, the moving direction of the virtual viewpoint during the movement of the viewpoint is basically consistent with the line of sight direction of the last viewpoint VL. This enables the user to more intuitively identify in which direction the peripheral image finally displayed on the display 3 was taken.

[0135] The above-mentioned structure is configured to form a synthetic image showing the periphery of the vehicle in a manner that transparently passes through the constituent members of the vehicle Hv, such as the instrument panel and the bottom part of the vehicle body, as a synthetic image taken from the in-vehicle viewpoint. The synthetic image taken from the in-vehicle viewpoint corresponds to an image formed by superimposing the vehicle boundary line Lvc and the tire boundary line Lt on the image of the photographed object existing under the vehicle and at the periphery of the vehicle. By recognizing the synthetic image, the user can recognize the state of the periphery of the vehicle at the viewpoint in the vehicle compartment, and can also intuitively understand the state of the periphery of the vehicle.

[0136] In addition, based on the vehicle boundary line Lvc included in the synthetic image, the user can easily perceive the distance from the vehicle body to a three-dimensional object (for example, a rock, a curb, or a flap that exists near the vehicle). This can reduce the risk of the vehicle body accidentally colliding with such a three-dimensional object. Using the structure for providing the tire boundary line Lt in the synthetic image, the user can easily identify the position of the vehicle body and the tire relative to the dividing line, curb, rock, etc. positioned under the vehicle body. For example, this enables the user to perform detailed driving maneuvers during off-road driving, such as placing the tire on a target rock. In addition, during driving on the road, similarly, it enables the user to easily perform detailed driving maneuvers such as bringing the vehicle body closer to the edge of the road to prevent the tire from colliding with the curb, etc.

[0137] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-mentioned embodiments. Various modifications to be described later are also included in the technical scope of the present disclosure. In addition, without departing from the spirit of the present disclosure, the present disclosure can be implemented by making various changes to it except for these changes to be described later. For example, various modifications to be described later can be implemented by appropriately combining with each other within the scope that does not cause technical inconsistency. Components with the same functions as those described in the above-mentioned embodiments are designated by the same figure numerals, and will not be repeatedly described. When describing only a portion of a structure, the structure according to the above-mentioned embodiments can be applied to other parts of the structure.

[0138] Although in the above-mentioned description, the aspect of performing the bottom part transparent processing after the vehicle body transparent processing has been shown, the present disclosure is not limited thereto. The bottom part transparent processing can be performed in parallel with the vehicle body transparent processing. The drawing of the floor image Pf is an arbitrary element and can be removed. For example, in a case where the under-floor image can be drawn when transitioning to the in-vehicle viewpoint, the floor image Pf can be made not to be displayed from the beginning. Such a state in which the under-floor image can be drawn corresponds to a state in which the image data required to form the under-floor image has been stored in the image memory M1. The floor image Pf is displayed only when an image that can be projected on the vehicle area R0 as an under-floor image has not been accumulated in the image memory M1.

[0139] Although the aspect of performing the viewpoint movement phase and the vehicle body transparency phase completely separately has been disclosed, the present disclosure is not limited thereto. The viewpoint movement phase and the vehicle body transparency phase may partially overlap each other. For example, at a timing when the position of the virtual viewpoint VP has reached a predetermined distance or less from the last viewpoint VL, such as at a timing when the virtual viewpoint VP has entered the vehicle compartment, the vehicle body transparency phase may be started. With this structure, the animation time may be shortened.

[0140] In the case where the virtual viewpoint VP passes through the top part of the vehicle Hv during the viewpoint movement phase, while the top part image Pr remains opaque, only the top part image Pr is shown on the screen while showing the scene where the virtual viewpoint VP is closer to the top part. As a result, the screen is temporarily occupied by the color of the top part, such as white or black. Immediately thereafter, the image inside the vehicle cabin is suddenly displayed thereon.

[0141] As a point of view, such a transition of the display screen provides an advantage of clearly indicating that the virtual viewpoint VP has entered the car from outside the car. However, on the other hand, the user may be disturbed by the above-mentioned sudden transition of the display screen. In order to cope with such concerns, for example, Fig.15As shown, under the control of the display control unit F6, the synthetic image forming unit F5 can also start making the top portion image Pr transparent at the timing when the virtual viewpoint VP has reached a predetermined transparent start distance Hn or less from the top portion. Fig.16 As shown, as the interval Hvp between the top portion and the virtual viewpoint VP as the midway viewpoint Vm decreases, the transparency of the top portion image Pr can be increased. More specifically, when the interval Hvp between the top portion and the virtual viewpoint VP is equal to or less than a predetermined threshold value Hth, the transparency can be made 100%. Fig.16 The alpha value represented by the vertical axis in is a parameter indicating the transparency of the top portion image Pr, wherein an alpha value of 255 corresponds to a completely opaque state, and an alpha value of 0 corresponds to a completely transparent state.

[0142] With the above-mentioned structure, during the viewpoint movement phase, at a time point when the virtual viewpoint VP has reached a predetermined distance or less from the top part, the top part image Pr starts to be made transparent, thereby starting to display the structure inside the vehicle compartment. The result is an animation expression that makes the content displayed when the virtual viewpoint VP exists outside the vehicle compartment continuous with the content displayed after the virtual viewpoint VP enters the vehicle compartment, which can reduce the risk of visually disturbing the user. Since the height of the top part from the road surface is constant, the interval Hvp between the virtual viewpoint VP and the top part can be rephrased as the height of the virtual viewpoint VP from the road surface.

[0143] Although in the above-mentioned embodiment, it has been disclosed that the final display image captured from the in-vehicle viewpoint is an aspect that includes an image of the vehicle CG made completely transparent, the present disclosure is not limited thereto. Some structures indicating the external shape of the vehicle, such as the tire image Pt, may be displayed with a transparency sufficient to be perceived by the user. Such a transparency sufficient to be perceived by the user is, for example, 60% to 85%.

[0144] In the above-mentioned description, an aspect of displaying an animation including moving the viewpoint and making the internal components transparent in the case of transitioning to the in-vehicle viewpoint mode has been shown. However, for example, in the case of transitioning to the front camera mode or the rear camera mode, the same animation can be similarly implemented. In the case of transitioning to the front camera mode, the front bird's-eye viewpoint Vbf can be set as the starting viewpoint. In the case of transitioning to the rear camera mode, the rear bird's-eye viewpoint Vbb can be set as the starting viewpoint. In the case of transitioning to the front camera mode, the front in-vehicle viewpoint Vdf can also be set as the starting viewpoint. In the case of transitioning to the rear camera mode, the rear in-vehicle viewpoint Vdb can also be set as the starting viewpoint. In the case of transitioning to the side camera mode, similarly, an animation can be displayed.

[0145] As the in-car viewpoint, the right in-car viewpoint with the sight oriented to the right of the vehicle and the left in-car viewpoint with the sight oriented to the left of the vehicle can also be adopted. The viewpoint positions of the right in-car viewpoint and the left in-car viewpoint can be at the estimated position of the driver's eyes, or at the center part of the indoor ceiling part. In the case of displaying the image taken from the right in-car viewpoint or the left in-car viewpoint from the standby mode, similarly, the display control involving the display of the animation mentioned above can be performed. In the case of setting these side in-car viewpoints as the last viewpoint VL, it is preferred to place the starting viewpoint VS at a position on the opposite side of the last viewpoint VL that deviates from the position directly above the vehicle Hv. In the above-mentioned description, it has been disclosed that the starting viewpoint VS for the animation is placed at a position on the opposite side of the last viewpoint VL in the front, rear, left and right that deviates from the position directly above the vehicle Hv. However, the present disclosure is not limited to this. The starting viewpoint VS can also be set at a position directly above the vehicle.

[0146] On the other hand, by using a structure configured to display the above-mentioned animation every time the in-car viewpoint image PD or the camera image is displayed, the user may be disturbed. In particular, it is expected that this tendency will be enhanced for users who are accustomed to the surrounding display system Sys. It is also possible that the above-mentioned animation is not displayed with each viewpoint change, but only when the in-car viewpoint image PD or the camera image is displayed for the first time after the driving power is turned on. In other words, the display of the animation of the internal image, etc. can also be removed at the second and subsequent viewpoint changes after the driving power is turned on. By using a structure configured to display the above-mentioned animation only when the in-car viewpoint image PD or the camera image is displayed for the first time after the driving power is turned on, the risk of disturbing the user can be reduced. Since the above-mentioned animation is displayed at least once (i.e., the first time) after the driving power is turned on, users who are not accustomed to the surrounding display system Sys can easily understand from which viewpoint the in-car viewpoint image PD is taken.

[0147] The display / non-display of the animation can be set by the user through the setting screen. However, it is expected that some users find it cumbersome to perform the operation for changing the setting, and some users give up changing the setting when the animation is needed because they do not know the method of changing the setting. In view of such a situation, the display of the animation can also be automatically turned off at a time point when the total driving distance of the vehicle Hv reaches a predetermined threshold of 10,000 km or more. Similarly, the display of the animation can also be automatically turned off at a time point when a predetermined period of time (for example, one year) has passed since the purchase date of the vehicle Hv. The display control unit F6 can be configured to ask the user whether to turn off the display of the animation from now on based on at least one of the following facts: the fact that the cumulative value of the driving distance has exceeded the predetermined value, and the fact that a predetermined period of time has passed since the purchase date of the vehicle. The inquiry can be implemented by displaying a predetermined confirmation screen, etc. The display control unit F6 can be configured to transform the continuation / shutdown of the display of the animation based on the user's response to the inquiry.

[0148] Furthermore, in the animation mentioned above, when the virtual viewpoint VP exists outside the vehicle, the final display range image Px, which is an image indicating the display range at the final viewpoint VL, may be displayed in the peripheral image. Fig.17 As shown, in a scene showing that the virtual viewpoint VP still exists in the sky above, the last display range image Px may be displayed at the beginning of the viewpoint movement phase, etc. In the animation, a viewpoint path line, which is a CG indicating the movement path of the virtual viewpoint VP to the last viewpoint VL, etc. may also be displayed.

[0149] In addition, in the above-mentioned description, a structure configured to provide a viewpoint movement stage Fs1 in an animation when displaying a transition has been disclosed. However, the present disclosure is not limited to this. The viewpoint movement stage Fs1 is not a necessary element and can be removed. That is, the animation when displaying a transition can be composed only of a vehicle body transparent stage Fs2 and a bottom part transparent stage Fs3. Similarly, the bottom part transparent stage Fs3 is not a necessary element and can be removed. The animation only needs to include a vehicle body transparent stage Fs2 for making at least the internal image transparent. The above-mentioned animation may also be composed only of a viewpoint movement stage Fs1 and a vehicle body transparent stage Fs2 and only of a vehicle body transparent stage Fs2. That is, the present disclosure may also be applied to a system that does not have the function of displaying an image under the floor.

[0150] In addition, although in the above-mentioned embodiment, the aspect that the display control unit F6 performs the display control for finally making the tire image non-displayed has been disclosed, the present disclosure is not limited thereto. The display control unit F6 may also continue to display the tire image Pt in a state with a predetermined opacity in the end. More specifically, the display control unit F6 may gradually reduce the transparency of the tire image Pt from 100% to a predetermined target level at a predetermined speed at the timing of starting to make the internal image transparent in step S207. For example, the target level may be set to 60% or 40%. A state with 60% transparency corresponds to a state with 40% opacity. A state with 40% transparency corresponds to a state with 60% opacity. The transparency as the target level may be set to a value sufficient to display the tire image with an opacity equal to or greater than a predetermined threshold value (e.g., 30%). With this animation expression, the tire image Pt is gradually displayed as the internal image is made transparent. In addition, after a predetermined period of time has passed since the start of making the internal image transparent, the tire image is displayed in a semi-transparent state in which the tire image has a predetermined opacity.

[0151] After the transparency of the tire image Pt reaches a predetermined target level, for example, even after step S209, the tire image Pt remains displayed. When the tire image Pt remains displayed in a semi-transparent or opaque state, similar to the above-mentioned embodiment, the transparency of the floor image Pf can be gradually changed from a completely opaque state to a completely transparent state. With the above-mentioned structure, even after a series of animations are completed, the tire image Pt remains displayed, which enables the driver to easily recognize the position of the tire on the road surface.

[0152] <Notes>

[0153] The apparatus, system and method disclosed in the present disclosure can be implemented by a special-purpose computer, which constitutes a processor programmed to perform one or more functions embodied by a computer program. The apparatus and method disclosed in the present disclosure can also be implemented using a dedicated hardware logic circuit. The apparatus and method disclosed in the present disclosure can also be implemented by one or more special-purpose computers, which are composed of a combination of a processor for executing a computer program and one or more hardware logic circuits. The computer program as an instruction to be executed by a computer can be stored in a computer-readable non-transitory tangible storage medium. That is, the devices and / or functions provided by the image forming ECU 1, etc. can be provided by software, only software, only hardware or a combination thereof stored in a tangible storage device and a computer for executing them. For example, some or all of the functions included in the image forming ECU 1 can be implemented as hardware. The aspect of implementing a specific function by hardware includes the aspect of implementing the function using one or more ICs, etc. The image forming ECU 1 can also be implemented by using an MPU, a GPU or a DFP (data stream processor) instead of a CPU. The image forming ECU 1 can also be implemented by a combination of multiple types of arithmetic processing devices such as a CPU, an MPU and a GPU. The image forming ECU 1 can also be implemented as a system on chip (SoC). In addition, various types of processing units can also be implemented by using FPGA (field programmable gate array) or ASIC (application-specific integrated circuit). Various types of programs can be stored in non-transitory tangible storage media. As a medium for storing programs, various storage media can be used, such as HDD (hard disk drive), SSD (solid state drive), flash memory, SD (secure digital) card, etc.

Claims

1. A peripheral image display device for a vehicle, comprising: a synthetic image forming unit (F5) configured to form a peripheral image that is a synthetic image showing an area of ​​the periphery of the vehicle viewed from an arbitrary virtual viewpoint by using a plurality of camera images captured by a plurality of camera devices configured to photograph the periphery of the vehicle; as well as a display control unit (F6) configured to execute processing for displaying the peripheral image formed by the synthetic image forming unit on a display, wherein: The synthetic image forming unit is configured to form an in-vehicle viewpoint image (PD) showing the periphery of the vehicle in a manner transparently passing through a part or all of the vehicle, the in-vehicle viewpoint image (PD) being the periphery image photographed from an in-vehicle viewpoint (Vd) positioned in a vehicle compartment as the virtual viewpoint, The synthetic image forming unit is further configured to form an interior added image (PDi) by using data (Dt) which is a synthetic image including an image of an interior component viewed from the in-vehicle viewpoint, the data (Dt) being used to draw the image of the interior component and being stored in a predetermined storage medium, and The display control unit is configured to: when the display screen of the display is changed to the in-vehicle viewpoint image in response to the user's manipulation of the vehicle, before displaying the in-vehicle viewpoint image, display the following animation: the animation gradually moves the position of the virtual viewpoint from a starting viewpoint set at a predetermined position outside the vehicle compartment to the in-vehicle viewpoint as a target viewpoint, and thereafter gradually increases the transparency of the image of the internal component.

2. The peripheral image display device according to claim 1, wherein: During the stage of moving the virtual viewpoint, the image of the internal component existing in the field of view at the virtual viewpoint is displayed in an opaque state or a semi-transparent state.

3. The peripheral image display device according to claim 2, wherein: The display control unit is configured to start processing of gradually increasing the transparency of the image of the interior component after the virtual viewpoint has reached the in-vehicle viewpoint as the target viewpoint, and to perform animation expression of displaying the image of the tire with an opacity equal to or greater than a predetermined threshold value after a predetermined period of time has passed since the start of making the interior component transparent.

4. The peripheral image display device according to claim 2 or 3, wherein: The display control unit is configured to set the start viewpoint at a different position according to at least one of a position and a line of sight direction of the target viewpoint.

5. The peripheral image display device according to claim 2 or 3, wherein: The display control unit is configured to: when the in-vehicle viewpoint as the target viewpoint is a first in-vehicle viewpoint with a line of sight oriented in front of the vehicle, set a first bird's-eye viewpoint placed above the vehicle and located at a position deviated by a predetermined amount from a position directly above the vehicle toward the rear of the vehicle as the starting viewpoint, and The display control unit is configured to set, as the starting viewpoint, a second bird's-eye viewpoint placed above the vehicle and located at a position deviated a predetermined amount from a position directly above the vehicle toward the front of the vehicle when the in-vehicle viewpoint serving as the target viewpoint is a second in-vehicle viewpoint with a line of sight oriented behind the vehicle.

6. The peripheral image display device according to claim 2 or 3, wherein: The display control unit is configured to: when the virtual viewpoint exists outside the vehicle cabin during the movement of the virtual viewpoint, place the starting viewpoint above the vehicle and display a composite image including a final display range image (Px), wherein the final display range image (Px) indicates the road surface range that is ultimately to be displayed as the in-vehicle viewpoint image.

7. The peripheral image display device according to any one of claims 1 to 3, wherein: The display control unit is configured to display the following composite image as the in-vehicle viewpoint image: the composite image is formed by superimposing at least one of the following on an image of the outside world formed by synthesizing the image of the camera device: a vehicle boundary line (Lvc) indicating a boundary of a range where a vehicle body exists, a tire boundary line (Lt) indicating a boundary of a range where a tire exists, an image of the tire (Pt), and a ground contact area image indicating an area where the tire is grounded to the ground surface.

8. The peripheral image display device according to any one of claims 1 to 3, wherein: When the in-vehicle viewpoint image is displayed for the first time after the vehicle power is turned on, the in-vehicle viewpoint image is displayed after the animation is displayed, and When the in-vehicle perspective image is displayed for a second time or later after the vehicle power is turned on, display of the animation is removed.

9. A display control method executed by at least one processor to control display of a surrounding image, the surrounding image being a composite image showing an area around a vehicle viewed from an arbitrary virtual viewpoint by using a plurality of camera images captured by a plurality of camera devices configured to capture the surroundings of the vehicle, the display control method comprising: forming an in-vehicle viewpoint image (PD) (S106A), the in-vehicle viewpoint image showing the periphery of the vehicle in a manner transparently passing through a part or all of the vehicle as the periphery image, the in-vehicle viewpoint image (PD) being the periphery image photographed from an in-vehicle viewpoint (Vd) positioned in a vehicle compartment as the virtual viewpoint; forming an interior added image (PDi) (S106B) by using data (Dt) which is a composite image including an image of an interior component viewed from the in-vehicle viewpoint, the data (Dt) being used to draw the image of the interior component and being stored in a predetermined storage medium; Displaying the formed composite image on a display (S203); as well as When the display screen of the display is changed to the in-vehicle viewpoint image in response to the user's manipulation of the vehicle, the following animation is displayed before the in-vehicle viewpoint image is displayed: the animation gradually moves the position of the virtual viewpoint from a starting viewpoint set at a predetermined position outside the vehicle compartment to the in-vehicle viewpoint as a target viewpoint, and thereafter gradually increases the transparency of the image of the internal component (S206-S208).

Images