Peripheral image display device

Abstract

The peripheral image display device is configured to: acquire the wheel speed of the vehicle (9); determine the slip state or non-slip state of the wheel of the vehicle based on the acquired wheel speed; acquire a plurality of camera images from each camera; store the camera image indicating the range in the vehicle's travel direction as a past image; generate a peripheral image indicating the periphery of the vehicle by synthesizing the camera images; and display the peripheral image on a display (3). In the non-slip state, the peripheral image display device displays a transparent image that transparently shows the portion below the floor of the vehicle using the past image selected based on the current wheel speed. In the slip state, the peripheral image display device disables the display of the transparent image.

Description

Technical Field

[0001] The present disclosure relates to a peripheral image display device. Background Art

[0002] Conventionally, there is known a perimeter monitoring device that displays the conditions under the floor on a display device. Summary of the invention

[0003] The present disclosure provides a peripheral image display device, which is configured to: acquire a vehicle wheel speed; determine a slip state or a non-slip state of the vehicle's wheels based on the acquired wheel speed; acquire a plurality of camera images from each camera; store a camera image indicating a range in the vehicle's travel direction as a past image; generate a peripheral image indicating the vehicle's periphery by synthesizing the camera images; and display the peripheral image on a display. In a non-slip state, the peripheral image display device displays a transparent image that transparently shows a portion below the floor of the vehicle using a past image selected based on the current wheel speed. In a slip state, the peripheral image display device deactivates the display of the transparent image. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description made with reference to the accompanying drawings. In the accompanying drawings:

[0005] Figure 1 is a block diagram showing the configuration of a peripheral image display system;

[0006] Figure 2 is a diagram showing the installation position and capturing range of each camera device;

[0007] Figure 3 is a block diagram showing the configuration of an image generation ECU;

[0008] Figure 4 is a diagram showing a projection surface for generating a traveling direction image;

[0009] Figure 5 is a diagram showing a traveling direction image including a transparent image;

[0010] Figure 6 is a diagram showing a display image;

[0011] Figure 7 is a flowchart showing a control process performed by the peripheral image display system;

[0012] Figure 8 is a flowchart showing a control process performed by the peripheral image display system according to the second embodiment;

[0013] Fig. 9 is a diagram showing a traveling direction image including a transparent image according to a second embodiment;

[0014] Fig.10 is a diagram showing a traveling direction image in a state of change from a transparent image to a non-transparent image according to a second embodiment;

[0015] Fig.11 is a diagram showing a traveling direction image including a non-transparent image according to a second embodiment;

[0016] Fig.12 is a flowchart showing a control process performed by a peripheral image display system according to a third embodiment;

[0017] Fig.13 is a diagram showing a traveling direction image of a change state from a transparent image to a non-transparent image according to a third embodiment; and

[0018] Fig.14 is another diagram showing a traveling direction image in a changing state from a transparent image to a non-transparent image according to the third embodiment. DETAILED DESCRIPTION

[0019] Japanese Unexamined Patent Publication No. 2016-21653 discloses a peripheral monitoring device that displays the underfloor condition of a vehicle on a display device. The underfloor area is a blind spot near the vehicle. The disclosure of this document is incorporated herein by reference.

[0020] In the structure of the above-mentioned document, the movement amount and position of the vehicle are estimated based on information such as optical flow, wheel speed, GPS, etc., and the past image is displayed as an under-floor image. However, image processing such as optical flow requires a large processing load, and it may take time to estimate the movement amount of the vehicle. In addition, when the movement amount is estimated based on the wheel speed, if the wheel is in a slipping state, the estimated movement amount may be significantly different from the actual movement amount. When an erroneous image is displayed as an under-floor image, the user may misunderstand the condition under the floor of the vehicle, and the reliability of the peripheral image may be reduced. Due to this reason and other reasons not described, further improvements to the peripheral image display device are needed.

[0021] An object of the present disclosure is to provide a peripheral image display device having high reliability.

[0022] According to one aspect of the present disclosure, a peripheral image display device includes: a wheel speed acquisition unit that acquires the wheel speed of a vehicle; a wheel slip determination unit that determines the slip state or non-slip state of the vehicle wheel based on the wheel speed acquired by the wheel speed acquisition unit; an image acquisition unit that acquires a plurality of camera images from respective cameras, each camera capturing the peripheral range of the vehicle in succession; an image storage device that stores the camera images indicating the range in the driving direction of the vehicle among the camera images acquired by the image acquisition unit as past images; a synthetic image generation unit that generates a peripheral image indicating the periphery of the vehicle by synthesizing the camera images acquired by the image acquisition unit; and a display control unit that displays the peripheral image generated by the synthetic image generation unit on a display. In a non-slip state, the display control unit displays a transparent image that displays a portion under the vehicle floor in a transparent manner using the past images selected based on the current wheel speed. In a slip state, the display control unit invalidates the display of a transparent image that displays a portion under the vehicle floor using the past images selected based on the current wheel speed.

[0023] According to the above peripheral image display device, in a non-skidding state, the display control unit displays a transparent image showing a portion under the vehicle floor in a transparent manner using a past image selected based on the current wheel speed. In a skidding state, the display control unit does not display a transparent image showing a portion under the vehicle floor using a past image selected based on the current wheel speed. Therefore, in a skidding state, by displaying an erroneous image as the under-floor image, it is possible to prevent the user from misunderstanding the under-floor condition. Therefore, a peripheral image display device with high reliability can be provided.

[0024] The various aspects disclosed in this specification adopt different technical solutions from each other in order to achieve their respective goals. The purposes, features and advantages disclosed in this specification will become apparent by referring to the following detailed description and the accompanying drawings.

[0025] The embodiments will be described with reference to the accompanying drawings. In some embodiments, functionally and / or structurally corresponding and / or associated parts may be given the same reference numerals or reference numerals with different numerals at or above the hundredth place. For corresponding parts and / or associated parts, reference may be made to the description of other embodiments.

[0026] First embodiment

[0027] exist Figure 1, the surrounding image display system 1 displays a surrounding image on a display 3, which indicates the surroundings of the subject vehicle 9 on which the system is installed. The surroundings of the vehicle 9 may include positions in various directions around the vehicle 9, such as the front, the back, the side, and downward. The surrounding image may include an image of a partial direction rather than the entire surroundings of the vehicle 9. For example, the surrounding image may include an image showing only the front or rear of the vehicle. In the following description, the vehicle 9 equipped with the surrounding image display system 1 is also referred to as the own vehicle.

[0028] For example, the present vehicle is a four-wheel vehicle having a driving power source, and is considered to be driven not only on roads with asphalt-paved surfaces but also on unpaved roads. As driving modes, the present vehicle has a normal mode and an off-road mode. A driving mode suitable for driving on paved roads is set to the normal mode. A driving mode suitable for driving on unpaved roads is set to the off-road mode. The driving force distribution control method for the front, rear, left and right wheels is different in the normal mode than in the off-road mode. As is well known, off-road indicates a surface with large bumps, such as a stone road. Off-road can also be understood as ground outside the road, that is, ground without maintenance. The present disclosure can also be applied to vehicles that are not expected to drive off-road. The present vehicle can be a gasoline-powered vehicle with an engine as a driving source device, or it can be an electric vehicle or a hybrid vehicle with a motor as a driving source device.

[0029] In the following description, the front-to-back direction, the left-to-right direction, and the up-to-down direction are defined with reference to the vehicle. Specifically, the front-to-back direction corresponds to the longitudinal direction of the vehicle. The left-to-right direction corresponds to the width direction of the vehicle. The up-to-down direction corresponds to the height direction of the vehicle. From another perspective, the up-to-down direction corresponds to a direction perpendicular to a plane parallel to both the front-to-back direction and the left-to-right direction. In the present disclosure, a plane perpendicular to the height direction of the vehicle is also referred to as a vehicle horizontal plane. The direction perpendicular to the height direction of the vehicle is also referred to as a vehicle horizontal direction, and the vehicle horizontal direction includes the front-to-back direction and the left-to-right direction.

[0030] The parallel state in the present disclosure is not limited to the completely parallel state. The state with an angle of less than 20 degrees from the completely parallel state can be regarded as a parallel state. That is, the parallel state can include a substantially parallel state with an inclination angle of 20 degrees or less. Similarly, the vertical state in the present invention is not limited to the completely vertical state.

[0031] The peripheral image display system 1 includes an image generation ECU 70, one or more cameras 2, a display 3, a touch panel 4, operation buttons 5, and a vehicle state sensor 6. In the present disclosure, ECU is used as an abbreviation of an electronic control unit and indicates any kind of electronic control device.

[0032] The image generation ECU 70 is communicably connected to each camera 2, display 3, touch panel 4, operation button 5, and vehicle state sensor 6. Each of the above devices and the image generation ECU 70 are individually connected via a dedicated line, or can be connected via a communication network equipped in the vehicle. For example, the camera 2 and the image generation ECU 70 can be directly connected via a dedicated video signal line.

[0033] The image generation ECU 70 generates a composite image of the vehicle's surroundings originating from an arbitrary viewpoint based on the image data captured by each camera device 2. The image generation ECU 70 displays the generated composite image on the display 3. The image generation ECU 70 can support the driving operation of the vehicle 9 by generating and displaying the composite image. The image generation ECU 70 is provided by a computer. The image generation ECU 70 includes a processor 71, a RAM 72, a storage device 73, a communication interface (IF) 74, and a bus connecting these components. The image generation ECU 70 corresponds to an example of a peripheral image display device of the present disclosure.

[0034] The processor 71 is combined with the RAM 72 and is provided by hardware for performing arithmetic processing. For example, the processor 71 is provided by a central processing unit (CPU). The processor 71 performs various processes for serving as a functional block described later by accessing the RAM 72. The RAM 72 is provided by a volatile storage medium.

[0035] The storage device 73 includes a non-volatile storage medium such as a flash memory. The storage device 73 stores an image generation program as firmware and various drawing data for generating a composite image. The drawing data includes data indicating the shape of the projection surface TS, data indicating a three-dimensional (3D) model of the appearance of the vehicle, data indicating a 3D model of each component of the vehicle, and the like. For example, components for which 3D model data is to be prepared include tires, steering wheels, instrument panels, pillars, body panels, and the like. Execution of the image generation program by the processor 71 corresponds to execution of a display control method as a method corresponding to the image generation program.

[0036] The communication IF 74 is a circuit module that serves as an interface for communicating with other devices. The communication IF 74 is provided by an analog circuit element, an IC, etc. The details of the image generation ECU 70 will be described later.

[0037] The camera device 2 is a vehicle-mounted camera device, and captures an image indicating the surroundings of the host vehicle, and outputs the captured image data to the image generation ECU 70. Each camera device 2 includes at least a lens and an image capturing element, and captures an image indicating the surroundings of the host vehicle. The camera device 2 is attached to different installation positions of the host vehicle to capture images with different ranges. Figure 2As shown, the surrounding image display system 1 of the present embodiment includes a front camera 2F, a rear camera 2B, a right camera 2R, and a left camera 2L as camera devices 2. These four camera devices 2 are attached to different positions of the host vehicle and capture images indicating the surrounding environment of the host vehicle in different directions. A specific example is described below.

[0038] The front camera 2F is an image capture device that captures images in front of the vehicle. The front camera 2F is attached to the front end of the vehicle, such as the front grille, in such a manner that the optical axis 2Fa faces the front of the vehicle. When the vehicle 9 is traveling forward, the front camera 2F serves as the camera 2 that captures images in the traveling direction.

[0039] The rear camera 2B is an image capture device that captures images behind the vehicle. The rear camera 2B is attached to the rear end of the vehicle, such as a predetermined position on the rear surface of the vehicle body near the rear license plate or rear window, in a manner such that the optical axis 2Ba faces the rear of the vehicle. When the vehicle 9 travels rearward, the rear camera 2B serves as the camera 2 that captures images in the traveling direction.

[0040] The right camera 2R is a capturing device that captures an image on the right side of the vehicle. For example, the right camera 2R is attached to the right rearview mirror in such a manner that the optical axis 2Ra faces the right side of the vehicle. The left camera 2L is a capturing device that captures an image on the left side of the vehicle. For example, the left camera 2L is attached to the left rearview mirror in such a manner that the optical axis 2La faces the left side of the vehicle.

[0041] As the lens of each camera device 2, a wide-angle lens such as a fisheye lens can be used, and each camera device 2 has a viewing angle θ of 180 degrees or more. Therefore, by using four camera devices 2, an image indicating the entire surrounding environment of the vehicle can be captured. The installation position of each of the above-mentioned camera devices 2 can be appropriately changed as needed. For example, the front camera device 2F can also be attached to the rearview mirror, the upper end of the windshield, etc. For example, the right camera device 2R or the left camera device 2L can be arranged near the base of the A-pillar or the B-pillar. The camera device 2 can be modified to, for example, on the roof, on the dashboard, near the window frame, etc.

[0042] Installation mode data indicating the installation position and installation posture of each camera device 2 relative to the vehicle 9 may be stored in the storage device 73. For example, the installation position of each camera device 2 may be represented as a point in a vehicle three-dimensional (3D) coordinate system, which is a three-dimensional coordinate system whose center is defined at a predetermined position of the vehicle 9. For example, the X-axis of the vehicle three-dimensional coordinate system may be set to be parallel to the left-right direction of the vehicle 9. The Y-axis of the vehicle three-dimensional coordinate system may be set to be parallel to the front-rear direction of the vehicle. The Z-axis of the vehicle three-dimensional coordinate system may be set to be parallel to the height direction of the vehicle. For example, the right direction of the vehicle may correspond to the positive direction of the X-axis, the front direction of the vehicle may correspond to the positive direction of the Y-axis, and the upward direction of the vehicle may correspond to the positive direction of the Z-axis. For example, the center of the vehicle 3D coordinate system may be set to the center of the rear wheel axle.

[0043] The display 3 includes a thin display panel such as a liquid crystal display panel, and various types of information and images are displayed on the thin display panel. The display 3 can be arranged on the dashboard of the vehicle so that the user can visually recognize the screen. The display 3 can be integrated with the image generation ECU 70 by being arranged in the same housing together with the image generation unit 70. The display 3 can also be provided by a separate device arranged separately from the image generation ECU 70. The display 3 includes a touch panel 4 provided on the display panel, and is capable of accepting operations performed by the user.

[0044] For example, the touch panel 4 can be provided by a capacitive touch panel, and outputs a signal indicating the position of the user's touch. In this article, the user mainly refers to the driver as the occupant sitting on the driver's seat of the vehicle. The user may include the occupant sitting on the co-pilot seat of the vehicle.

[0045] The operation button 5 is an operation component that accepts the user's operation on the content displayed on the display 3. For example, the operation button 5 can be provided by a switch for displaying the synthetic image generated by the image generation ECU 70 on the screen and changing the viewpoint position of the synthetic image. For example, the operation button 5 can be set on the steering wheel of the vehicle and mainly receive the operation of the driver of the vehicle. The user can perform various operations on the peripheral image display system 1 using the operation button 5 and the touch panel 4. When the user operates the operation button 5 or the touch panel 4, an operation signal indicating the operation content is input to the image generation ECU 70. The operation button 5 can also be set on the instrument panel.

[0046] The peripheral image display system 1 may include a driving mode switch as an operating member for enabling the driver to switch the driving mode of the vehicle 9 from the normal mode to the off-road mode. The operating member for switching the driving mode may be a dial switch. The shift lever of the vehicle may be equipped with a driving mode switch.

[0047] The vehicle state sensor 6 detects a state quantity related to the driving control of the vehicle. More specifically, the vehicle state sensor 6 is a sensor group including one or more sensors. The vehicle state sensor 6 includes a wheel speed sensor 6s. The wheel speed sensor 6s detects the wheel speed based on the rotation state of the wheel of the vehicle. As the wheel speed sensor 6s, a sensor that detects the rotation of the wheel based on the change of the pulse signal can be used. In this case, the number of pulse signals obtained per unit time increases as the wheel rotation speed increases. The wheel speed sensor 6s does not have to be a sensor that directly detects the wheel speed as a detection target, but can be a sensor that detects a state quantity for calculating the wheel speed. The wheel speed sensor 6s detects the wheel speed of each wheel of the vehicle. For example, in the case where the vehicle has four wheels, namely, the right front wheel, the left front wheel, the right rear wheel, and the left rear wheel, the rotation speed of each wheel is detected separately.

[0048] In addition to the wheel speed sensor 6s, the vehicle state sensor 6 may also include a shift sensor, a vehicle speed sensor, a steering angle sensor, an acceleration sensor, and the like. The shift sensor detects the shift position. The vehicle speed sensor detects the driving speed of the vehicle. The steering angle sensor detects the rotation angle of the steering wheel of the vehicle. The acceleration sensor detects the acceleration acting on the vehicle in at least one of the front-to-back direction, the left-to-right direction, and the up-to-down direction. For example, a 3-axis accelerometer may be used as the acceleration sensor. The detection value of the acceleration sensor may be used to determine the posture of the vehicle relative to the horizontal plane.

[0049] The type of sensor included in the surrounding image display system 1 as the vehicle state sensor 6 can be appropriately designed as needed, and it is not necessary to include all of the above sensors in the surrounding image display system. The vehicle state sensor 6 may also include a vehicle height sensor, a gyro sensor, an azimuth sensor, etc. Each sensor outputs data indicating the current value of the state quantity set as the detection target to the image generation ECU 70.

[0050] like Figure 3 As shown, the image generation ECU 70 includes an image acquisition unit F1, an operation receiving unit F2, a vehicle state acquisition unit F3, a display setting acquisition unit F4, an image recognition unit F5, a synthetic image generation unit F7, a display control unit F8, and an image output unit F9 as functional blocks. The image generation ECU 70 includes an image storage device M1, which is a memory for temporarily storing image data. For example, the image storage device M1 can be provided by a partial storage area of ​​the RAM 72. In the image storage device M1, the vehicle position information and direction information of the vehicle 9 when the image is captured can be stored in association with the image captured by each camera device 2.

[0051] The image acquisition unit F1 acquires image data captured by each camera 2. Image data indicating the entire surrounding environment of the host vehicle can be obtained by combining the image data captured by each camera 2. The image acquisition unit F1 converts the image signal output from the camera 2 into digital image data in a data format, and then outputs the converted image data to the image recognition unit F5 and the synthetic image generation unit F7.

[0052] For example, the image storage device M1 stores image data captured by each camera device 2 whenever the vehicle moves a predefined predetermined distance. The stored image data is associated with the position information and posture information of the vehicle respectively obtained from the image data. For example, the predetermined distance can be set to 0.1m, 0.3m, 0.5m, etc. In the following, the predetermined distance for storing image data is also referred to as the storage distance. When the storage distance is set to 0.1m, the image data to be stored next is image data obtained by capturing a range shifted by 0.1m in the driving direction compared to the most recently stored image data. Instead of the storage distance, a storage time interval can be set, and the image data captured by each camera device 2 can be stored each time the storage time interval has passed.

[0053] For example, when the vehicle is traveling forward, the image storage device M1 may store image data captured by the front camera 2F. The front camera 2F captures images of the ground from just below the front end of the vehicle to 3 meters in front. When the vehicle is moving backward, the image storage device M1 may store image data captured by the rear camera 2B. The rear camera 2B captures images of the ground from just below the rear end of the vehicle to 3 meters in the rear.

[0054] As a storage method of the image data in the image storage device M1, various methods can be adopted. For example, the image acquisition unit F1 overwrites and stores the new data in the oldest update area in the image storage device M1. That is, the image storage device M1 can be configured as a ring buffer. The ring buffer is a storage area that is logically arranged in a ring. The image acquisition unit F1 can be configured to store the image frame acquired within the most recent storage interval in an area different from the image data acquired for each of the above-mentioned storage distances. The image data stored in the image storage device M1 is used by the synthetic image generation unit F7. The image acquisition unit F1 can perform image processing such as distortion correction, enlargement or reduction of size, and cropping on the image acquired by the camera device 2 according to the lens characteristics. In this case, the image data after image processing is stored in the image storage device M1.

[0055] The operation receiving unit F2 receives an operation signal output from the operation button 5 or the touch panel 4. The operation signal indicates the content of the operation performed by the user on the operation button 5 or the touch panel 4. Therefore, the operation receiving unit F2 accepts the user operation to display image data, such as a composite image. The operation receiving unit F2 outputs data corresponding to the received operation signal to the display control unit F8.

[0056] The vehicle state acquisition unit F3 acquires information indicating the state of the host vehicle from other devices of the host vehicle, such as the vehicle state sensor 6. The vehicle state acquisition unit F3 includes a wheel speed acquisition unit F31 and a wheel slip determination unit F32. The wheel speed acquisition unit F31 acquires the wheel speed of each wheel detected by the wheel speed sensor 6s. For example, the wheel speed acquisition unit F31 can acquire the wheel speed of the right front wheel, the wheel speed of the left front wheel, the wheel speed of the right rear wheel, and the wheel speed of the left rear wheel, respectively.

[0057] The wheel slip determination unit F32 determines whether the wheels of the host vehicle are in a slipping state. For example, when one of the front wheels is in a slipping state, the magnitude of friction between the right front wheel and the ground is significantly different from the magnitude of friction between the left front wheel and the ground. Therefore, there is a large difference between the wheel speed of the right front wheel and the wheel speed of the left front wheel. When the difference between the wheel speed of the right front wheel and the wheel speed of the left front wheel is equal to or greater than a predetermined slipping state speed, the wheel slip determination unit determines that the wheel is in a slipping state. In a non-slipping state, when the vehicle is traveling on a curve, the wheel speed of the right front wheel may be different from the wheel speed of the left wheel. In consideration of this situation, the slipping state speed is appropriately set so that the non-slipping state and the slipping state of the vehicle can be appropriately determined.

[0058] The method for determining the slip state of the wheel is not limited to the above method. Instead of the wheel slip determination unit F32 determining whether the wheel is in a slip state based on the wheel speed, an ECU other than the image generation ECU 70 may determine the slip state of the wheel. In this configuration, the wheel slip determination unit F32 acquires the wheel slip determination result determined by an ECU different from the image generation ECU 70.

[0059] The vehicle state acquisition unit F3 acquires the gear position, the acceleration in each detection axis direction, the vehicle speed, the steering angle, etc. from various sensors. The vehicle state acquisition unit F3 can acquire various state quantities from other ECUs. For example, information such as the steering angle can be acquired from the ECU included in the steering system. The vehicle state acquisition unit F3 can acquire the change in moving distance per unit time and the change in direction per unit time from the image recognition unit F5. That is, the vehicle state acquisition unit F3 can acquire various information related to the state of the vehicle not only from sensors but also from other ECUs, image recognition unit F5, etc. The vehicle state acquisition unit F3 can combine various types of information and calculate the moving distance of the vehicle 9, the change in direction of the vehicle 9, etc. The direction of the vehicle 9 can include not only the yaw angle faced by the vehicle 9, but also the pitch angle, the roll angle, etc.

[0060] The display setting acquisition unit F4 acquires the settings of the image to be displayed on the display 3. The display settings may include settings related to the brightness of the display 3. The display settings may include settings related to the display time and the display duration. The display settings may include settings related to the viewpoint position of the image to be displayed. The display settings may include settings related to the display effect when switching the displayed image. The display settings may be changed by the user's operation on the touch panel 4 or the operation button 5.

[0061] The image recognition unit F5 detects the position of a predetermined detection target, the type of a predetermined detection target, and the like by analyzing an image acquired from the camera 2. The image recognition unit F5 has a function identifier for identifying the type of the detection target based on an image feature vector. The image recognition unit F5 recognizes an object by using, for example, a convolutional neural network (CNN) or a deep neural network (DNN) to which deep learning is applied. The detection targets include pedestrians, other vehicles, road markings (such as lane markings marked on paved roads), and road edges. In addition to the lines provided as lane markings, road markings may also include lines indicating parking boxes and the like.

[0062] The image recognition unit F5 is able to estimate the amount of travel and the amount of change in direction of the vehicle based on feature points pre-set on the image by comparing the feature points in one frame with the feature points in different frames. That is, the amount of travel and the amount of change in direction are estimated based on the amount of change in the vehicle's position. For example, the image recognition unit F5 can estimate the current position relative to a reference point, the amount of change in direction of the vehicle body, the yaw rate, etc. by using an optical flow method, etc. The optical flow is called an inter-frame difference method. The optical flow is information indicating the movement of an object included in the image data as a vector. The object recognition result of the image recognition unit F5 and the estimation result of the position of the vehicle, etc. are output to the display control unit F8, the vehicle state acquisition unit F3, and the synthetic image generation unit F7.

[0063] The synthetic image generation unit F7 performs image processing for generating synthetic images such as the traveling direction image AP and the transparent image CP described later. The synthetic image generation unit F7 projects a plurality of image data captured by the camera device 2 onto the virtual projection surface TS, and generates a synthetic image observed from the virtual viewpoint VP using the data projected onto the projection surface TS.

[0064] like Figure 4 As conceptually shown, the projection surface TS is a virtual three-dimensional plane corresponding to the surrounding area of ​​the vehicle. The central area of ​​the projection surface TS is defined as the vehicle area corresponding to the position of the vehicle. For example, the vehicle area is set to a rectangular flat portion. The vehicle area includes an area that overlaps with the vehicle in the height direction of the vehicle. In other words, the vehicle area includes an area that cannot be directly imaged by the camera device 2. The projection surface TS has a planar shape along the horizontal direction of the vehicle near the vehicle area, and has a curved shape at a position away from the vehicle area, and the inclination of the curved shape increases as the distance from the vehicle area increases. That is, the projection surface TS has a curved shape that is convex downward as a whole. Projecting the camera image onto the projection surface TS corresponds to mapping the camera image texture onto the projection surface TS.

[0065] The shape of the projection surface TS may be appropriately changed. The shape of the projection surface TS may be set to a horizontal plane in the entire region. The projection surface TS may have a shape in which a curved region (eg, a region convex downward) starts from an edge portion of the vehicle region without having the above-mentioned flat region.

[0066] The display control unit F8 controls the operation of the synthetic image generation unit F7. For example, the display control unit F8 controls the virtual viewpoint VP for generating the synthetic image. The synthetic image generation unit F7 includes a camera image synthesis unit F71 and a vehicle model assignment unit F72. The camera image synthesis unit F71 projects the image captured by each camera onto the projection surface TS. The vehicle model assignment unit F72 arranges the model image of the vehicle 9 at a predetermined position on the projection surface TS on which the image captured by the camera is projected.

[0067] The display control unit F8 comprehensively controls the overall operation of the image generation ECU 70. For example, the display control unit F8 controls the synthetic image generation unit F7 and the display image generation unit F81 based on the information output from the operation receiving unit F2 and the vehicle state acquisition unit F3. Thus, the synthetic image and the display image DP are generated according to the driving state of the host vehicle and the user's settings.

[0068] The display control unit F8 determines whether the vehicle is traveling forward or backward based on a signal output from a shift sensor or the rotation direction of a tire, for example. The display control unit F8 determines whether the driving environment is off-road based on an input signal from a driving mode switch.

[0069] When generating a synthetic image, the display control unit F8 sets the position and sight direction of the virtual viewpoint VP based on the travel direction of the host vehicle and signals from the touch panel 4 or the operation button 5. For example, the virtual viewpoint VP may be set to a bird's eye view or a driver's eye view. Figure 4 The viewpoint position when the virtual viewpoint VP is set to a bird's-eye view is shown.

[0070] In the case where the bird's-eye view is set as the virtual viewpoint VP, the viewpoint position is directly above the host vehicle, and the line of sight is directed directly downward. The bird's-eye view image can be applied when generating a bird's-eye view image as an image of the host vehicle and its surroundings observed from directly above the host vehicle. The viewing angle of the bird's-eye view can be appropriately adjusted to include the vicinity of the host vehicle. The viewpoint position of the bird's-eye view is not limited to directly above the host vehicle. For example, the observation position may be a position offset from directly above the host vehicle in the rearward direction, forward direction, or lateral direction. The bird's-eye view corresponds to an example of an outside-vehicle viewpoint, which is a viewpoint where the virtual viewpoint VP is arranged outside the vehicle compartment.

[0071] The driver's viewpoint is a virtual viewpoint VP in which the viewpoint position is set to the assumed position of the driver's eyes in the vehicle compartment. For example, the line of sight direction of the driver's viewpoint can be set to diagonally forward and downward so as to include the vicinity of the front wheel. For example, diagonally downward can be a direction pointing downward from the horizontal plane of the vehicle at about 20 to 30 degrees. With the front diagonally downward as the reference direction, the line of sight direction of the driver's viewpoint can be configured to change in any direction based on the user's operation on the touch panel 4. The viewing angle of the driver's viewpoint can be appropriately adjusted to include the vicinity of the front wheel.

[0072] The assumed position of the driver's eyes can be set to a headrest near the driver's seat. As the assumed position of the driver's eyes, the eye ellipses set for each vehicle model can be used. The eye ellipses are virtual space areas defined for each vehicle corresponding to the vehicle type. The eye ellipses are set to virtual elliptical shapes based on eye ranges, which statistically represent the spatial distribution of the points of the occupant's eyes. The driver's viewpoint corresponds to an example of an in-car viewpoint, which is a viewpoint where the virtual viewpoint VP is arranged in the car compartment. As another example, the driver's viewpoint position can be set at a position deviated from the assumed position of the driver's eyes. For example, the driver's viewpoint can be set at a position offset from the assumed position of the driver's eyes to the co-pilot seat. For example, the position between the driver's seat and the co-pilot seat can be set as the position of the driver's viewpoint.

[0073] like Figure 5 As shown, the image generation ECU 70 of the present embodiment has a peripheral image display mode for displaying the traveling direction image AP as an operation mode. Details of a method of generating the traveling direction image AP will be described later.

[0074] The driving direction image AP may include a transparent image CP which is a composite image showing the road surface under the floor of the vehicle as if the floor under the vehicle body is transparent. When the driving direction image AP includes a transparent image CP showing the road surface under the floor of the vehicle, the peripheral image display mode may also be referred to as an under-floor display mode. The display control unit F8 may switch the display mode based on at least one of a specific position of the shift lever or a user operation. For example, the display control unit F8 may set the display mode to the peripheral image display mode in response to satisfying a peripheral image display condition described later. In the following description, as an example, an image generated by setting the virtual viewpoint VP as the driver's viewpoint is described as the driving direction image AP.

[0075] The display image generation unit F81 generates Figure 6 The display image DP is shown. The display image DP is an image to be displayed on the display 3. The display image DP includes a driving direction image AP and an icon image 3i. The icon image 3i indicates the range of the peripheral image displayed on the display 3 with symbols. Figure 6 An image is shown in which the area in front of the vehicle is set as the range of the surrounding image.

[0076] The display image generation unit F81 generates a display image DP using the composite image synthesized by the composite image generation unit F7, the camera image acquired by the image acquisition unit F1, the icon image 3i, etc. The combination of images to be included in the display image DP is determined by the display control unit F8 according to the driving direction of the vehicle and the user's operation on the touch panel 4, etc.

[0077] The image output unit F9 converts the display image DP generated by the display image generation unit F81 into a video signal, and outputs the video signal to the display 3. Thus, the display image DP is displayed on the display 3. The display image DP is displayed on the display 3, and the display image DP includes a synthetic image showing the periphery of the host vehicle and observed from the virtual viewpoint VP.

[0078] Hereinafter, the operation performed by the composite image generation unit F7 to generate the traveling direction image AP including the transparent image CP will be described. The camera image synthesis unit F71 projects each camera image acquired by the image acquisition unit F1 onto a Figure 4The projection position of each camera image on the projection surface TS is pre-defined in association with corresponding information such as table data.

[0079] For example, the camera image synthesis unit F71 projects the image data of the front camera 2F onto the front area of ​​the projection surface TS. The camera image synthesis unit F71 projects the image data of the rear camera 2B onto the rear area of ​​the projection surface TS. The camera image synthesis unit F71 projects the image data of the right camera 2R onto the right area of ​​the projection surface TS, and projects the image data of the left camera 2L onto the left area of ​​the projection surface TS.

[0080] Various methods may be used to display images in an overlapping area. An overlapping area is an area where the image capturing ranges of two or more cameras 2 overlap each other. For example, image data of a plurality of cameras 2 may be mixed in an appropriate ratio, or image data of a plurality of cameras 2 may be connected together at a predetermined boundary line.

[0081] From the past images stored in the image storage device M1, an image showing a portion that is presumed to be located in the vehicle area is extracted. Then, rotation correction is performed on the extracted image as necessary, and then the image is projected onto the vehicle area. Since the image projected onto the vehicle area is an image of the area located directly below the host vehicle, that is, an image below the floor, the image can be called an under-floor image.

[0082] For example, when the host vehicle moves forward, the under-floor image may be generated using a front camera image captured 3 meters behind the current position as a past image. When the host vehicle moves backward, the under-floor image may be generated by using a rear camera image captured 3 meters ahead of the current position as a past image. That is, the under-floor image is generated using a past image when the host vehicle was located at a position opposite to the current position in the traveling direction.

[0083] When the vehicle moves forward, the underfloor image can be generated using the front camera device images captured at multiple time points between the position 3 meters behind the current position and the current position as past images. In this case, one underfloor image is generated based on multiple past images. The same applies when the vehicle moves backward. The image data of the past images of the portion under the floor corresponding to the vehicle area projected can be updated at any time as the vehicle 9 moves.

[0084] The display control unit F8 controls the synthetic image generation unit F7 to set a virtual viewpoint VP for the three-dimensional space including the projection surface TS. The synthetic image generation unit F7 can set a virtual viewpoint VP pointing to an arbitrary line of sight direction from an arbitrary viewpoint position in the three-dimensional space.

[0085] The vehicle model assignment unit F72 arranges various image elements about the vehicle on a virtual three-dimensional space including the projection surface TS on which the camera image is projected. When the virtual viewpoint VP is set outside the vehicle compartment, such as in a bird's-eye view, a vehicle boundary line Lvc indicating the range of the road surface where the bottom of the vehicle is located is assigned to the vehicle area. The vehicle boundary line Lvc indicates the outline of the vehicle when the vehicle body is projected vertically above the road surface, and corresponds to a line indicating the outer shape of the vehicle 9 in a top view.

[0086] The following will refer to Figure 5 The traveling direction image AP generated when the virtual viewpoint VP is arranged in the vehicle compartment (eg, the driver's viewpoint) is described. Figure 5 The driving direction image AP shown assumes that the host vehicle is located in a parking space in a parking lot marked with a parking frame. The driving direction image AP mainly shows the periphery of the driving direction of the host vehicle.

[0087] The driving direction image AP includes a first driving direction image AP1 and a second driving direction image AP2. The first driving direction image AP1 corresponds to a current image currently captured by the camera device 2 and projected on the projection surface. The second driving direction image AP2 corresponds to a past camera device image captured by the camera device 2 in the past and projected on the projection surface. The first driving direction image AP1 shows a capture range that can be captured by the camera device 2 in real time, and the second driving direction image AP2 shows a capture range that cannot be captured in real time. The underfloor image showing the road surface in the vehicle area corresponds to the second driving direction image AP2 showing a capture range that cannot be captured by the camera device 2 in real time. In the figure, the boundary between the first driving direction image AP1 and the second driving direction image AP2 is shown by a double-dotted chain line.

[0088] In addition to the vehicle boundary line Lvc, when the virtual viewpoint VP is the driver's viewpoint, the vehicle model allocation unit F72 can arrange the 3D tire model Pt, the 3D vehicle body model, and the 3D interior model at a predetermined position in the three-dimensional space. In the accompanying drawings, as an example, a case where only the 3D tire model Pt is arranged is shown. The 3D tire model Pt is a 3D model of a tire, and is set to be semi-transparent as a whole. The 3D tire model Pt may include only an opaque outline or a semi-transparent outline of the tire. That is, the portion other than the outline of the 3D tire model Pt may be set to be colorless and transparent.

[0089] The term semi-transparency herein is not limited to a state where the transparency is 50%, and may include, for example, a transparency ranging from 50% to 80%. That is, the term semi-transparency may include a state where the existence of an object is vaguely indicated. Transparency is a parameter where the higher the parameter indication value, the more transparent it is. In the present disclosure, a transparency of 100% corresponds to a completely transparent state. The transparency or opacity of an image is a concept corresponding to the alpha value of a pixel. Generally speaking, the smaller the alpha value, the higher the transparency.

[0090] The 3D body model is a 3D model of the body, and is set to be semi-transparent as a whole. The 3D body model may include only an opaque outline or a semi-transparent outline of the body. That is, the part other than the outline of the 3D body model may be set to be colorless and transparent. The 3D interior model is a 3D model showing the dashboard, steering wheel, and A-pillar.

[0091] The 3D tire model Pt, the 3D vehicle body model, the 3D interior model, etc. are 3D models showing the components of the vehicle and can be referred to as component models. The 3D models arranged in the three-dimensional space for generating the synthetic image can be adjusted appropriately. For example, the arrangement of the 3D tire model Pt, the 3D vehicle body model, the 3D interior model, etc. can be omitted.

[0092] The vehicle model assignment unit F72 can draw a scale line indicating the distance from the end of the vehicle to a part of the image and a vehicle width line indicating the width of the vehicle. The vehicle model assignment unit F72 can draw a planned trajectory line as another image element related to the vehicle 9, which is a line indicating a planned driving trajectory corresponding to the steering angle of the vehicle 9. The vehicle width line can also serve as a planned trajectory line. The vehicle boundary line Lvc, the planned trajectory line, etc. show information about the vehicle 9. Therefore, the vehicle boundary line Lvc, the planned trajectory line, etc. can be referred to as a vehicle information image. The planned trajectory line, the scale line, the vehicle width line, etc. can also serve as guidance information for the driver's driving operation. Therefore, the displayed lines such as the planned trajectory line can be referred to as guide lines.

[0093] The synthetic image generation unit F7 performs rendering of various image elements existing in the three-dimensional space including the projection plane TS according to the virtual viewpoint VP. The synthetic image generation unit F7 cuts out the image elements included in the predetermined viewing angle when observed from the virtual viewpoint VP by using the image data presenting the various image elements. Therefore, the synthetic image generation unit F7 generates a synthetic image showing the underfloor area of ​​the vehicle and the surroundings of the vehicle as observed from the virtual viewpoint VP. For example, as a transparent image CP, an underfloor image in which the bottom of the vehicle body is transparent is generated. In the above description, an embodiment of generating a synthetic image in the order of projecting the camera image onto the projection surface TS and then adding the image elements is described as an example. Alternatively, each of the above-mentioned processes may be performed in parallel.

[0094] Various methods can be used to generate a composite image. For example, a road surface image, a vehicle boundary line Lvc, and a three-dimensional model of components of the vehicle 9 generated based on a camera image can be arranged in different layers. For example, a composite image can be generated by combining three layers such as a camera image layer, a boundary line layer, and a 3D model layer. The camera image layer includes a road surface image as an image of a projection surface TS on which the camera image is projected. The boundary line layer is a layer on which the vehicle boundary line Lvc is arranged. The 3D model layer includes a three-dimensional model such as a 3D tire model Pt. When multiple layers are combined, the camera image layer is set as the bottommost background layer, and the boundary line layer and the 3D model layer can be superimposed on the camera image layer. The layer structure can also be changed appropriately. When each image element is processed separately using multiple layers, the image - each layer is ultimately integrated into the image - becomes a composite image.

[0095] The display image DP to be displayed on the display 3 will be described below. Figure 6 As shown, the display image generation unit F81 generates an image including the traveling direction image AP and the icon image 3i as the display image DP.

[0096] The display content of the display image DP is not limited to the above-mentioned examples, and can be changed appropriately. For example, the right camera image captured by the right camera 2R and the left camera image captured by the left camera 2L can be arranged together with the driving direction image AP. Alternatively, the driving direction image AP, the right camera image, and the left camera image can be displayed in a panoramic manner as a continuous panoramic image. According to such a layout of the display image DP, the left and right surroundings can be visually notified to the driver while maintaining good visibility of the driving direction image AP. By observing the display image DP, the driver can recognize the ground state near the front wheels when moving forward, and can also recognize the situation of the lateral surroundings of the vehicle.

[0097] The right camera image may be generated by cutting out a portion from the camera image captured by the right camera 2R. In another example, when the right side portion of the host vehicle is set as the virtual viewpoint VP, the right camera image may be generated by cutting out a portion of the composite image obtained by rendering within a predetermined viewing angle. The left camera image may be generated by the same generation method as the right camera image using the camera image of the left camera 2L.

[0098] The icon image 3i can act as a switch for switching the display content of the display image DP in response to a touch operation performed by the user. Whether the user touches the icon image 3i can be determined based on the touch position signal output from the touch panel 4. When the user performs a touch operation on the icon image 3i and the touch operation is detected, the display control unit F8 also uses, for example, a display image DP including only the driving direction image AP as a peripheral image, to a panoramic image including left and right camera images in addition to the driving direction image as a peripheral image.

[0099] The following will describe the surrounding image display processing including a series of processing performed when the image generation ECU 70 generates and displays the display image DP. In response to the surrounding image display condition being satisfied, the image generation ECU 70 starts Figure 7 The surrounding image display condition is a condition for activating the display of the display image DP. For example, when the user operates the operation button 5 by, for example, pressing the operation button 5, the image generation ECU 70 may determine that the surrounding image display condition is satisfied. In another example, in response to detecting an operation performed by the user on the touch panel 4 for displaying the display image DP, the image generation ECU 70 may determine that the surrounding image display condition is satisfied. The image generation ECU 70 may start processing in response to the operation receiving unit F2 acquiring a signal indicating that the user has performed an operation for displaying a composite image.

[0100] Specific conditions for displaying the peripheral image may include a gear position indicating the setting state of the transmission, a vehicle speed, etc. For example, the peripheral image display condition may include at least one of the following: the gear position is set to a specific range; or the vehicle speed is lower than a predetermined low speed threshold. The specific range that satisfies the peripheral image display condition may include: a predetermined low range, a rearward range, etc. The predetermined low speed threshold may be, for example, 10km / h, 15km / h, 20km / h, etc. When the operation receiving unit F2 receives the user's operation of the viewpoint switch via the touch panel 4, etc. in a state where the front camera image or the rear camera image is displayed on the display 3, the image generation ECU determines that the peripheral image display condition is satisfied.

[0101] Can be executed repeatedly Figure 7 The processing shown in the figure is repeated until the display end condition is satisfied. For example, the display end condition may be satisfied when the gear position is set to a range outside a specific range, when the vehicle speed changes to a value equal to or greater than a predetermined low speed threshold, or when the user presses the operation button 5 again. That is, when the peripheral image display condition is no longer satisfied, the display end condition is set to be satisfied. When the image generation ECU 70 detects that the user has performed an operation on the touch panel 4 for invalidating the display of the display image DP, the image generation ECU 70 may determine that the display end condition is satisfied.

[0102] In S101, the image acquisition unit F1 acquires a camera image acquired by each of the four cameras 2. The acquisition of the camera images is repeated with the passage of time, and the latest camera image is acquired in a continuous manner.

[0103] Whenever the host vehicle travels a storage distance, the image storage device M1 stores the camera image captured in the travel direction by the camera 2. When the state that does not satisfy the surrounding image display condition is switched to the state that satisfies the surrounding image display condition, the image acquisition unit F1 stores the image captured in the travel direction by the camera 2 as the first storage process. After the camera image is acquired, the process proceeds to S102.

[0104] In S102, the display setting acquisition unit F4 acquires display settings. The display setting acquisition unit F4 acquires settings such as whether the virtual viewpoint VP is a driver's view or a bird's-eye view.

[0105] When the transparent image CP is generated for the first time in response to the driving power being turned on, the default setting of the virtual viewpoint VP is activated. The default setting may include a position and a sight line direction preset by a designer or user. The default setting may, for example, set the driver's viewpoint so that the sight line direction points diagonally forward and downward. The virtual viewpoint VP may be set to the position and direction of the virtual viewpoint VP that is the same as the last display of the transparent image CP. In this case, as a preparatory process, the display control unit F8 may be configured to save the setting data of the last virtual viewpoint VP used to display the transparent image CP in the RAM 72 or the like.

[0106] In addition, the position and sight direction of the virtual viewpoint VP can be determined according to the driving direction of the vehicle. For example, when the driving direction of the vehicle is forward, the driver's viewpoint with the sight direction pointing diagonally forward and downward as described above can be used as the virtual viewpoint VP. For example, when the driving direction of the vehicle is backward, the driver's viewpoint with the sight direction pointing diagonally backward and downward can be used as the virtual viewpoint VP. The sight direction of the virtual viewpoint VP can be adjusted according to the steering angle.

[0107] An example in which a driver's viewpoint whose line of sight is directed diagonally forward and downward is adopted as the virtual viewpoint VP will be described below. After the display setting is acquired, the process proceeds to S111.

[0108] In S111, the vehicle state acquisition unit F3 acquires the vehicle state. The vehicle state indicates the state of the host vehicle, such as the gear position and the vehicle speed. Specifically, the wheel speed acquisition unit F31 of the vehicle state acquisition unit F3 acquires the wheel speed of the host vehicle. After acquiring the vehicle state, the process proceeds to S112.

[0109] In S112, the vehicle state acquisition unit F3 calculates the wheel speed difference between the left wheel speed and the right wheel speed. More specifically, the difference between the wheel speed of the right front wheel and the wheel speed of the left front wheel is calculated as the wheel speed difference. As another example, the difference between the wheel speed of the right rear wheel and the wheel speed of the left rear wheel can be calculated as the wheel speed difference. After the wheel speed difference is calculated, the process proceeds to S113.

[0110] In S113, the wheel slip determination unit F32 determines whether the wheels of the vehicle are in a slipping state. When the difference between the wheel speed of the right front wheel and the wheel speed of the left front wheel is equal to or greater than a predetermined slipping state wheel speed, the wheel slip determination unit F32 determines that one of the front wheels of the vehicle is in a slipping state. Alternatively, the wheel slip determination unit F32 may determine that one of the rear wheels of the vehicle is in a slipping state when the difference between the wheel speed of the right rear wheel and the wheel speed of the left rear wheel is equal to or greater than a predetermined slipping state wheel speed. At this time, the difference between the left wheel speed and the right wheel speed is preferably calculated by using the average speed value per unit time as the wheel speed of each wheel instead of using the instantaneous speed value as the wheel speed of each wheel. With this configuration, erroneous determination of the slipping state can be suppressed.

[0111] The method for determining the slipping state is not limited to the above method for determining the slipping state based on the difference between the left wheel speed and the right wheel speed. For example, when the wheel speed is very high relative to the output power of the power source (such as the engine or the driving motor), it can be determined that the friction between the tire and the ground is very small, that is, the vehicle is in a slipping state.

[0112] When a determination result is acquired from an ECU other than the image generation ECU 70 that determines the slip state based on the wheel speed, the wheel slip determination unit F32 may acquire the determination result and adopt the acquired result as the slip state determination result.

[0113] If at least one of the front wheels or rear wheels of the host vehicle is determined to be in a slipping state (S113: Yes), the process proceeds to S119. If the wheels of the host vehicle as a whole are not in a slipping state, that is, in a non-slipping state (S113: No), the process proceeds to S118.

[0114] In S118, the synthetic image generation unit F7 updates the past image used to generate the transparent image CP based on the wheel speed. More specifically, the distance traveled by the host vehicle can be calculated based on the wheel speed, and the camera image, which previously captured the road surface under the floor of the host vehicle at the current time, can be acquired based on the past image stored in the image storage device M1 and set as the past image.

[0115] The latest camera image in the traveling direction is stored in the image storage device M1 as a candidate for a new past image. Among the camera images stored as candidates for past images, the camera image obtained by capturing the position of the vehicle passing under the floor of the own vehicle is deleted from the image storage device M1. In other words, with the position under the floor of the own vehicle as a reference, only the camera image in the traveling direction is stored as a candidate for the past image, and the camera image on the opposite side of the traveling direction is deleted from the image storage device. After the past image is updated based on the wheel speed, the process proceeds to S121.

[0116] In S119, the synthetic image generation unit F7 holds the past image immediately before the wheel is determined to be in the slipping state. In the case where it is determined based on the wheel speed that the host vehicle has traveled the stored distance in the traveling direction, the newly captured camera image is not stored, and the candidate of the currently stored past image is not deleted. After holding the past image immediately before the slipping state, the process proceeds to S121.

[0117] In S121, the camera image synthesis unit F71 synthesizes the camera image. More specifically, the camera image synthesis unit F71 maps the image data of each camera 2 and the past image to a predetermined position on the projection surface TS as described above. In a non-slip state, a past image updated based on the wheel speed is projected on the vehicle area. In a slip state, a past image immediately before the slip state is projected on the vehicle area. After synthesizing the camera image, the process proceeds to S122.

[0118] In S122, the vehicle model assignment unit F72 assigns a vehicle model, such as a vehicle boundary line Lvc or a 3D tire model Pt, to the synthesized image. As other image elements, 3D models other than the 3D tire model Pt, such as a planned track line, a scale line, a vehicle width line, etc., may also be added to the synthesized image. After the vehicle model is assigned, the process proceeds to S125.

[0119] In S125, the synthetic image generation unit F7 generates a driving direction image AP including the transparent image CP. More specifically, the synthetic image after adding the vehicle model is rendered according to the virtual viewpoint VP, and image elements included in a predetermined viewing angle when viewed from the virtual viewpoint VP are cut out. When the virtual viewpoint VP is set to the driver's viewpoint, that is, diagonally downward and forward, image elements including the front wheels and the front end of the host vehicle are cut out as the driving direction image AP.

[0120] In the traveling direction image AP, the cutout portion where the live camera image is projected corresponds to the first traveling direction image AP1. In the traveling direction image AP, the cutout portion where the past image is projected corresponds to the second traveling direction image AP2.

[0121] The first traveling direction image AP1 is a composite image obtained by synthesizing camera images captured by the three cameras 2, namely, the front camera 2F, the right camera 2R, and the left camera 2L. The second traveling direction image AP2 is a composite image obtained by synthesizing a plurality of past images captured at different time points. Each past image included in the second traveling direction image AP2 is a camera image obtained by performing correction such as rotation correction on a camera image captured by the front camera 2F.

[0122] In the case where the wheels of the host vehicle are not in a slipping state, the second driving direction image AP2 is updated with an appropriate past image according to the movement of the host vehicle. In the case where the wheels of the host vehicle are in a slipping state, the past image captured at the position immediately before the slipping state starts is maintained as the second driving direction image AP2, and the past image does not change in the slipping state. Regardless of whether the wheels are in a slipping state or in a non-slipping state, the first driving direction image AP1 is always set as an image synthesized with the currently captured camera image.

[0123] In S141, the display image generation unit F81 generates the display image DP to include the traveling direction image AP generated by the synthetic image generation unit F7. After the display image DP is generated, the process proceeds to S142.

[0124] In S142, the image output unit F9 outputs the display image DP to the display 3. Specifically, the image output unit F9 outputs a signal obtained by converting the digital image data of the display image DP generated by the display image generation unit F81 into a predetermined signal format to the display 3. Therefore, the display image DP including the driving direction image AP is displayed on the display 3. After the display image DP is output, the surrounding image display process is terminated. This process may be repeatedly performed until the display end condition is satisfied, and the latest surrounding image may be continuously displayed.

[0125] The technical effect of the above-mentioned embodiment will be described below. In the above-mentioned embodiment, in a non-slip state of the vehicle, the display control unit F8 uses a past image selected based on the current wheel speed to display the driving direction image AP as including the transparent image CP. In a slip state, the display control unit F8 uses a past image immediately before the start of the slip state to display the driving direction image AP as including the transparent image CP. In other words, in a non-slip state, the display control unit F8 uses a past image selected based on the current wheel speed to display a transparent image CP showing the road surface under the floor of the vehicle 9. In a slip state, the display control unit F8 does not display the transparent image CP using the past image selected based on the current wheel speed. Therefore, the display of the transparent image CP that uses a camera image at a position significantly different from the actual under-floor position as a past image can be suppressed. Therefore, in a slip state, it is possible to suppress the display of an image that is significantly different from the actual surroundings of the vehicle as a peripheral image. Therefore, a highly reliable peripheral image display device can be provided.

[0126] Past image updating and slip state determination are performed based on the wheel speed. Therefore, it is easy to reduce the processing load compared to the case where the past image is updated by a method including image processing of a camera image such as an optical flow method. Therefore, it is easy to update the past image at high speed and shorten the time delay occurring in the peripheral image display process.

[0127] In the slipping state, the display control unit F8 displays the traveling direction image AP as including the transparent image CP generated by using the past image immediately before the wheels of the vehicle start to slip. In other words, the display control unit F8 displays the transparent image CP displayed immediately before the wheels start to slip when determining that the wheels are in the slipping state. Therefore, compared with the case where the transparent image CP is generated and displayed using the past image selected based on the wheel speed in the slipping state, it is easy to reduce the deviation between the transparent image CP and the actual underfloor position.

[0128] The traveling direction image AP includes the camera image of the front camera 2F, the camera image of the left camera 2L, and the camera image of the right camera 2R. That is, the display control unit F8 displays a peripheral image synthesized by using the camera image in the traveling direction and the camera images in the left and right directions of the vehicle 9. Therefore, compared with the case of displaying a peripheral image generated using only the camera image in the traveling direction, a wider peripheral range can be displayed by the peripheral image.

[0129] When the difference between the wheel speed of the left wheel and the wheel speed of the right wheel is equal to or greater than the predetermined slipping state wheel speed, the wheel slip determination unit F32 determines that the wheel is in the slipping state. Therefore, the wheel slip determination unit can determine the wheel slip based only on the wheel speed. For example, compared with the case where the wheel slip is determined by other devices other than the wheel speed sensor 6s, the wheel slip determination of the above-mentioned embodiment can be performed with a simple configuration by determining the wheel slip by an acceleration sensor for detecting the acceleration of the vehicle 9 and a position detection device for detecting the position of the vehicle.

[0130] The traveling direction image AP includes a transparent image CP showing the surroundings of the host vehicle. The transparent image is generated in a transparent manner, just as components of the vehicle 9 such as the dashboard and the floor of the vehicle body are transparent. Therefore, by confirming such a surrounding image, the user can confirm the state of the surroundings of the host vehicle from the viewpoint in the vehicle compartment, and can intuitively grasp the state of the surroundings of the host vehicle.

[0131] The driving direction image AP includes a vehicle model, such as a vehicle boundary line Lvc and a 3D tire model Pt. Based on the vehicle boundary line Lvc included in the driving direction image AP, the user can easily recognize the sense of distance from the vehicle body to three-dimensional objects such as stones, curbs, or fenders that exist near the vehicle. Therefore, the risk of unintentional contact between the vehicle body and the three-dimensional object can be reduced. The user can easily recognize the position of the vehicle body and the tire relative to lane markings, curbs, stones, etc. located under the vehicle body. Therefore, during off-road driving, fine driving operations such as placing the tire on a target stone can be performed. During road driving, fine driving operations such as moving the vehicle body toward the edge of the road can be easily performed without causing the tire to come into contact with the curb.

[0132] The effect of the present embodiment has been described for an example in which the driver's viewpoint with the line of sight pointing forward is set as the virtual viewpoint VP. The same effect can be obtained when the line of sight points to other directions, such as backward, laterally, and obliquely laterally. The same effect can be obtained when the virtual viewpoint VP is arranged at any position in the vehicle compartment other than the driver's viewpoint. The same effect can be expected when the virtual viewpoint VP is set on the outer surface of the vehicle 9 or at a point near the outer side of the outer surface of the vehicle. For example, the vicinity of the outer side of the vehicle refers to an area within 0.2m from the outer surface of the vehicle 9. The outer surface may include a rear surface portion, a front end portion, a roof, and left and right side portions. Each side surface portion may include a door panel, a fender, a pillar, etc.

[0133] The virtual viewpoint VP can be appropriately set at different positions in the vehicle cabin. For example, the virtual viewpoint VP can be set so that the viewpoint position is set near the side mirror or at the center of the ceiling in the vehicle cabin, etc. The display control unit F8 can set the internal viewpoint in the vehicle cabin so that the viewpoint position is set behind the eye ellipse and the line of sight is diagonally rearward and downward. This internal viewpoint is also called an internal rear viewpoint. According to such an internal rear viewpoint, when the vehicle moves backward, the vicinity of the rear wheels can be displayed as a driving direction image AP in a larger size. With this configuration, it is easier for the driver to recognize the situation around the rear wheels and the rear bumper when moving backward.

[0134] In the above-mentioned embodiment, the display 3 is provided by the vehicle-mounted display. The device for displaying the image generated by the image generation ECU 70 is not limited to the vehicle-mounted display. The display 3 may be a display device for remotely controlling the vehicle 9 provided by a mobile terminal such as a smartphone. The display 3 may be a display device provided in a management center that manages the vehicle 9 by remote control.

[0135] Second embodiment

[0136] The second embodiment is a modification of the above-described first embodiment. In the present embodiment, the traveling direction image AP includes a non-transparent image BP generated by gradually reducing the transparency of a slip state image displayed in a superimposed manner on a past image, instead of including a transparent image CP.

[0137] The following will refer to Figure 8 The flowchart shown describes the surrounding image display processing. In the following description, the same parts as the first embodiment may be omitted. The image generation ECU 70 starts the surrounding image display processing in response to the surrounding image display condition being satisfied. When the wheel slip determination unit F32 determines in S113 that the vehicle is in a non-slip state (S113: No), the processing proceeds to S118. When the wheel slip determination unit F32 determines in S113 that the vehicle is in a slip state (S113: Yes), the processing proceeds to S219.

[0138] In S118, the synthetic image generation unit F7 updates the past image based on the wheel speed. Then, the process proceeds to S121, and the camera image synthesis unit F71 synthesizes the camera image. Then, the process proceeds to S122, and the vehicle model assignment unit F72 assigns a vehicle model to the synthetic image. Then, the process proceeds to S125, and the synthetic image generation unit F7 generates a traveling direction image AP including the transparent image CP. Then, the process proceeds to S241.

[0139] In S219, the synthetic image generation unit F7 adopts a slipping state image to replace the past image. The slipping state image is an image to be displayed when the wheel is in a slipping state. The slipping state image can be appropriately set when the image does not mislead the user as to the current condition displayed below the floor. As the slipping state image, a monochrome image such as a black and white image can be adopted. As the slipping state image, an image indicating that the vehicle is in a slipping state with a character message can be adopted. The slipping state image can be set with a color different from the road surface color or the vehicle model color. After the slipping state image is adopted to replace the past image, the processing proceeds to S231.

[0140] In S231, the composite image generation unit F7 uses the camera image and the slip state image to composite an image. More specifically, the image data of each camera 2 and the slip state image are mapped to a predetermined position on the projection surface TS. The slip state image is projected on the vehicle area. Therefore, the slip state image is combined with the portion corresponding to the underside of the vehicle floor. After the image is composited, the process proceeds to S232.

[0141] In S232, the vehicle model assignment unit F72 assigns a vehicle model, such as the vehicle boundary line Lvc or the 3D tire model Pt, to the synthesized image. After the vehicle model is assigned, the process proceeds to S235.

[0142] In S235, the synthetic image generation unit F7 generates a driving direction image AP including a non-transparent image BP. The non-transparent image BP is an image different from the transparent image CP. The non-transparent image BP does not transparently show the road surface under the floor of the vehicle as the transparent image CP does. Therefore, the user's misunderstanding of the transparent image CP can be avoided. The synthetic image generation unit F7 performs rendering on the synthetic image after allocating the vehicle model according to the virtual viewpoint VP. The synthetic image generation unit F7 generates the driving direction image AP including the non-transparent image BP by cutting out image elements included in a predetermined viewing angle when observed from the virtual viewpoint VP.

[0143] The following will describe a method of generating the traveling direction image AP when the transparent image CP is switched to the non-transparent image BP to generate the traveling direction image AP. Fig. 9 , the first driving direction image AP1 is an image obtained by synthesizing the current camera images captured by the front camera 2F, the right camera 2R, and the left camera 2L. The second driving direction image AP2 is a past image captured by the front camera 2F. In this state, each of the first driving direction image AP1 and the second driving direction image AP2 includes a white line indicating a parking frame.

[0144] exist Fig.10In the embodiment of the present invention, the second driving direction image AP2 is generated by synthesizing the past image and the slipping state image. More specifically, the second driving direction image AP2 is generated by superimposing the slipping state image with increased transparency on the past image immediately before the slipping state. In other words, the second driving direction image AP2 is generated by superimposing a semi-transparent image in the slipping state on the past image immediately before the slipping state. Fig.10 In order to show that the second traveling direction image AP2 is generated by superimposing a semi-transparent image in a slipping state on a past image immediately before the slipping, hatching with a large pitch is applied to the past image immediately before the slipping state.

[0145] When the slipping state image is a black image, the past image is displayed in a black transparent manner. Therefore, it is difficult for the user to visually recognize the past image immediately before the slipping state. When the slipping state image is a character image indicating the slipping state of the vehicle, the characters indicating the slipping state may be light-colored, and the slipping state image is superimposed on the past image immediately before the slipping state. In the above example, since a transparent image showing the road surface under the vehicle is displayed, the white line of the parking box is included in the second driving direction image AP2.

[0146] exist Fig.11 In FIG. 1 , the second driving direction image AP2 only includes the slip state image provided by the non-transparent image BP. Fig.11 In order to show that the second driving direction image AP2 only includes the slip state image, the same Fig.10 When the slipping state image is a black image, the entire second driving direction image AP2 becomes black, and the user cannot visually recognize the past image immediately before the slipping state. When the slipping state image is a character image indicating the slipping state, the user can only visually recognize the characters indicating the slipping state, and the user cannot visually recognize the past image immediately before the slipping state.

[0147] Since the slipping state image is not an image showing the road surface under the vehicle in a transparent manner, the white line of the parking frame is not included in the second driving direction image AP2. However, when the current camera image is combined and displayed in the first driving direction image AP1, the white line of the parking frame is included in the first driving direction image. The user can confirm the surrounding image other than under the floor of the vehicle by confirming the first driving direction image AP1. The user can recognize that the wheel is in a slipping state by confirming that the slipping state image is displayed in the second driving direction image AP2 instead of the under-floor image.

[0148] When generating the driving direction image AP including the non-transparent image BP, as shown in FIG. Fig. 9 , Fig.10 and Fig.11 As shown in the sequence of , the image including only the past image is gradually switched to the image including only the slipping state image through image processing. More specifically, the transparency of the slipping state image is continuously reduced from 100% to zero. After generating the driving direction image AP including the non-transparent image BP, the process proceeds to S241.

[0149] In S241, the display image generation unit F81 generates a display image DP including the traveling direction image AP generated by the synthetic image generation unit F7. During the non-slip state, the display image DP is generated including the transparent image CP. During the slip state, the display image DP is generated including the non-transparent image BP. Immediately after the wheel enters the slip state from the non-slip state, the display image DP is generated including the transparent image CP in which a semi-transparent slip state image is superimposed on the past image immediately before the slip state. After the display image DP is generated, the process proceeds to S242.

[0150] In S242, the image output unit F9 outputs the display image DP to the display 3. In the non-slip state, the display 3 displays the display image DP including the transparent image CP. In the slip state, the display 3 displays the display image DP including the non-transparent image BP. Immediately after the wheel enters the slip state from the non-slip state, the display 3 displays the display image DP, which is generated to include the transparent image CP in which the semi-transparent slip state image is superimposed on the past image immediately before the slip state. After the display image DP is output, the peripheral image display process ends. This process can be repeatedly performed until the display end condition is met, and the latest peripheral image can be continuously displayed.

[0151] The technical effect of this embodiment will be described below. According to this embodiment, the display control unit F8 displays a non-transparent image BP in a slippery state, and the non-transparent image BP shows the road surface under the vehicle floor in an opaque manner, that is, in a non-transparent manner. Therefore, it is possible to prevent the user from misunderstanding the situation under the vehicle floor in a slippery state.

[0152] In the non-skidding state, the transparent image CP is displayed. In the skidding state, the non-transparent image BP is displayed. Therefore, by recognizing whether the transparent image CP or the non-transparent image BP is displayed, the user can recognize whether the vehicle is in the skidding state or in the non-skidding state.

[0153] When the display control unit F8 switches the transparent image CP displayed in the non-slip state to the non-transparent image BP displayed in the slip state, the transparency is gradually reduced to display the non-transparent image BP. Therefore, the user can visually recognize the display switching effect as a display effect associated with the image switching. Therefore, when switching from the transparent image CP to the non-transparent image BP, the user can be prevented from mistaking the display 3 or the like for a malfunction.

[0154] The method of displaying the traveling direction image AP including the non-transparent image BP is not limited to the above method. For example, the non-transparent image BP may be displayed instead of the transparent image CP by superimposing the past image on the slip state image and gradually increasing the transparency of the past image.

[0155] Third embodiment

[0156] The third embodiment is a modification of the above-described embodiment. In this embodiment, in a slippery state, a traveling direction image AP including a non-transparent image BP instead of a transparent image CP is displayed by gradually expanding an area where the non-transparent image BP is displayed.

[0157] The following will refer to Fig.12 The flowchart shown describes the surrounding image display processing. In the following description, the same parts as the first embodiment may be omitted. The image generation ECU 70 starts the surrounding image display processing in response to the surrounding image display condition being met. The camera image is acquired in S101. Then, the processing proceeds to S102, and the display setting is acquired. Then, the processing proceeds to S111, and the vehicle state is acquired. Then, the processing enters S112, and the wheel speed difference is calculated. Then, the processing proceeds to S313.

[0158] In S313, the wheel slip determination unit F32 determines the possibility of wheel slip based on the wheel speed. When the difference between the wheel speed of the right front wheel and the wheel speed of the left front wheel is greater than a predetermined non-slip state wheel speed, the wheel slip determination unit F32 may determine that the front wheels of the host vehicle may be in a slip state. The wheel speed in the non-slip state is set to a value lower than the wheel speed in the slip state.

[0159] When the wheel speed difference is equal to or lower than the non-slip state wheel speed, it is determined that the wheel is in a non-slip state. When the wheel speed difference is greater than the non-slip state wheel speed and lower than the slip state wheel speed, the wheel may be in a slip state or in a non-slip state. That is, when the wheel speed difference is greater than the non-slip state wheel speed and lower than the slip state wheel speed, it is determined that there is a possibility of wheel slip. When the wheel speed difference is greater than or equal to the slip state wheel speed, there is a high possibility that the wheel is in a slip state. When there is a possibility of wheel slip, that is, the vehicle is in a potential slip state (S313: Yes), the process proceeds to S314. When there is no possibility of wheel slip (S313: No), it is determined that the vehicle is in a non-slip state, and the process proceeds to S118.

[0160] In S314, the image recognition unit F5 calculates the movement amount of the vehicle 9 based on the camera image. The movement amount of the vehicle 9 can be calculated by using image processing such as an optical flow method. When the movement amount of the vehicle 9 is calculated based on the camera image, the movement amount of the vehicle 9 is substantially zero even if the wheels rotate in a slipping state, and in a non-slipping state, the movement amount of the vehicle 9 increases according to the rotation of the wheels. After the movement amount of the vehicle 9 is calculated based on the camera image, the process proceeds to S315.

[0161] In S315, the wheel slip determination unit F32 determines whether the wheels of the host vehicle are in a slipping state. More specifically, when the movement amount of the vehicle 9 is substantially zero, it is determined that the vehicle is in a slipping state. When the movement amount of the vehicle 9 is equal to the movement amount calculated based on the wheel speed, it is determined that the vehicle is in a non-slipping state. When it is determined that the vehicle is in a slipping state (S315: Yes), the process proceeds to S219. When it is determined that the vehicle is in a non-slipping state (S315: No), the process proceeds to S118.

[0162] In S219, the synthetic image generation unit F7 replaces the past image with the slip state image. Then, the process proceeds to S231, and the camera image and the slip state image are synthesized into a synthetic image. Then, the process proceeds to S232, and a vehicle model is assigned to the synthetic image. Then, the process proceeds to S335.

[0163] In S335, the synthetic image generation unit F7 generates the traveling direction image AP including the non-transparent image BP. A method of generating the traveling direction image AP when the transparent image CP is switched to the non-transparent image BP to generate the traveling direction image AP will be described below.

[0164] exist Fig.13In the figure, the first driving direction image AP1 is an image obtained by synthesizing the current camera images captured by the front camera 2F, the right camera 2R, and the left camera 2L. The second driving direction image AP2 is an image including both a transparent image CP as a past image captured by the front camera 2F and a non-transparent image BP as a slip state image. The transparent image CP is maintained in the left part and the central part of the second driving direction image AP2, and the non-transparent image BP is adopted in the right part of the second driving direction image AP2. That is, the non-transparent image BP is adopted as a part of the second driving direction image AP2. Fig.13 In the embodiment, the second traveling direction image AP2 is displayed in a state where the area of ​​the transparent image CP is wider than the area of ​​the non-transparent image BP.

[0165] like Fig.14 As shown, the second driving direction image AP2 includes both a transparent image CP as a past image captured by the front camera device 2F and a non-transparent image BP as a slip state image. The transparent image CP is maintained on the left side of the second driving direction image AP2, and the non-transparent image BP is used on the right side and the center of the second driving direction image AP2. Fig.14 In the embodiment, the second traveling direction image AP2 is displayed in a state where the area of ​​the non-transparent image BP is wider than the area of ​​the transparent image CP.

[0166] When generating the driving direction image AP including the non-transparent image BP, as shown in FIG. Fig. 9 , Fig.13 and Fig.14 As shown in the sequence, the image including only the past image is gradually switched to the image including only the slipping state image through image processing. More specifically, the area of ​​the transparent image CP in the second driving direction image AP2 gradually decreases, while the area of ​​the non-transparent image BP gradually increases. The non-transparent image BP is not limited to the mode of extending from the right end to the left end of the second driving direction image AP2 as described in the above example. For example, the area of ​​the non-transparent image BP may increase from the left end to the right end in the second driving direction image AP2. For example, the area of ​​the non-transparent image BP may increase from the upper end to the lower end in the second driving direction image AP2. For example, the area of ​​the non-transparent image BP may increase from the central part to the peripheral edge in the second driving direction image AP2.

[0167] After generating the driving direction image AP including the non-transparent image BP, the process proceeds to S241 to generate the display image DP. Then, the process proceeds to S242, and the display image DP is output. After the display image DP is output, the surrounding image display process ends. The surrounding image display process may be repeatedly performed until the display end condition is satisfied, and the latest surrounding image may be continuously displayed.

[0168] The technical effect of the present embodiment will be described below. According to the above embodiment, the display control unit F8 displays the non-transparent image BP by gradually widening the area of ​​the non-transparent image BP when switching the transparent image CP to the non-transparent image BP. Therefore, when switching from the transparent image CP to the non-transparent image BP, it is possible to prevent the user from mistaking the display 3, etc. for a malfunction.

[0169] When the wheel slip determination unit F32 determines that the wheel is in a potential slip state based on the wheel speed, the wheel slip determination unit F32 further determines whether the wheel is in a slip state based on the camera image captured over time. Therefore, a high-precision slip state determination can be performed using the camera image. When it is determined that the vehicle is not in a potential slip state based on the wheel speed, the slip state determination using the camera image is not performed. The slip state determination using the camera image requires a greater processing load than the slip state determination based on the wheel speed. Therefore, compared with the case where the slip state determination is always performed using the camera image, the processing load required for the slip state determination can be reduced. Therefore, it is easy to reduce the processing load of the slip state determination and the peripheral image output, and the processing time can be shortened.

[0170] Other Implementations

[0171] The disclosure in this specification and the drawings is not limited to the exemplary embodiments. The present disclosure includes the above-mentioned embodiments and modifications made to the above-mentioned embodiments by those skilled in the art. For example, the present disclosure is not limited to the combination of parts and / or elements described in the embodiments. The present disclosure can be performed in various different combinations. The present disclosure may include additional configurations that can be added to the above-mentioned embodiments. The present disclosure also includes modifications of some parts / elements of the above-mentioned embodiments. The present disclosure also includes replacement or combination of parts and / or elements between one embodiment and another. The technical scope disclosed in the present disclosure is not limited to the above-mentioned embodiments. It should be understood that some disclosed technical scopes are indicated by the description of the claims and include every modification within the equivalent meaning and scope of the description of the claims.

[0172] The disclosure in the specification, drawings, etc. is not limited to the description of the claims. The disclosure in the specification, drawings, etc. includes the technical ideas described in the claims, and further extends to technical ideas broader than the technical ideas in the claims. Therefore, various technical ideas can be extracted from the disclosure in the specification, drawings, etc. without being limited to the description of the present disclosure.

[0173] The image generation unit and method thereof according to the present disclosure can be implemented by one or more special-purpose computers. Such a special-purpose computer can be provided in the following manner: (i) configuring (a) a processor and a memory, which are programmed to perform one or more functions implemented by a computer program; or (ii) configuring (b) a processor including one or more special-purpose hardware logic circuits; or (iii) by configuring a combination of (a) a memory and a processor programmed to perform one or more functions implemented by a computer program and (b) a processor including one or more special-purpose hardware logic circuits. The technology for implementing the function of each functional unit included in the method of the device or the device does not necessarily need to include software, and one or more hardware circuits can be used to implement all functions. In addition, the computer program can be stored as a program product in a computer-readable non-transitory tangible storage medium as instructions to be executed by a computer.

Claims

1. A peripheral image display device, comprising: A wheel speed acquisition unit (F31) that acquires the wheel speed of the vehicle (9); a wheel slip determination unit (F32) that determines a slip state or a non-slip state of a wheel of the vehicle based on the wheel speed acquired by the wheel speed acquisition unit; An image acquisition unit (F1) that acquires a plurality of camera images from respective cameras (2), each of the cameras sequentially capturing a surrounding area of ​​the vehicle; an image storage device (M1) for storing, as a past image, the camera image indicating a range in the traveling direction of the vehicle among the camera images acquired by the image acquisition unit; a composite image generating unit (F7) for generating a peripheral image indicating the periphery of the vehicle by synthesizing the camera image acquired by the image acquiring unit; as well as a display control unit (F8) for displaying the peripheral image generated by the synthetic image generation unit on a display (3), It is characterized in that In the non-slip state, the display control unit displays a transparent image that shows a portion below a floor of the vehicle in a transparent manner using the past image selected based on a current wheel speed, In the slipping state, the display control unit disables display of the transparent image showing a portion under a floor of the vehicle using the past image selected based on the current wheel speed, and In the slipping state, the display control unit displays the transparent image displayed immediately before the wheels of the vehicle enter the slipping state.

2. The peripheral image display device according to claim 1, wherein: In the slipping state, the display control unit displays a non-transparent image that does not show a portion below a floor of the vehicle in a transparent manner.

3. The peripheral image display device according to claim 2, wherein: When the display control unit switches the transparent image displayed in the non-slip state to the non-transparent image displayed in the slip state, the display control unit gradually changes transparency to display the non-transparent image.

4. The peripheral image display device according to claim 2, wherein: When the display control unit switches the transparent image displayed in the non-slip state to the non-transparent image displayed in the slip state, the display control unit gradually increases the area of ​​the non-transparent image to display the non-transparent image.

5. The peripheral image display device according to any one of claims 1 to 4, wherein: The image acquisition unit acquires a camera image in a traveling direction of the vehicle and a camera image in a left-right direction of the vehicle. The display control unit displays the peripheral image generated by using a camera image in a traveling direction of the vehicle and a camera image in a left-right direction of the vehicle.

6. The peripheral image display device according to any one of claims 1 to 4, wherein: The wheel speed acquisition unit acquires the wheel speed of the right wheel and the wheel speed of the left wheel as the wheel speeds, and The wheel slip determination unit determines the slip state of the wheel when a difference between a wheel speed of the right wheel and a wheel speed of the left wheel is equal to or greater than a predetermined slip state wheel speed.

7. The peripheral image display device according to any one of claims 1 to 4, wherein: In response to determining that there is no possibility of wheel slip based on the wheel speed, the wheel slip determination unit determines that the wheels of the vehicle are in a non-slip state, and In response to determining that there is a possibility of wheel slip based on the wheel speed, the wheel slip determination unit further determines whether the wheel of the vehicle is in the slipping state or the non-slipping state based on a change in the camera image over time.

8. A peripheral image display device, comprising: A wheel speed acquisition unit (F31) that acquires the wheel speed of the vehicle (9); a wheel slip determination unit (F32) that determines a slip state or a non-slip state of a wheel of the vehicle based on the wheel speed acquired by the wheel speed acquisition unit; An image acquisition unit (F1) that acquires a plurality of camera images from respective cameras (2), each of the cameras sequentially capturing a surrounding area of ​​the vehicle; an image storage device (M1) for storing, as a past image, the camera image indicating a range in the traveling direction of the vehicle among the camera images acquired by the image acquisition unit; a composite image generating unit (F7) for generating a peripheral image indicating the periphery of the vehicle by synthesizing the camera image acquired by the image acquiring unit; as well as a display control unit (F8) for displaying the peripheral image generated by the synthetic image generation unit on a display (3), It is characterized in that In the non-slip state, the display control unit displays a transparent image that shows a portion below a floor of the vehicle in a transparent manner using the past image selected based on a current wheel speed, in the slipping state, the display control unit invalidates display of the transparent image showing a portion below a floor of the vehicle using the past image selected based on the current wheel speed, In response to determining that there is no possibility of wheel slip based on the wheel speed, the wheel slip determination unit determines that the wheels of the vehicle are in a non-slip state, and In response to determining that there is a possibility of wheel slip based on the wheel speed, the wheel slip determination unit further determines whether the wheel of the vehicle is in the slipping state or the non-slipping state based on a change in the camera image over time.

Images