Peripheral image display device
The peripheral image display device addresses inaccuracies in vehicle surroundings by using wheel speed to determine idling and combining current and past images, ensuring reliable display of vehicle surroundings, particularly during idling.
Patent Information
- Application Number
- JP2021023678
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-17
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-02-17
AI Technical Summary
Existing peripheral image display devices for vehicles face challenges in providing reliable underfloor images due to high processing loads and inaccuracies in estimating vehicle movement, leading to potential misidentification of the vehicle's surroundings, especially when wheels are idling.
A peripheral image display device that includes a wheel speed acquisition unit to determine if the vehicle is idling, using past camera images when wheels are idling, and combining current and past images to generate a transparent underfloor image when not idling, ensuring accurate display of vehicle surroundings.
The device provides a highly reliable peripheral image display by reducing processing load and minimizing inaccuracies, allowing users to accurately understand their surroundings, especially during idling conditions.
Smart Images

Figure 0007585853000001 
Figure 0007585853000002 
Figure 0007585853000003
Abstract
Description
[Technical Field]
[0001] The disclosure herein relates to a peripheral image display device. [Background technology]
[0002] Patent Document 1 discloses a surroundings monitoring device that displays the underfloor area, which is a blind spot area at the vehicle's position, on a display device. The contents of the prior art document are incorporated by reference as explanations of the technical elements in this specification. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-21653 Summary of the Invention [Problem to be solved by the invention]
[0004] In the configuration of the prior art document, the vehicle's movement amount and position are estimated based on information such as optical flow, wheel speed, and GPS, and a past image is displayed as an underfloor image. However, image processing such as optical flow can have a large processing load and require a long time to estimate the movement amount. Furthermore, when estimating the movement amount based on wheel speed, wheel spin can result in the estimated movement amount significantly differing from the actual movement amount. If an incorrect image is displayed as the underfloor image, the user may misinterpret the underfloor situation, which may reduce the reliability of the peripheral image. Further improvements are required for peripheral image display devices in the above-mentioned and other unmentioned respects.
[0005] One disclosed object is to provide a highly reliable peripheral image display device. [Means for solving the problem]
[0006] The peripheral image display device disclosed herein includes a wheel speed acquisition unit (F31) that acquires a wheel speed of a vehicle (9); a wheel spin determination unit (F32) that determines whether the wheels are spinning or not based on the wheel speeds acquired by the wheel speed acquisition unit; an image acquisition unit (F1) that acquires a plurality of camera images obtained by a plurality of cameras (2) that capture images of the periphery of the vehicle; an image storage unit (M1) that stores, as a past image, a camera image captured in the direction of travel of the vehicle, among the camera images acquired by the image acquisition unit; a composite image generating unit (F7) that generates a surrounding image showing the surroundings of the vehicle by synthesizing the multiple camera images acquired by the image acquiring unit; a display control unit (F8) that displays the peripheral image generated by the composite image generation unit on a display (3); When the wheels are not spinning, the display control unit displays a transparent image in which the floor of the vehicle is made transparent using a past image selected based on the current wheel speed. and the surrounding image including the current camera image. If the wheel is spinning, a past image selected based on the current wheel speed is displayed. Display surrounding images including the current camera image but not including the current camera image .
[0007] The disclosed peripheral image display device includes a display control unit that displays a see-through image of the vehicle floor using a past image selected based on the current wheel speed when the vehicle is not spinning, and does not display a see-through image using a past image selected based on the current wheel speed when the vehicle is spinning. This prevents an incorrect image from being displayed as an underfloor image during spinning, which can lead to the user misunderstanding the underfloor situation. Therefore, a highly reliable peripheral image display device can be provided.
[0008] The various aspects disclosed in this specification employ different technical means to achieve their respective objectives. The reference numerals in parentheses in the claims and in this section are intended to exemplify correspondences with the following embodiments and are not intended to limit the technical scope. The objectives, features, and advantages disclosed in this specification will become more apparent by reference to the following detailed description and the accompanying drawings. [Brief explanation of the drawings]
[0009] [Figure 1] 1 is a block diagram showing the overall configuration of a peripheral image display system; [Figure 2] FIG. 2 is an explanatory diagram for explaining the installation positions and imaging ranges of each camera. [Figure 3] FIG. 2 is a block diagram showing the configuration of an image generation ECU 70. [Figure 4] FIG. 10 is an explanatory diagram for explaining a projection plane used to generate a traveling direction image. [Figure 5] FIG. 1 shows a travel image with a transmission image. [Figure 6] FIG. 10 is a diagram showing a display image. [Figure 7] 10 is a flowchart relating to control of the peripheral image display system. [Figure 8] 10 is a flowchart relating to control of a peripheral image display system in the second embodiment. [Figure 9] FIG. 10 is a diagram showing a travel direction image having a transmission image in the second embodiment. [Figure 10] FIG. 11 is a diagram showing a travel direction image in the middle of switching a transparent image to an opaque image in the second embodiment. [Figure 11] FIG. 10 is a diagram showing a travel direction image having an opaque image in the second embodiment. [Figure 12] 10 is a flowchart relating to control of a peripheral image display system in the third embodiment. [Figure 13] FIG. 11 is a diagram showing a travel direction image in the middle of switching a transparent image to an opaque image in the third embodiment. [Figure 14] FIG. 11 is a diagram showing a travel direction image in the middle of switching a transparent image to an opaque image in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0010] Several embodiments will be described with reference to the drawings. In several embodiments, functionally and / or structurally corresponding and / or associated parts may be assigned the same reference numerals or reference numerals that differ in the hundredth or higher digit. For corresponding and / or associated parts, reference may be made to the descriptions of other embodiments.
[0011] First embodiment In FIG. 1, a peripheral image display system 1 displays a peripheral image, which is an image of the surroundings of a vehicle 9 equipped with the system, on a display 3. Here, the surroundings of the vehicle 9 may include positions around the vehicle 9 in various directions, such as forward, backward, to the side, and downward. Furthermore, the peripheral image does not need to include an image of the entire periphery of the vehicle 9; an image showing only the front or only the rear may also be a peripheral image. Hereinafter, the vehicle 9 equipped with the peripheral image display system 1 may be referred to as the host vehicle.
[0012] The host vehicle is, for example, a four-wheel drive vehicle designed to travel not only on roads paved with asphalt or the like, but also on unpaved roads. The host vehicle has a normal mode and an off-road mode as driving modes. The normal mode is a driving mode suitable for on-road driving, i.e., paved roads. The off-road mode is a driving mode suitable for off-road driving, i.e., unpaved roads. Each driving mode has a different method for controlling the distribution of driving force to the front, rear, left, and right wheels. Here, "off-road" primarily refers to uneven ground such as rocky roads. Of course, "off-road" can also be interpreted as ground other than on-road, i.e., unmaintained ground in general. The present disclosure can also be applied to cases where off-road driving is not designed. The host vehicle may be a gasoline-powered vehicle using an engine as a driving source, or an electric vehicle or hybrid car using a motor as a driving source.
[0013] In the following description, the front-rear, left-right, and up-down directions are defined relative to the host vehicle. Specifically, the front-rear direction corresponds to the longitudinal direction of the host vehicle. The left-right direction corresponds to the width direction of the host vehicle. The up-down direction corresponds to the height direction of the host vehicle. From another perspective, the up-down direction corresponds to the direction perpendicular to a plane parallel to the front-rear and left-right directions. In the present disclosure, a plane perpendicular to the height direction of the host vehicle is also referred to as the vehicle horizontal plane. Furthermore, the direction perpendicular to the height direction of the host vehicle, including the front-rear and left-right directions, is also referred to as the vehicle horizontal direction.
[0014] Additionally, in the present disclosure, "parallel" does not necessarily mean a completely parallel state. A state tilted by several degrees to approximately 20 degrees from a completely parallel state may also be considered parallel. In other words, "parallel" may include a substantially parallel state. Similarly, the term "perpendicular" in the present disclosure does not necessarily mean a completely perpendicular state.
[0015] The peripheral image display system 1 includes an image generation ECU 70, multiple cameras 2, a display 3, a touch panel 4, operation buttons 5, and a vehicle state sensor 6. ECU in the component names is an abbreviation for Electronic Control Unit, and means an electronic control device.
[0016] The image generation ECU 70 is communicatively connected to each of the multiple cameras 2, the display 3, the touch panel 4, the operation buttons 5, and the vehicle condition sensors 6. Each of the above-mentioned devices and the image generation ECU 70 may be individually connected by a dedicated line, or may be connected via a communication network established within the vehicle. For example, the camera 2 and the image generation ECU 70 may be directly connected by a dedicated video signal line.
[0017] The image generation ECU 70 is an ECU that generates a composite image of the vehicle periphery viewed from an arbitrary viewpoint based on image data captured by each camera 2. The image generation ECU 70 is an ECU that displays the generated composite image on the display 3. The image generation ECU 70 generates and displays the composite image, thereby assisting the driving operation of the vehicle 9. The image generation ECU 70 is realized using a computer. The image generation ECU 70 includes a processing unit 71, a RAM 72, a storage 73, a communication IF 74, and a bus line that connects these components. The image generation ECU 70 provides an example of a peripheral image display device.
[0018] The processing unit 71 is hardware for arithmetic processing coupled to the RAM 72. The processing unit 71 is, for example, a CPU. The processing unit 71 accesses the RAM 72 to execute various processes for realizing the functions of each functional unit, which will be described later. The RAM 72 is a volatile storage medium.
[0019] The storage 73 includes a non-volatile storage medium such as a flash memory. The storage 73 stores an image generation program as firmware and various drawing data for generating a composite image. The drawing data includes data indicating the shape of the projection surface TS (described later), 3D model data indicating the vehicle's exterior, and 3D model data indicating components of the vehicle. Examples of components for which 3D model data is provided include tires, a steering wheel, an instrument panel, pillars, and body panels. Execution of the image generation program by the processing unit 71 corresponds to execution of a display control method corresponding to the image generation program.
[0020] The communication IF 74 is a circuit module that functions as an interface for communicating with other devices. The communication IF 74 is realized using analog circuit elements, ICs, etc. Details of the image generation ECU 70 will be described separately later.
[0021] The cameras 2 are in-vehicle cameras that capture images of the surroundings of the host vehicle and output the captured image data to the image generation ECU 70. Each camera 2 has at least a lens and an imaging element, and acquires images showing the surroundings of the host vehicle. The multiple cameras 2 are attached to the host vehicle at mounting positions so that they each capture images of different ranges. The surroundings image display system 1 of this embodiment includes the cameras 2 as shown in FIG. 2: a front camera 2F, a rear camera 2B, a right side camera 2R, and a left side camera 2L. These four cameras 2 are arranged at different positions on the host vehicle and capture images of the surroundings of the host vehicle in different directions. Specifically, this is as follows.
[0022] The front camera 2F is an imaging device that captures an image in front of the vehicle. The front camera 2F is attached to the front end of the vehicle, such as the front grille, with its optical axis 2Fa facing the front direction of the vehicle. When the vehicle 9 is traveling forward, the front camera 2F functions as a camera 2 that captures an image in the traveling direction.
[0023] The rear camera 2B is an imaging device that captures images behind the vehicle. The rear camera 2B is disposed in a predetermined position on the rear of the vehicle body, such as near the rear license plate or the rear window, with its optical axis 2Ba facing rearward. When the vehicle 9 is traveling backward, the rear camera 2B functions as a camera 2 that captures images in the traveling direction.
[0024] The right side camera 2R is an imaging device that captures images of the right side of the vehicle. The right side camera 2R is attached to the right side mirror with its optical axis 2Ra facing toward the right side of the vehicle. The left side camera 2L is an imaging device that captures images of the left side of the vehicle. The left side camera 2L is attached to the left side mirror with its optical axis 2La facing toward the left side of the vehicle.
[0025] The cameras 2 employ wide-angle lenses such as fisheye lenses, each with a viewing angle θ of 180° or more. Therefore, by using four cameras 2, it is possible to capture images of the entire surroundings of the vehicle. The mounting positions of the above-mentioned cameras 2 can be changed as needed. The front camera 2F may be mounted on the rearview mirror or the upper end of the windshield. The right side camera 2R and the left side camera 2L may be positioned near the base of the A-pillar or B-pillar. The cameras 2 may be retrofitted, for example, on the roof, on the dashboard, or near the window frame.
[0026] Mounting mode data, which is data indicating the mounting position and mounting posture of each camera 2 on the vehicle 9, is stored in the storage 73. The mounting position of each camera 2 may be expressed, for example, as a point on a three-dimensional vehicle coordinate system, which is a three-dimensional coordinate system centered on an arbitrary position on the vehicle 9. The X-axis forming the vehicle three-dimensional coordinate system may be, for example, an axis parallel to the left-right direction of the vehicle 9. The Y-axis forming the vehicle three-dimensional coordinate system may be an axis parallel to the front-rear direction. The Z-axis forming the vehicle three-dimensional coordinate system may be an axis parallel to the vehicle height direction. The right direction of the vehicle corresponds to the positive X-axis direction, the front of the vehicle corresponds to the positive Y-axis direction, and the top of the vehicle corresponds to the positive Z-axis direction. The center of the vehicle three-dimensional coordinate system may be, for example, the center of the rear wheel axle.
[0027] In FIG. 1, the display 3 is a device that includes a thin display panel such as a liquid crystal display, and displays various information and images. The display 3 is disposed on an instrument panel of the vehicle so that the user can view the screen. The display 3 may be integrated with the image generation ECU 70 by being disposed in the same housing as the image generation ECU 70. Of course, the display 3 may be a device separate from the image generation ECU 70. The display 3 includes a touch panel 4 stacked on the display panel, and is configured to be able to receive user operations.
[0028] The touch panel 4 is, for example, a capacitance type, and outputs a signal indicating the position touched by the user. The user here refers primarily to the driver, who is a passenger sitting in the driver's seat. In addition to the driver, the user can also include a passenger sitting in the passenger seat.
[0029] The operation button 5 is an operation component that accepts user operations on the display content of the display 3. The operation button 5 is a switch for displaying the composite image generated by the image generation ECU 70 on the display 3, changing the viewpoint position of the composite image, etc. The operation button 5 is provided, for example, on the steering wheel of the vehicle, and accepts operations mainly from the driver. The user can perform various operations on the peripheral image display system 1 via the operation button 5 and the touch panel 4. When the user operates the operation button 5 or the touch panel 4, an operation signal indicating the content of the operation is input to the image generation ECU 70. The operation button 5 may be arranged on an instrument panel.
[0030] The peripheral image display system 1 may include a driving mode switch as an operating component for the driver to switch the driving mode of the vehicle 9 to the off-road mode. The operating component for switching the driving mode may be a dial type. The function of the driving mode switch may be provided by a shift lever.
[0031] The vehicle state sensor 6 is a sensor that detects state quantities related to the driving control of the host vehicle. The vehicle state sensor 6 includes a wheel speed sensor 6s. The wheel speed sensor 6s is a sensor that detects the wheel speed from the rotation of the wheels of the host vehicle. A sensor that detects the rotation of the wheels as a change in a pulse signal can be used as the wheel speed sensor 6s. In this case, the number of pulse signals obtained per unit time increases as the wheel speed increases. The wheel speed sensor 6s does not need to be a sensor that detects the wheel speed itself as speed, but may be a sensor that detects state quantities used to calculate the wheel speed. The wheel speed sensor 6s detects the wheel speed of each wheel of the host vehicle. For example, if the host vehicle has four wheels, a right front wheel, a left front wheel, a right rear wheel, and a left rear wheel, the wheel speed of each of the four wheels is detected individually.
[0032] In addition to the wheel speed sensor 6s, the vehicle state sensor 6 also includes a shift sensor, a vehicle speed sensor, a steering angle sensor, an acceleration sensor, etc. The shift sensor is a sensor that detects the shift position. The vehicle speed sensor is a sensor that detects the traveling speed of the vehicle. The steering angle sensor is a sensor that detects the rotation angle of the steering wheel. The acceleration sensor is a sensor that detects acceleration acting on the vehicle in at least one of the longitudinal, lateral, and vertical directions. Here, it is assumed that a three-axis acceleration sensor is used as the acceleration sensor. The detected value of the acceleration sensor can be used as information for determining the vehicle's attitude relative to a horizontal plane.
[0033] The type of sensor used by the peripheral image display system 1 as the vehicle state sensor 6 may be designed as appropriate, and it is not necessary to include all of the sensors described above. In addition, a vehicle height sensor, a gyro sensor, an azimuth angle sensor, etc. may be included in the vehicle state sensor 6. Each sensor outputs data indicating the current value of the state quantity to be detected to the image generation ECU 70.
[0034] 3, the image generation ECU 70 includes, as functional units, an image acquisition unit F1, an operation reception unit F2, a vehicle state acquisition unit F3, a display setting acquisition unit F4, an image recognition unit F5, a composite image generation unit F7, a display control unit F8, and an image output unit F9. The image generation ECU 70 also includes an image storage unit M1, which is a memory for temporarily storing image data. The image storage unit M1 is realized, for example, by using a portion of the storage area of the RAM 72. The image storage unit M1 can store images from each camera 2 in association with position information and orientation information of the vehicle 9 at the time the image was captured.
[0035] The image acquisition unit F1 acquires image data captured by each of the four cameras 2. Image data of the entire surroundings of the vehicle is obtained by combining the image data from each of the cameras 2. The image acquisition unit F1 converts the image signals input from the cameras 2 into digital image data in a data format, and then outputs the data to the image recognition unit F5 and the composite image generation unit F7.
[0036] Furthermore, the image storage unit M1 stores image data captured by each camera 2, for example, each time the vehicle moves a storage distance. At this time, the stored image data is associated with separately acquired position information and vehicle body posture information of the vehicle. The storage distance can be, for example, 0.1 m, 0.3 m, or 0.5 m. If the storage distance is 0.1 m, the next image data to be stored and the most recent stored image data will be image data captured within a range 0.1 m apart in the direction of travel. A storage time may be set instead of a storage distance, and image data captured by each camera 2 may be stored each time the storage time elapses.
[0037] For example, when the vehicle is moving forward, the image storage unit M1 stores image data from the front camera 2F, which captures an image of the ground from directly below the front end of the vehicle to 3 m ahead, in the image storage unit M1. When the vehicle is moving backward, the image storage unit M1 stores image data from the rear camera 2B, which captures an image of the ground from directly below the rear end of the vehicle to 3 m behind.
[0038] Various modes for storing image data in the image storage unit M1 can be adopted. For example, the image acquisition unit F1 overwrites and stores new data in the oldest updated area in the image storage unit M1. That is, the image storage unit M1 may be configured as a ring buffer. A ring buffer is a storage area logically arranged in a ring shape. The image acquisition unit F1 may also be configured to store image frames acquired within the most recent storage time in an area separate from the image data for each storage distance described above. The image data stored in the image storage unit M1 is referenced by the composite image generation unit F7. The image acquisition unit F1 may perform image processing such as distortion correction, enlargement / reduction, and cropping according to the lens characteristics on the images acquired from the camera 2. In this case, the image data after image processing is stored in the image storage unit M1.
[0039] The operation reception unit F2 receives operation signals output from the operation buttons 5 and the touch panel 4. The operation signals are signals that indicate the content of the user's operation on the operation buttons 5 or the touch panel 4. In response to this, the operation reception unit F2 receives user operations on the display of image data such as a composite image. The operation reception unit F2 inputs data corresponding to the received operation signals to the display control unit F8.
[0040] The vehicle state acquisition unit F3 is configured to acquire information indicating the state of the host vehicle from other devices provided on the host vehicle, such as the vehicle state sensor 6. The vehicle state acquisition unit F3 includes a wheel speed acquisition unit F31 and a spin determination unit F32. The wheel speed acquisition unit F31 acquires the wheel speed of each wheel detected by the wheel speed sensor 6s. For example, the wheel speed acquisition unit F31 acquires the wheel speed of the right front wheel, the wheel speed of the left front wheel, the wheel speed of the right rear wheel, and the wheel speed of the left rear wheel.
[0041] The wheel spin determination unit F32 determines whether the wheels of the host vehicle are spinning. For example, when the front wheels are spinning, the right and left front wheels have significantly different levels of friction with the ground. This results in a large difference between the wheel speeds of the right and left front wheels. Therefore, if the difference between the wheel speeds of the right and left front wheels is equal to or greater than the wheel speed during spinning, it can be determined that the wheels are spinning. Even when the vehicle is not spinning, the wheel speeds of the right and left front wheels may differ from each other when turning a corner. For this reason, it is preferable to set the wheel speed during spin to a value that allows for appropriate determination of whether the vehicle is spinning or not.
[0042] The method for determining whether the wheels are spinning is not limited to the above-described method. Furthermore, instead of the spin determination unit F32 determining whether the wheels are spinning based on the wheel speed, an ECU other than the image generation ECU 70 may determine whether the wheels are spinning. In this case, the spin determination unit F32 obtains a spin determination result determined by an ECU other than the image generation ECU 70.
[0043] The vehicle state acquisition unit F3 acquires the shift position, acceleration in each detection axis direction, vehicle speed, steering angle, and the like from various sensors. The vehicle state acquisition unit F3 may acquire various state quantities from another ECU. For example, information such as the steering angle may be acquired from an ECU constituting a steering system. The vehicle state acquisition unit F3 may also acquire the travel distance and the amount of change in orientation over time from the image recognition unit F5. In other words, the vehicle state acquisition unit F3 can acquire various information regarding the state of the host vehicle not only from sensors but also from other ECUs, the image recognition unit F5, and the like. The vehicle state acquisition unit F3 may also be configured to combine multiple types of information to calculate the travel distance of the vehicle 9, the amount of change in orientation of the vehicle 9, and the like. The orientation of the vehicle 9 may include not only the yaw angle, which is the direction in which the vehicle 9 is facing, but also the pitch angle, roll angle, and the like.
[0044] The display setting acquisition unit F4 acquires the display settings of the image to be displayed on the display 3. An example of the display setting is a setting related to the brightness of the display 3. An example of the display setting is a setting related to the timing of display. An example of the display setting is a setting related to the viewpoint position of the image to be displayed. An example of the display setting is a setting related to the presentation when switching the display. The display setting can be changed by the user operating the touch panel 4 or the operation button 5.
[0045] The image recognition unit F5 is configured to detect the position and type of a detection target by analyzing the image input from the camera 2. The image recognition unit F5 functions as a classifier that identifies the type of object based on, for example, the feature vector of the image. The image recognition unit F5 identifies objects using, for example, deep learning-based convolutional neural network (CNN) or deep neural network (DNN) technology. Detectable objects include pedestrians, other vehicles, road markings such as lane markings on paved roads, and road edges. The lane markings can include lines indicating parking spaces, etc.
[0046] The image recognition unit F5 is configured to estimate the amount of movement and change in orientation of the vehicle based on the amount of change in position of feature points appropriately set on the image between image frames. For example, the image recognition unit F5 can estimate the current position relative to a reference point, the amount of change in the vehicle's orientation, the yaw rate, etc. using an optical flow method, which is a type of frame difference method. Optical flow is information that represents the movement of objects captured in image data as vectors. The object recognition results and estimation results of the vehicle's position, etc., by the image recognition unit F5 are output to the display control unit F8, the vehicle state acquisition unit F3, and the composite image generation unit F7.
[0047] The composite image generation unit F7 is configured to perform image processing to generate composite images such as a traveling direction image AP and a transmission image CP, which will be described later. The composite image generation unit F7 projects multiple image data captured by the camera 2 onto a virtual projection surface TS, and generates a composite image viewed from a virtual viewpoint VP using the data on the projection surface TS.
[0048] As conceptually shown in FIG. 4, the projection surface TS is a virtual three-dimensional surface corresponding to the area surrounding the host vehicle. The central area of the projection surface TS is defined as the vehicle area where the host vehicle is located. The vehicle area is set as, for example, a rectangular flat surface. The vehicle area is set to include an area that overlaps with the host vehicle in the height direction of the host vehicle, in other words, an area that cannot be directly imaged by the camera 2. The projection surface TS forms a flat surface along the horizontal direction of the vehicle near the vehicle area, and forms a curved surface that becomes increasingly inclined as it moves away from the vehicle area. In other words, the projection surface TS forms a curved surface that is convex downward as a whole. Projecting a camera image onto the projection surface TS is equivalent to texture mapping the camera image onto the projection surface TS.
[0049] The shape of the projection surface TS can be changed as appropriate. The shape of the projection surface TS may be set so that the entire area is a horizontal plane. The shape of the projection surface TS may be set so that it does not have a flat area, but has a downwardly convex curved area starting from the edge of the vehicle area.
[0050] The operation of the composite image generation unit F7 is controlled by the display control unit F8. For example, the virtual viewpoint VP used to generate the composite image is controlled by the display control unit F8. The composite image generation unit F7 includes a camera image synthesis unit F71 and a vehicle model assignment unit F72. The camera image synthesis unit F71 is configured to project each camera image onto the projection surface TS described above. The vehicle model assignment unit F72 is configured to place a model image of the vehicle 9 at a predetermined position on the projection surface TS onto which the camera image is projected.
[0051] The display control unit F8 is configured to comprehensively control the entire image generation ECU 70. For example, the display control unit F8 controls the composite image generation unit F7 and the display image generation unit F81 based on information input from the operation reception unit F2 and the vehicle state acquisition unit F3. In this way, a composite image and a display image DP are generated according to the running state of the vehicle and user settings.
[0052] The display control unit F8 determines whether the vehicle is traveling forward or backward based on, for example, a signal from a shift sensor or the rotation direction of the tires.The display control unit F8 determines whether the driving environment is off-road based on an input signal from a driving mode switch.
[0053] The display control unit F8 sets the position and line of sight of the virtual viewpoint VP when generating a composite image (described later) based on the traveling direction of the vehicle or signals from the touch panel 4 and the operation button 5. Possible patterns of the virtual viewpoint VP that can be set include a bird's-eye view and a driver's view. Figure 4 illustrates an example of the viewpoint position when the virtual viewpoint VP is a bird's-eye view.
[0054] The bird's-eye view is a setting pattern of a virtual viewpoint VP in which the viewpoint position is directly above the vehicle and the line of sight is directly below. The bird's-eye view can be applied when generating a bird's-eye image, which is an image of the vehicle and its surroundings viewed from directly above the vehicle. The viewing angle of the bird's-eye view can be adjusted as appropriate to include the surroundings of the vehicle. The viewpoint position of the bird's-eye view is not limited to directly above the vehicle, but may be a position behind, in front, or laterally shifted from directly above the vehicle. The bird's-eye view is an example of an outside-cabin viewpoint, which is a viewpoint in which the virtual viewpoint VP is located outside the vehicle cabin.
[0055] The driver's viewpoint is a setting pattern of a virtual viewpoint VP in which the viewpoint position is set at the expected position of the driver's eyes in the vehicle cabin. The line of sight of the driver's viewpoint can be set, for example, in a diagonally downward direction forward so as to include the vicinity of the front wheels. The diagonally downward direction can be, for example, a direction directed downward at an angle of about 20 to 30 degrees from the horizontal plane of the vehicle. The line of sight of the driver's viewpoint may be configured to be changeable to any direction based on a user operation on the touch panel 4, while the diagonally downward direction forward is used as the reference direction. The viewing angle of the driver's viewpoint can also be adjusted as appropriate so as to include the vicinity of the front wheels.
[0056] The assumed position of the driver's eyes is set, for example, near the headrest of the driver's seat. The assumed position of the driver's eyes can be determined by using an iris set for each vehicle model. The iris is a virtual spatial region defined for each vehicle model. The iris is set in a virtual ellipsoid shape based on an eye range that statistically represents the spatial distribution of the occupant's eye points. The driver's viewpoint corresponds to an example of an in-vehicle viewpoint, which is a viewpoint in which the virtual viewpoint VP is placed inside the vehicle cabin. The position of the driver's viewpoint may also be located at a position shifted from the assumed position of the driver's eyes. For example, the driver's viewpoint may be located at a position shifted toward the passenger seat from the assumed position of the driver's eyes, for example, at a position midway between the driver's seat and the passenger seat.
[0057] The image generation ECU 70 of this embodiment has, as an operation mode, a peripheral image display mode in which a traveling direction image AP is displayed as shown in Fig. 5. The method for generating the traveling direction image AP will be described in detail later.
[0058] The traveling direction image AP may include a transparent image CP, which is a composite image showing the road surface through the underfloor, which is the bottom of the vehicle body. When the traveling direction image AP includes a transparent image CP showing the underfloor, the peripheral image display mode can also be called an underfloor display mode. The display control unit F8 can switch the display mode based on at least one of the shift lever being set to a specific position and a user operation. For example, the display control unit F8 sets the display mode to the peripheral image display mode when a peripheral image display condition described below is satisfied. In the following, the traveling direction image AP is assumed to be an image generated when the virtual viewpoint VP is set to the driver's viewpoint.
[0059] The display image generation unit F81 generates a display image DP as shown in Fig. 6. The display image DP is an image to be displayed on the display 3. The display image DP includes a traveling direction image AP and an icon image 3i. The icon image 3i is an image that symbolically indicates the range displayed on the display 3 as the peripheral image. The figure shows an example of an image in which the area ahead of the vehicle is the range of the peripheral image.
[0060] The display image generation unit F81 generates a display image DP using the composite image generated by the composite image generation unit F7, the camera image acquired by the image acquisition unit F1, the icon image 3i, etc. The combination of images included in the display image DP is determined by the display control unit F8 according to the traveling direction of the vehicle and user operations on the touch panel 4, etc.
[0061] The image output unit F9 converts the display image DP generated by the display image generation unit F81 into a video signal, outputs the video signal to the display 3, and displays the display image DP on the display 3. As a result, the display image DP including a composite image showing the surroundings of the host vehicle as seen from the virtual viewpoint VP is displayed on the display 3.
[0062] The following describes the operation of the composite image generation unit F7 when generating a traveling direction image AP including a transmission image CP. The camera image synthesis unit F71 projects each camera image acquired by the image acquisition unit F1 onto a projection surface TS in a virtual three-dimensional space as shown in Fig. 4. The projection positions of each camera image relative to the projection surface TS are associated in advance by correspondence information such as table data.
[0063] For example, the camera image synthesis unit F71 projects image data from the front camera 2F onto a front area of the projection surface TS. Furthermore, the camera image synthesis unit F71 projects image data from the rear camera 2B onto a rear area of the projection surface TS. Furthermore, the camera image synthesis unit F71 projects image data from the right side camera 2R onto a right area of the projection surface TS, and projects image data from the left side camera 2L onto a left area of the projection surface TS.
[0064] Various methods can be employed for the overlapping region, which is a region that is captured in an overlapping manner by multiple cameras 2. For example, a method of blending the image data from multiple cameras 2 at an appropriate ratio or a method of joining the image data from multiple cameras 2 at a boundary line can be employed.
[0065] Furthermore, an image of a portion estimated to be located in the vehicle area is extracted from the past images stored in the image storage unit M1, and is projected onto the vehicle area after being subjected to rotation correction, etc. The image projected onto the vehicle area is an image of the area located directly below the vehicle, that is, an image of the underfloor, and is therefore sometimes referred to as an underfloor image.
[0066] For example, when the vehicle is moving forward, a front camera image captured 3 meters behind the current position is used as a past image to generate an underfloor image. When the vehicle is moving backward, a rear camera image captured 3 meters ahead of the current position is used as a past image to generate an underfloor image. In other words, an underfloor image is generated using an image taken when the vehicle was located in a position opposite the current position in the traveling direction.
[0067] When the vehicle is moving forward, an underfloor image may be generated using front camera images captured at multiple points in time from a position 3 m behind the current position to the current position as past images. In this case, one underfloor image is generated from multiple past images. The same can be done when the vehicle is reversing. Image data of past images to be projected onto the underfloor area, which is the vehicle area, may be updated as the vehicle 9 moves.
[0068] Next, under the control of the display control unit F8, the composite image generation unit F7 sets a virtual viewpoint VP in the three-dimensional space including the projection surface TS. The composite image generation unit F7 can set the virtual viewpoint VP at an arbitrary viewpoint position in the three-dimensional space, facing an arbitrary line of sight.
[0069] The vehicle model assignment unit F72 then places various image elements of the vehicle in a virtual three-dimensional space including the projection surface TS onto which the camera image is projected. If the virtual viewpoint VP is located outside the vehicle cabin, such as from a bird's-eye view, a vehicle boundary line Lvc indicating the range of the road surface where the bottom of the vehicle is located is assigned to the vehicle area. The vehicle boundary line Lvc is a line that indicates the outline of the area formed by projecting the vehicle body perpendicularly onto the road surface, and corresponds to the line that indicates the outline of the vehicle 9 from a bird's-eye view.
[0070] Using Figure 5, we will explain the forward direction image AP that is generated when the virtual viewpoint VP is located inside the vehicle cabin, such as the driver's viewpoint. This figure shows the forward direction image AP assuming that the vehicle is located in a parking lot separated by parking spaces. The forward direction image AP is an image that mainly shows the surroundings of the vehicle in the direction of travel.
[0071] The traveling direction image AP is composed of a first traveling direction image AP1 and a second traveling direction image AP2. The first traveling direction image AP1 is a portion onto which a current camera image captured by camera 2 is projected. The second traveling direction image AP2 is a portion onto which a past image, which is a past camera image captured by camera 2, is projected. In summary, the first traveling direction image AP1 is an image of a range that can be captured in real time using camera 2, and the second traveling direction image AP2 is an image of a range that cannot be captured in real time. An underfloor image showing the road surface in the vehicle area is a range that cannot be captured in real time using camera 2, and constitutes the second traveling direction image AP2. In the figure, the boundary between the first traveling direction image AP1 and the second traveling direction image AP2 is indicated by a two-dot chain line.
[0072] When the virtual viewpoint VP is the driver's viewpoint, the vehicle model assignment unit F72 places, in addition to the vehicle boundary line Lvc, a 3D tire model Pt, a 3D body model, a 3D interior model, and the like at predetermined positions in three-dimensional space. However, the figure illustrates a case where only the 3D tire model Pt is placed. The 3D tire model Pt is a 3D model of a tire and is set to be translucent overall. The 3D tire model Pt may be composed only of an opaque or translucent outline. In other words, the parts of the 3D tire model Pt other than the outline can be set to be colorless and transparent.
[0073] Here, semi-transparency is not limited to a state where transparency is 50%, but can also include, for example, a state where transparency is 80%. In other words, the expression "semi-transparency" can include a level where its presence is dimly indicated. Transparency is a parameter where a higher value indicates greater transparency, and in this disclosure, 100% transparency is considered to be completely transparent. The transparency or opacity of an image element is a concept that corresponds to the alpha value of a pixel. Generally, the smaller the alpha value, the greater the degree of transparency.
[0074] The 3D body model is a 3D model of the vehicle body, and is set to be translucent overall. The 3D body model may be composed only of opaque or translucent outlines. In other words, the parts of the 3D body model other than the outlines may be set to be colorless and transparent. The 3D interior model is a 3D model of the instrument panel, steering wheel, A-pillar, etc.
[0075] The 3D tire model Pt, 3D body model, 3D interior model, etc. are 3D models that represent components of the vehicle and can be called component models. The 3D models placed in the three-dimensional space used for generating a synthetic image can be changed as appropriate. For example, the placement of the 3D tire model Pt, 3D body model, 3D interior model, etc. may be omitted.
[0076] Additionally, the vehicle model assignment unit F72 may draw scale lines indicating the distance from the vehicle end and vehicle width lines indicating the vehicle width. Furthermore, the vehicle model assignment unit F72 may draw planned trajectory lines, which are lines indicating the planned driving trajectory according to the steering angle of the vehicle 9, as other image elements related to the vehicle 9. The vehicle width lines may also serve as planned trajectory lines. The vehicle boundary line Lvc, planned driving trajectory lines, etc. are images that indicate information about the vehicle 9, and therefore can be called vehicle information images. Furthermore, the planned trajectory lines, scale lines, vehicle width lines, etc. can also function as information to guide the driver's driving operations. For this reason, display lines such as planned trajectory lines can be called guide lines.
[0077] The composite image generation unit F7 then performs rendering on various image elements present in a three-dimensional space including the projection surface TS according to the virtual viewpoint VP. Furthermore, the composite image generation unit F7 uses the image data into which the various image elements have been rendered to extract image elements that are within a predetermined viewing angle as viewed from the virtual viewpoint VP. As a result, the composite image generation unit F7 generates a composite image showing the underfloor of the vehicle and the area around the vehicle as viewed from the virtual viewpoint VP. For example, an underfloor image with the bottom of the vehicle body transparent may be generated as a transparent image CP. While the above describes an example of generating a composite image in the order in which a camera image is projected onto the projection surface TS and then image elements are added, each process may be executed in parallel.
[0078] Furthermore, various methods can be employed to generate a composite image. For example, the road surface image generated based on the camera image, the vehicle boundary line Lvc, and the three-dimensional model of the components constituting the vehicle 9 may each be arranged on a separate layer and processed. For example, the composite image may be generated by combining three layers: a camera image layer, a boundary line layer, and a 3D model layer. Here, the camera image layer is a layer including a road surface image, which is an image of the projection surface TS onto which the camera image is projected. The boundary line layer is a layer including the vehicle boundary line Lvc. The 3D model layer is a layer including a three-dimensional model such as a 3D tire model Pt. When combining multiple layers, the camera image layer may be the lowest background layer, and the boundary line layer and the 3D model layer may be layered on top of it, in that order. Of course, the layer configuration can also be changed as appropriate. When using multiple layers to handle each image element individually, the composite image is ultimately an image obtained by integrating the layers.
[0079] The following describes the display image DP displayed on the display 3. As shown in Fig. 6, the display image generating unit F81 generates an image including a traveling direction image AP and an icon image 3i as the display image DP.
[0080] The display contents of the display image DP are not limited to the above example and can be changed as appropriate. For example, a right-side camera image captured by the right-side camera 2R and a left-side camera image captured by the left-side camera 2L may be displayed side by side on the traveling direction image AP. Alternatively, the traveling direction image AP, the right-side camera image, and the left-side camera image may be displayed as a single continuous panoramic image. This layout of the display image DP can maintain good user visibility of the traveling direction image AP while visually notifying the driver of the conditions on the left and right sides. In other words, by looking at the display image DP, the driver can recognize the condition of the ground near the front wheels while driving forward, and can also simultaneously recognize the conditions on the sides of the vehicle.
[0081] The right side camera image can be generated by cutting out the image captured by the right side camera 2R. However, it may also be generated by cutting out the image at a predetermined viewing angle from the rendered composite image, with the right side of the vehicle as the virtual viewpoint VP. The left side camera image can also be generated using the camera image from the left side camera 2L in the same way as the right side camera image.
[0082] The icon image 3i can function as a switch for switching the display content of the display image DP when touched by the user. Whether or not the user has touched the icon image 3i can be determined based on a touch position signal output from the touch panel 4. When the user's touch operation on the icon image 3i is detected, the display control unit F8 switches the display image DP, for example, from one in which only the traveling direction image AP is the peripheral image to one in which a panoramic image including left and right camera images is the peripheral image.
[0083] The following describes the flow of the peripheral image display process, which is a series of processes executed by the image generation ECU 70 when generating and displaying the display image DP. The flowchart shown in FIG. 7 is started when a peripheral image display condition is satisfied. The peripheral image display condition can be understood as a condition for displaying the display image DP. For example, the image generation ECU 70 determines that the peripheral image display condition is satisfied when the operation button 5 is pressed by the user. Alternatively, the image generation ECU 70 may be configured to determine that the peripheral image display condition is satisfied when it detects that a user operation has been performed via the touch panel 4 to display the display image DP. In other words, this flow can be started when the operation receiving unit F2 acquires a signal indicating that an operation for displaying a composite image has been performed.
[0084] Items constituting the peripheral image display condition may include the shift position, which is the setting state of the transmission, and the vehicle speed. For example, the peripheral image display condition may include at least one of the shift position being set to a specific range and the vehicle speed being less than a low-speed threshold. The specific range that satisfies the peripheral image display condition may be the low range or the reverse range. The low-speed threshold may be, for example, 10 km / h, 15 km / h, or 20 km / h. Alternatively, the peripheral image display condition may be determined to be satisfied when the operation receiving unit F2 receives a viewpoint switching operation via the touch panel 4 or the like while a front camera image or a rear camera image is displayed on the display 3.
[0085] The processing flow shown in Fig. 7 may be repeatedly executed until a display termination condition is satisfied. The display termination condition may be, for example, when the shift position is set to a position other than a specific range, when the vehicle speed is equal to or greater than a low-speed threshold, or when the operation button 5 is pressed again. In other words, the display termination condition may be considered to be satisfied when the peripheral image display condition is no longer satisfied. Alternatively, the image generation ECU 70 may determine that the display termination condition is satisfied when it detects that an operation to terminate the display of the display image DP has been performed via the touch panel 4.
[0086] First, in step S101, the image acquisition unit F1 acquires camera images captured by the four cameras 2. The acquisition of camera images is repeated over time, and the latest camera images are continuously acquired.
[0087] Furthermore, the image storage unit M1 stores the camera image captured by the camera 2 in the direction of travel each time the host vehicle travels the stored distance. When the peripheral image display condition transitions from an unsatisfied state to an satisfied state, the image acquisition unit F1 stores the image captured by the camera 2 in the direction of travel as the first storage process. After acquiring the camera image, the process proceeds to step S102.
[0088] In step S102, the display setting acquisition unit F4 acquires the display setting, for example, whether the virtual viewpoint VP is a driver's viewpoint or a bird's-eye viewpoint.
[0089] When the see-through image CP is generated for the first time after the driving power supply is turned on, a default setting of the virtual viewpoint VP is applied. The default setting may be a position and line of sight direction previously set by a designer or a user. For example, the default setting may be a driver's viewpoint with the line of sight directed diagonally downward and forward. The virtual viewpoint VP may also be set to the position and direction of the virtual viewpoint VP when the see-through image CP was last displayed. In this case, the display control unit F8 is configured to store, as a preparation process, the setting data of the virtual viewpoint VP when the see-through image CP was last displayed in the RAM 72 or the like.
[0090] Furthermore, the position and line of sight of the virtual viewpoint VP may be determined according to the traveling direction of the host vehicle. For example, when the traveling direction of the host vehicle is a forward direction, the driver's viewpoint with the line of sight directed diagonally downward and forward as described above is adopted as the virtual viewpoint VP. On the other hand, when the traveling direction of the host vehicle is a backward direction, for example, the driver's viewpoint with the line of sight directed diagonally downward and backward may be adopted as the virtual viewpoint VP. In addition, the line of sight of the virtual viewpoint VP may be adjusted in a direction according to the steering angle.
[0091] In the following, as an example, it is assumed that the virtual viewpoint VP is a driver's viewpoint with the line of sight directed diagonally downward and forward. After acquiring the display settings, the process proceeds to step S111.
[0092] In step S111, the vehicle state acquisition unit F3 acquires the vehicle state, which is information indicating the state of the host vehicle, such as the shift position and vehicle speed. In particular, the wheel speed acquisition unit F31, which constitutes a part of the vehicle state acquisition unit F3, acquires the wheel speed of the host vehicle. After acquiring the vehicle state, the process proceeds to step S112.
[0093] In step S112, the vehicle state acquisition unit F3 calculates the difference between the left and right wheel speeds. More specifically, the difference between the wheel speed of the right front wheel and the wheel speed of the left front wheel is calculated. The vehicle state acquisition unit F3 also calculates the difference between the wheel speed of the right rear wheel and the wheel speed of the left rear wheel. After calculating the wheel speed difference, the process proceeds to step S113.
[0094] In step S113, the wheel spin determination unit F32 determines whether the wheels of the host vehicle are spinning. For example, if the difference in wheel speed between the right front wheel and the left front wheel is equal to or greater than the wheel speed during spinning, the wheel spin determination unit F32 determines that the front wheels of the host vehicle are spinning. Alternatively, for example, if the difference in wheel speed between the right rear wheel and the left rear wheel is equal to or greater than the wheel speed during spinning, the wheel spin determination unit F32 determines that the rear wheels of the host vehicle are spinning. At this time, it is preferable to calculate the difference between the left and right wheel speeds using the average value per unit time as the wheel speed of each wheel, rather than the instantaneous value of the wheel speed of each wheel. This makes it easier to prevent erroneous wheel spin determination.
[0095] The method for determining wheel spin is not limited to the above-mentioned method of determining from the difference in wheel speed between the left and right wheels. For example, if the wheel speed is significantly higher than the power output of the engine or traction motor, it can be determined that the friction between the tire and the ground is very small, i.e., the vehicle is spinning.
[0096] Furthermore, if there is a determination result obtained by an ECU other than the image generation ECU 70 determining whether or not there is a spin based on the wheel speed, the spin determination unit F32 may acquire this determination result and use it as the spin determination result.
[0097] If at least one of the front or rear wheels of the host vehicle is spinning (S113: YES), the process proceeds to step S119. On the other hand, if all of the wheels of the host vehicle are not spinning, i.e., not spinning (S113: NO), the process proceeds to step S118.
[0098] In step S118, the composite image generating unit F7 updates the past image used to generate the transparent image CP based on the wheel speed. More specifically, the distance traveled by the host vehicle is calculated based on the wheel speed, and a camera image capturing an area currently located under the floor of the host vehicle is selected from the camera images stored in the image storage unit M1 and set as the past image.
[0099] Furthermore, the most recent camera image in the traveling direction is stored in the image storage unit M1 as a new candidate for a past image. Meanwhile, among the camera images stored as candidate past images, camera images captured at a position past the underfloor of the host vehicle are deleted from the image storage unit M1. In other words, only camera images on the traveling side of the underfloor of the host vehicle are stored as candidate past images, and camera images on the opposite side of the traveling direction from the underfloor of the host vehicle are deleted. After updating the past image based on the wheel speed, the process proceeds to step S121.
[0100] In step S119, the composite image generation unit F7 maintains the previous image taken immediately before the wheels began to spin. Even if it is determined based on the wheel speed that the vehicle has traveled the stored distance in the traveling direction, no new camera image is stored, and no currently stored candidate previous images are deleted. After maintaining the previous image taken immediately before the wheels began to spin, the process proceeds to step S121.
[0101] In step S121, the camera image synthesis unit F71 synthesizes the camera images. More specifically, the camera image synthesis unit F71 maps the image data and the past image from each camera 2 to a predetermined position on the projection plane TS as described above. When the vehicle is not spinning, the past image updated based on the wheel speed is projected onto the vehicle area. On the other hand, when the vehicle is spinning, the past image taken immediately before the spin is projected onto the vehicle area. After the camera images are synthesized, the process proceeds to step S122.
[0102] In step S122, the vehicle model attachment unit F72 attaches vehicle models such as vehicle boundary lines Lvc and 3D tire models Pt to the composite image. Other image elements such as planned track lines, scale lines, vehicle width lines, and 3D models other than the 3D tire models Pt may also be attached. After the vehicle models have been attached, the process proceeds to step S125.
[0103] In step S125, the composite image generation unit F7 generates a traveling direction image AP including the transparent image CP. More specifically, rendering is performed on the composite image to which the vehicle model has been added according to the virtual viewpoint VP, and image elements included in a predetermined viewing angle as seen from the virtual viewpoint VP are extracted. If the virtual viewpoint VP is set to the driver's viewpoint diagonally downward and forward, image elements in a range including the front wheels and front end of the host vehicle are extracted from the traveling direction image AP.
[0104] The first traveling direction image AP1 is a cutout portion of the traveling direction image AP where a real-time camera image is projected, while the second traveling direction image AP2 is a cutout portion of the traveling direction image AP where a past image is projected.
[0105] The first traveling direction image AP1 is a composite image created by combining camera images captured by three cameras 2: the front camera 2F, the right side camera 2R, and the left side camera 2L. The second traveling direction image AP2 is a composite image created by combining multiple past images captured at different times. The past images that make up the second traveling direction image AP2 are camera images captured by the front camera 2F that have been subjected to corrections such as rotation correction.
[0106] If the wheels of the vehicle are not spinning, the second forward direction image AP2 is updated with an appropriate past image in accordance with the movement of the vehicle. On the other hand, if the wheels of the vehicle are spinning, the second forward direction image AP2 maintains the past image of the position immediately before the wheels started spinning, and the past image does not change while the wheels are spinning. The first forward direction image AP1 is always an image synthesized with the current camera image, regardless of whether the wheels are spinning or not.
[0107] In step S141, the display image generation unit F81 generates a display image DP including the traveling direction image AP generated by the composite image generation unit F7. After the display image DP is generated, the process proceeds to step S142.
[0108] In step S142, the image output unit F9 outputs the display image DP to the display 3. More specifically, the image output unit F9 outputs a signal obtained by converting the digital image data of the display image DP generated by the display image generation unit F81 into a predetermined signal format to the display 3. As a result, the display image DP including the traveling direction image AP is displayed on the display 3. After the display image DP is output, the surrounding image display process ends. However, the series of processes is repeatedly executed until the display end condition is satisfied, and the latest surrounding image continues to be displayed.
[0109] The effects of the above-described embodiment will be described below. According to the above-described embodiment, when the vehicle is not spinning, the display control unit F8 displays a traveling direction image AP having a see-through image CP using a past image selected based on the current wheel speed. Furthermore, when the vehicle is spinning, the display control unit F8 displays a traveling direction image AP having a see-through image CP using a past image immediately before the vehicle started spinning. In other words, when the vehicle is not spinning, the display control unit F8 displays a see-through image CP that shows the floor of the vehicle 9 through a past image selected based on the current wheel speed. Furthermore, when the vehicle is spinning, the display control unit F8 does not display a see-through image CP that shows the floor of the vehicle 9 through a past image selected based on the current wheel speed. This prevents the display of a see-through image CP that uses a camera image taken at a position significantly different from the actual underfloor position as a past image. Therefore, it is unlikely that an image significantly different from the actual vehicle surroundings will be displayed as a surrounding image displayed during a spin. This makes it possible to provide a highly reliable surrounding image display device.
[0110] Furthermore, the updating of past images and the determination of wheel spin are based on wheel speed. This makes it easier to reduce the processing load compared to updating past images using methods that involve image processing of camera images, such as the optical flow method. This makes it easier to update past images quickly and shorten the time lag that occurs in the peripheral image display processing.
[0111] When the wheels are spinning, the display control unit F8 displays a traveling direction image AP including a transparent image CP generated using a past image taken immediately before the wheels started spinning. In other words, when the wheels are spinning, the display control unit F8 displays the transparent image CP that was displayed immediately before the wheels started spinning. This makes it easier to reduce the deviation between the transparent image CP and the actual underfloor position compared to when the transparent image CP is generated and displayed using a past image selected based on the wheel speed during spinning.
[0112] The traveling direction image AP includes a camera image from the front camera 2F, a camera image from the left side camera 2L, and a camera image from the right side camera 2R. In other words, the display control unit F8 displays a peripheral image synthesized using the camera image in the traveling direction of the vehicle 9 and the camera images in the left and right directions. Therefore, a wider range of image can be displayed as the peripheral image compared to when a peripheral image generated using only the camera image in the traveling direction is displayed.
[0113] The wheel spin determination unit F32 determines that the wheels are spinning when the difference between the wheel speed of the left wheel and the wheel speed of the right wheel is equal to or greater than the wheel speed during spinning. This allows wheel spin determination to be made based solely on wheel speed information. This allows wheel spin determination to be made with a simpler configuration than when wheel spin determination is made using a combination of devices other than the wheel speed sensor 6s, such as an acceleration sensor for measuring the acceleration of the vehicle 9 or a positioning device for determining the vehicle position.
[0114] A traveling direction image AP is generated that includes a see-through image CP that shows the surroundings of the vehicle while allowing components of the vehicle 9, such as the instrument panel and the bottom of the vehicle, to be seen through. Therefore, by checking such a surrounding image, the user can check the surroundings of the vehicle from a viewpoint inside the vehicle and can intuitively grasp the surroundings of the vehicle.
[0115] The forward direction image AP includes vehicle models such as the vehicle boundary line Lvc and the 3D tire model Pt. Therefore, based on the vehicle boundary line Lvc included in the forward direction image AP, the user can easily recognize the distance between the vehicle and three-dimensional objects such as rocks, curbs, and flaps that are present near the vehicle. This reduces the risk of the vehicle unintentionally hitting a three-dimensional object. The user can also easily recognize the position of the vehicle and tires relative to the lane markings, curbs, rocks, and other objects located below the vehicle. As a result, for example, when driving off-road, the user can perform precise driving maneuvers such as placing the tires on a targeted rock. Furthermore, when driving on-road, the user can easily perform precise driving maneuvers such as pulling the vehicle to the edge of the road to avoid hitting a curb or other object.
[0116] The effects of this embodiment have been described above using an example in which the virtual viewpoint VP is a driver's viewpoint with a line of sight directed forward. However, similar effects can be obtained when the line of sight is directed in other directions, such as rearward, lateral, or diagonally lateral. Furthermore, similar effects can be obtained when the virtual viewpoint VP is located at any position within the vehicle cabin other than the driver's viewpoint. In addition, similar effects can be expected when the virtual viewpoint VP is set on the outer surface of the vehicle 9 or in a vehicle vicinity area outside the vehicle cabin. The vehicle vicinity area here refers to, for example, an area within 0.2 m of the outer surface of the vehicle 9. The outer surface can include the left and right side surfaces, as well as the back, front end, and roof. The side surfaces can include door panels, fenders, pillars, etc.
[0117] The virtual viewpoint VP can be set at various positions within the vehicle cabin. For example, the virtual viewpoint VP can be set near the side mirror or at the center of the ceiling of the vehicle cabin. The display control unit F8 may also be configured to set an interior rear viewpoint in which the viewpoint is located behind the iris and the line of sight is directed diagonally downward and rearward. Such an interior rear viewpoint allows the rear wheels and the surrounding area to be displayed larger in the forward direction image AP displayed when the vehicle is reversing. As a result, the driver can easily recognize the area around the rear wheels and rear bumper when reversing, just as when driving forward.
[0118] Although the above describes an embodiment in which the display 3 is an in-vehicle display, the display destination of the image generated by the image generation ECU 70 is not limited to the in-vehicle display. The display 3 may be a display device provided in a mobile terminal such as a smartphone for remotely controlling the vehicle 9. Alternatively, the display 3 may be a display device provided in a management center that remotely manages the vehicle 9.
[0119] Second embodiment This embodiment is a modification of the preceding embodiment, in which the transparency of the wheel-spinning image displayed over the previous image is gradually reduced, so that a traveling direction image AP including a non-transparent image BP is displayed instead of the transparent image CP.
[0120] The flow of the peripheral image display process will be explained below using the flowchart shown in Fig. 8. However, explanations of processes similar to those described above may be omitted. The peripheral image display process is started when the peripheral image display conditions are met. If the spin determination unit F32 determines that the wheels are not spinning in step S113 (S113: NO), the process proceeds to step S118. On the other hand, if the spin determination unit F32 determines that the wheels are spinning (S113: YES), the process proceeds to step S219.
[0121] In step S118, the composite image generation unit F7 updates the past image based on the wheel speed. Then, the process proceeds to step S121, where the camera image synthesis unit F71 synthesizes the camera images. Then, the process proceeds to step S122, where the vehicle model assignment unit F72 assigns a vehicle model to the composite image. Then, the process proceeds to step S125, where the composite image generation unit F7 generates a traveling direction image AP including the transparent image CP, and the process proceeds to step S241.
[0122] In step S219, the composite image generation unit F7 uses the spinning image in place of the past image. The spinning image is an image to be displayed when the wheels are spinning. The spinning image may be any image that does not mislead the user into thinking that the current underfloor area is being displayed. An image of a single color such as black or white may be used as the spinning image. An image that uses text to indicate that the vehicle is spinning may also be used as the spinning image. The spinning image is preferably an image of a color different from the color of the road surface or the color of the vehicle model. After using the spinning image in place of the past image, the process proceeds to step S231.
[0123] In step S231, the composite image generation unit F7 combines the camera image and the spinning image to form a composite image. More specifically, the image data from each camera 2 and the spinning image are mapped to a predetermined position on the projection surface TS. The spinning image is projected onto the vehicle area. This results in the spinning image being combined with the area corresponding to the underfloor. After the images are combined, the process proceeds to step S232.
[0124] In step S232, the vehicle model attachment unit F72 attaches vehicle models such as vehicle boundary lines Lvc and 3D tire models Pt to the composite image. After the vehicle models have been attached, the process proceeds to step S235.
[0125] In step S235, the composite image generation unit F7 generates a traveling direction image AP including an opaque image BP. The opaque image BP is an image different from the transparent image CP, which is an image showing the road surface with the underfloor transparent, and is an image that the user will not mistake for the transparent image CP. The composite image generation unit F7 performs rendering on the composite image to which the vehicle model has been added, according to the virtual viewpoint VP. Furthermore, the composite image generation unit F7 generates the traveling direction image AP including the opaque image BP by cutting out image elements included in a predetermined viewing angle as viewed from the virtual viewpoint VP.
[0126] The following describes a method for generating a traveling direction image AP by switching the transparent image CP to an opaque image BP to generate the traveling direction image AP. In FIG. 9, the first traveling direction image AP1 is a composite image of current camera images captured by the front camera 2F, the right side camera 2R, and the left side camera 2L. The second traveling direction image AP2 is a past image captured by the front camera 2F. In this state, both the first traveling direction image AP1 and the second traveling direction image AP2 include white lines indicating parking spaces.
[0127] In FIG. 10, the second traveling direction image AP2 is an image that combines a past image and an image of the vehicle spinning. More specifically, the second traveling direction image AP2 is an image in which an image of the vehicle spinning image with increased transparency is superimposed on the past image of the vehicle just before the vehicle spinning. In other words, the second traveling direction image AP2 is an image in which a semi-transparent image of the vehicle spinning is superimposed on the past image of the vehicle just before the vehicle spinning. In the figure, the fact that the second traveling direction image AP2 is an image in which a semi-transparent image of the vehicle spinning is displayed on the past image of the vehicle just before the vehicle spinning is expressed by applying large diagonal lines to the past image of the vehicle just before the vehicle spinning.
[0128] If the image during spinning is a black image, the previous image is displayed in black through the image, making it difficult for the user to see the previous image immediately before spinning. If the image during spinning is a text image indicating that the vehicle is spinning, the text indicating that the vehicle is spinning is displayed in a faint manner, superimposed on the previous image immediately before spinning. In this state, the image that is still visible through the underfloor is displayed, so the white lines of the parking spaces are included in the second travel direction image AP2.
[0129] In Fig. 11, the second traveling direction image AP2 is an image in which only the spinning image, which is an opaque image BP, is displayed. In the figure, the fact that the second traveling direction image AP2 is an image of only the spinning image is indicated by the addition of diagonal lines with a smaller pitch than the diagonal lines in Fig. 10. If the spinning image is a black image, the entire second traveling direction image AP2 will be black, and the user will not be able to see the previous image just before the spinning. If the spinning image is a text image indicating that the vehicle is spinning, only the text indicating that the vehicle is spinning will be visible, and the user will not be able to see the previous image just before the spinning.
[0130] The spinning image is not an image showing the underfloor area through which the wheels are rotated, so in this state the white lines of the parking space are not included in the second traveling direction image AP2. However, the first traveling direction image AP1 displays an image synthesized with the current camera image, so the white lines of the parking space are included. By looking at the first traveling direction image AP1, the user can check the surrounding image other than the underfloor area. Furthermore, by looking at the second traveling direction image AP2 and recognizing that an image of the spinning wheels is displayed instead of an underfloor image, the user can recognize that the wheels are rotating.
[0131] When generating the traveling direction image AP including the opaque image BP, the image is generated so as to switch gradually from an image of only past images to an image of only spinning images, as shown in the order of Figures 9, 10, and 11. More specifically, the transparency of the spinning image is continuously reduced from 100% to zero. After generating the traveling direction image AP including the opaque image BP, the process proceeds to step S241.
[0132] In step S241, the display image generation unit F81 generates a display image DP including the traveling direction image AP generated by the composite image generation unit F7. When the wheels are not spinning, a display image DP including a transparent image CP is generated. On the other hand, when the wheels are spinning, a display image DP including an opaque image BP is generated. Also, immediately after the wheels start to spin when not spinning, a display image DP including a transparent image CP in which a semi-transparent spinning image is superimposed on a past image taken immediately before the wheels started spinning is generated. After generating the display image DP, the process proceeds to step S242.
[0133] In step S242, the image output unit F9 outputs the display image DP to the display 3. When the wheels are not spinning, the display image DP including the transparent image CP is displayed on the display 3. On the other hand, when the wheels are spinning, the display image DP including the opaque image BP is displayed on the display 3. Also, immediately after the wheels start to spin when not spinning, the display image DP including the transparent image CP in which a semi-transparent past image is superimposed on the image at the time of spinning is displayed on the display 3. After the display image DP is output, the peripheral image display process ends. However, the series of processes is repeatedly executed and the latest peripheral image continues to be displayed until the display end condition is met.
[0134] The effects of the above-described embodiment will be described below. According to the above-described embodiment, the display control unit F8 displays an opaque image BP that does not allow the floor of the vehicle 9 to be seen through when the vehicle is spinning. This prevents the user from misunderstanding the situation under the floor when the vehicle is spinning.
[0135] When the vehicle is not spinning, the transparent image CP is displayed, and when the vehicle is spinning, the opaque image BP is displayed. Therefore, the user can recognize whether the vehicle is spinning or not by recognizing whether the transparent image CP or the opaque image BP is displayed.
[0136] When switching the display from the transparent image CP displayed during non-spinning to the opaque image BP displayed during spinning, the display control unit F8 gradually reduces the transparency of the opaque image BP. This allows the user to visually recognize the switching effect that accompanies the image switching. Therefore, when switching from the transparent image CP to the opaque image BP, it is possible to prevent the user from misunderstanding that the display 3 or the like is malfunctioning.
[0137] The method of displaying the travel direction image AP including the opaque image BP is not limited to the above-described method. For example, a configuration may be adopted in which a past image is superimposed on the image during wheel rotation, and the opaque image BP may be displayed instead of the transparent image CP by gradually increasing the transparency of the past image.
[0138] Third embodiment This embodiment is a modification of the preceding embodiment, in which the area in which the opaque image BP is displayed during wheel spin is gradually widened, and a forward direction image AP including the opaque image BP is displayed instead of the transparent image CP.
[0139] The flow of the peripheral image display process will be explained below using the flowchart shown in FIG. 12. However, explanations of processes similar to those described above may be omitted. The peripheral image display process is started when the peripheral image display conditions are met. In step S101, a camera image is acquired. Then, the process proceeds to step S102, where the display settings are acquired. Then, the process proceeds to step S111, where the vehicle state is acquired. Then, the process proceeds to step S112, where the wheel speed difference is calculated. Then, the process proceeds to step S313.
[0140] In step S313, the wheel spin determination unit F32 determines whether there is a possibility of wheel spinning based on the wheel speed. For example, if the difference in wheel speed between the right front wheel and the left front wheel is greater than the non-spinning wheel speed, the wheel spin determination unit F32 determines that there is a possibility that the front wheels of the host vehicle are spinning. The non-spinning wheel speed is set to a value lower than the spinning wheel speed.
[0141] If the difference in wheel speeds is equal to or less than the non-spinning wheel speed, the wheels are not spinning. If the difference in wheel speeds is greater than the non-spinning wheel speed but less than the spinning wheel speed, it is unclear whether the wheels are spinning or not. In other words, the wheels are possibly spinning. If the difference in wheel speeds is equal to or greater than the spinning wheel speed, it is highly likely that the wheels are spinning. If there is a possibility that the wheels are spinning (S313: YES), the process proceeds to step S314. On the other hand, if there is no possibility that the wheels are spinning (S313: NO), it is determined that the wheels are not spinning, and the process proceeds to step S118.
[0142] In step S314, the image recognition unit F5 calculates the amount of movement of the vehicle 9 from the camera image. The amount of movement of the vehicle 9 can be calculated using image processing such as the optical flow method. When the amount of movement of the vehicle 9 is calculated from the camera image, when the vehicle is spinning, the amount of movement of the vehicle 9 is approximately zero even if the wheels are rotating, and when the vehicle is not spinning, the amount of movement of the vehicle 9 increases in accordance with the rotation of the wheels. After calculating the amount of movement of the vehicle 9 from the camera image, the process proceeds to step S315.
[0143] In step S315, the wheel spin determination unit F32 determines whether the wheels of the host vehicle are spinning. More specifically, if the amount of movement of the vehicle 9 is approximately zero, it is determined that the vehicle is spinning. On the other hand, if the amount of movement of the vehicle 9 is equal to the amount of movement calculated from the wheel speed, it is determined that the vehicle is not spinning. If it is determined that the vehicle is spinning (S315: YES), the process proceeds to step S219. On the other hand, if it is determined that the vehicle is not spinning (S315: NO), the process proceeds to step S118.
[0144] In step S219, the composite image generation unit F7 uses the spinning image instead of the previous image. Then, the process proceeds to step S231, where the camera image and the spinning image are used to composite an image. Then, the process proceeds to step S232, where a vehicle model is assigned to the composite image. Then, the process proceeds to step S335.
[0145] In step S335, the composite image generation unit F7 generates a traveling direction image AP including the opaque image BP. The following describes a method for generating the traveling direction image AP, taking as an example a case where the transparent image CP is switched to the opaque image BP to generate the traveling direction image AP.
[0146] In FIG. 13, the first traveling direction image AP1 is a composite image of current camera images captured by the front camera 2F, right side camera 2R, and left side camera 2L. The second traveling direction image AP2 is an image that includes both a transparent image CP, which is a past image captured by the front camera 2F, and an opaque image BP, which is an image captured during wheel spin. The left and center portions of the second traveling direction image AP2 maintain the transparent image CP, while the right end of the second traveling direction image AP2 uses the opaque image BP. In other words, the opaque image BP is used in part of the second traveling direction image AP2. In the figure, the second traveling direction image AP2 has a larger area for the transparent image CP than for the opaque image BP.
[0147] In Figure 14, the second traveling direction image AP2 is an image that includes both a transparent image CP, which is a past image captured by the front camera 2F, and an opaque image BP, which is an image taken when the vehicle is spinning. The left edge of the second traveling direction image AP2 maintains the transparent image CP, while the right edge and center of the second traveling direction image AP2 use the opaque image BP. In the figure, the second traveling direction image AP2 has a larger area for the opaque image BP than for the transparent image CP.
[0148] When generating a traveling direction image AP including an opaque image BP, the image is generated by gradually switching from an image including only past images to an image including only spinning images, as shown in the order of FIGS. 9, 13, 14, and 11. More specifically, the area of the transparent image CP in the second traveling direction image AP2 is gradually narrowed, and the area of the opaque image BP is gradually widened. The opaque image BP is not limited to being enlarged from the right edge to the left edge of the second traveling direction image AP2. For example, the opaque image BP may be enlarged from the left edge to the right edge of the second traveling direction image AP2. For example, the opaque image BP may be enlarged from the top edge to the bottom edge of the second traveling direction image AP2. For example, the opaque image BP may be enlarged from the center of the second traveling direction image AP2 toward the outer periphery.
[0149] After generating the traveling direction image AP including the opaque image BP, the process proceeds to step S241, where a display image DP is generated. Then, the process proceeds to step S242, where the display image DP is output. After the display image DP is output, the series of peripheral image display processes ends. However, the series of peripheral image display processes is repeatedly executed until the display end condition is satisfied, and the latest peripheral image continues to be displayed.
[0150] The effects of the above-described embodiment will be described below. According to the above-described embodiment, when switching the display from the transparent image CP to the opaque image BP, the display control unit F8 displays the opaque image BP by gradually widening the area of the opaque image BP. This makes it possible to prevent the user from misunderstanding that the display 3 or the like is malfunctioning when switching from the transparent image CP to the opaque image BP.
[0151] When the wheel spin determination unit F32 determines that there is a possibility of wheel spinning based on the wheel speed, it determines whether the wheels are spinning or not based on changes in the camera image over time. This allows for highly accurate wheel spin determination using the camera image. Furthermore, when it determines that there is no possibility of wheel spinning based on the wheel speed, wheel spin determination using the camera image is not performed. Here, wheel spin determination using the camera image imposes a larger processing load than wheel spin determination based on the wheel speed. Therefore, the processing load for wheel spin determination can be reduced compared to when wheel spin determination is always performed using the camera image. This reduces the processing load from wheel spin determination to output of peripheral images, making it easier to shorten the time required for processing.
[0152] Other embodiments The disclosure in this specification and drawings, etc. is not limited to the exemplified embodiments. The disclosure encompasses the exemplified embodiments and modifications thereto by those skilled in the art. For example, the disclosure is not limited to the combinations of parts and / or elements shown in the embodiments. The disclosure can be implemented in various combinations. The disclosure can have additional parts that can be added to the embodiments. The disclosure encompasses the omission of parts and / or elements from the embodiments. The disclosure encompasses the substitution or combination of parts and / or elements between one embodiment and another embodiment. The disclosed technical scope is not limited to the description of the embodiments. Some disclosed technical scopes are defined by the claims, and should be interpreted as including all modifications within the meaning and scope equivalent to the claims.
[0153] The disclosure in the specification, drawings, etc. is not limited by the claims. The disclosure in the specification, drawings, etc. encompasses the technical ideas described in the claims, and extends to more diverse and broader technical ideas than the technical ideas described in the claims. Therefore, various technical ideas can be extracted from the disclosure in the specification, drawings, etc. without being bound by the claims.
[0154] The controller and methods described herein may be implemented by a special-purpose computer having a processor programmed to perform one or more functions embodied in a computer program. Alternatively, the apparatus and methods described herein may be implemented by special-purpose hardware logic circuitry. Alternatively, the apparatus and methods described herein may be implemented by one or more special-purpose computers configured by a combination of a processor executing a computer program and one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium. [Explanation of symbols]
[0155] 1 Surrounding image display system, 2 Camera, 3 Display, 6 Vehicle state sensor, 6s Wheel speed sensor, 9 Vehicle, 70 Image generation ECU, M1 Image memory unit, F1 Image acquisition unit, F2 Operation reception unit, F3 Vehicle state acquisition unit, F31 Wheel speed acquisition unit, F32 Spin determination unit, F4 Display setting acquisition unit, F5 Image recognition unit, F7 Composite image generation unit, F71 Camera image synthesis unit, F72 Vehicle model assignment unit, F8 Display control unit, F81 Display image generation unit, F9 Image output unit, AP Travel direction image, AP1 First travel direction image, AP2 Second travel direction image, CP Transparent image, BP Opaque image, Pt 3D tire model, Lvc Vehicle boundary line,
Claims
1. A wheel speed acquisition unit (F31) that acquires a wheel speed of a vehicle (9); a wheel spin determination unit (F32) that determines whether the wheels are spinning or not based on the wheel speed acquired by the wheel speed acquisition unit; An image acquisition unit (F1) that acquires a plurality of camera images obtained by a plurality of cameras (2) that capture images of the surroundings of the vehicle; an image storage unit (M1) that stores, as a past image, a camera image captured in a traveling direction of the vehicle among the camera images acquired by the image acquisition unit; a composite image generating unit (F7) that generates a surrounding image showing the surroundings of the vehicle by synthesizing the plurality of camera images acquired by the image acquiring unit; A display control unit (F8) that displays the peripheral image generated by the composite image generation unit on a display (3), The display control unit, when the vehicle is not spinning, displays the peripheral image including a transparent image through the floor of the vehicle and the current camera image using the past image selected based on the current wheel speed, and when the vehicle is spinning, displays the peripheral image including the current camera image but not the transparent image using the past image selected based on the current wheel speed.
2. The peripheral image display device according to claim 1 , wherein the display control unit displays an opaque image that does not allow the floor of the vehicle to be seen through when the vehicle is in the idle state.
3. The peripheral image display device according to claim 2, wherein the display control unit, when switching the display from the transparent image displayed during the non-spinning period to the opaque image displayed during the spinning period, displays the opaque image by gradually changing the transparency.
4. The peripheral image display device according to claim 2, wherein when switching the display from the transparent image displayed during the non-spinning to the opaque image displayed during the spinning, the display control unit gradually widens the area of the opaque image to display the opaque image.
5. The peripheral image display device according to claim 1 , wherein, when the wheels are spinning, the display control unit displays the transparent image that was displayed immediately before the wheels started to spin.
6. The image acquisition unit acquires the camera image in a traveling direction of the vehicle and the camera image in a left-right direction, The peripheral image display device according to claim 1 , wherein the display control unit displays the peripheral image generated using the camera image in the traveling direction of the vehicle and the camera image in the left and right directions.
7. the wheel speed acquisition unit acquires the wheel speed of a right wheel that is a wheel on the right side and the wheel speed of a left wheel that is a wheel on the left side, 7. The peripheral image display device according to claim 1, wherein the wheel speed determination unit determines that the vehicle is in a state of freewheeling when a difference in wheel speed between the right wheel and the left wheel is equal to or greater than a wheel speed during freewheeling.
8. The peripheral image display device according to any one of claims 1 to 7, wherein the spin determination unit determines that the wheel is not spinning when it determines that there is no possibility of the wheel spinning based on the wheel speed, and determines whether the wheel is spinning or not spinning based on changes in the camera image over time when it determines that there is a possibility of the wheel spinning based on the wheel speed.
Citation Information
Patent Citations
Periphery monitoring device and program
JP2014209713A
Periphery monitoring device and program
JP2016021653A
Display control device and display control program
JP2017076890A
Periphery monitoring device
JP2018006921A
Vehicle periphery monitoring device
JP2018006941A
