Image generation method and image generation device

Abstract

Provided is an image generation method which is for generating a surroundings image of a vehicle and is executed by a controller (10), wherein the controller (10) acquires a camera image from a camera (20) that images the surroundings of the vehicle, acquires orientation information indicating the orientation of the camera (20) with respect to the road surface, determines, on the basis of the orientation information, a parameter to be used for viewpoint conversion, generates, on the basis of the parameter and the camera image, a viewpoint-converted image showing a vehicle surroundings region as viewed from a virtual viewpoint, stores, as a past viewpoint-converted image in a memory (19), at least a partial image included in the viewpoint-converted image, and generates a synthesized image obtained by synthesizing the viewpoint-converted image and the past viewpoint-converted image.

Description

Image generation method and image generation device

[0001] The present invention relates to an image generation method and an image generation device.

[0002] 2. Description of the Related Art Conventionally, there has been known a peripheral image generating device that generates an underfloor see-through image, which is a composite image obtained by transmitting the underside of a vehicle body, using previously captured camera images (for example, see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-82487

[0004] When the vehicle height changes due to suspension control, etc., the peripheral image generation device described in Patent Document 1 generates a composite image from camera images taken at different camera positions, and there is a problem that the change in viewpoint of the generated composite image can cause a sense of discomfort to the user viewing the composite image.

[0005] The problem to be solved by the present invention is to provide an image generating method and an image generating device that can reduce the sense of discomfort felt by the user.

[0006] The present invention solves the above problem by acquiring attitude information indicating the attitude of the camera relative to the road surface, determining parameters to be used for viewpoint conversion based on the attitude information, generating a viewpoint conversion image showing the area around the vehicle as seen from a virtual viewpoint based on the parameters and the camera image, storing at least a portion of the images included in the viewpoint conversion image in memory as past viewpoint conversion images, and generating a composite image by combining the viewpoint conversion image and the past viewpoint conversion image.

[0007] According to the present invention, it is possible to suppress the sense of discomfort felt by the user.

[0008] FIG. 1 is a block diagram showing the configuration of an image generation system according to this embodiment. FIG. 2 is a plan view showing a portion of an instrument panel. FIG. 3 is a diagram showing a display screen of a display. FIG. 4 is a conceptual diagram in which the display screen shown in FIG. 3 is divided into a peripheral area of ​​the vehicle and an underfloor area. FIG. 5 is a conceptual diagram explaining the relationship between a vehicle driving state and a virtual viewpoint. FIG. 6 is a conceptual diagram explaining the relationship between a vehicle driving state and a virtual viewpoint. FIG. 7 is a conceptual diagram explaining the relationship between a vehicle driving state and a virtual viewpoint. FIG. 8 is a flowchart of an image generation method executed by a controller. FIG. 9 is a conceptual diagram showing a portion of a contour line, a tire icon, and a tire outline frame superimposed on a past viewpoint converted image. FIG. 10 is a diagram showing a display screen displayed on the display when the vehicle is driving on flat ground. FIG. 11 is a diagram showing a display screen displayed on the display when the vehicle is driving on a road where the left wheel is higher than the right wheel.

[0009] An image generation system including an image generation device according to an embodiment of the present invention will be described below with reference to the drawings. The image generation system is a system that generates an image of the surroundings of a vehicle. FIG. 1 is a block diagram showing the configuration of the image generation system according to this embodiment. The image generation system 1 includes a front camera 21, a right camera 22, a left camera 23, a rear camera 24, an electronically controlled suspension system 30, a first display 41, a second display 42, and an image generation device 100. The image generation system 1 is installed in a vehicle. The front camera 21, the right camera 22, the left camera 23, the rear camera 24, the electronically controlled suspension system 30, the first display 41, the second display 42, and the image generation device 100 are connected via an in-vehicle communication network such as CAN or LIN.

[0010] The front camera 21, right camera 22, left camera 23, and rear camera 24 are provided on the vehicle and capture images of the surroundings of the vehicle. The front camera 21, right camera 22, left camera 23, and rear camera 24 capture images of the surroundings of the vehicle while the vehicle is traveling. The front camera 21 captures images in front of the vehicle, the right camera 22 captures images on the right side of the vehicle, the left camera 23 captures images on the left side of the vehicle, and the rear camera 24 captures images behind the vehicle. The front camera 21 is provided on the grille of the vehicle, the right camera 22 and left camera 23 are provided on the side mirrors, and the rear camera 24 is provided on the back door. Note that the various cameras may be provided in other positions on the vehicle. Note that in the following description, the front camera 21, right camera 22, left camera 23, and rear camera 24 will be collectively referred to as camera 20. Furthermore, the vehicle does not need to be equipped with all of the front camera 21, right camera 22, left camera 23, and rear camera 24, and may be equipped with, for example, only the front camera 21 and / or the rear camera 24.

[0011] The electronically controlled suspension system 30 variably controls the damping force of the suspension based on the detected values ​​of various sensors, the state quantities of the actuators, etc. The electronically controlled suspension system 30 includes coil springs, variable damping force dampers, actuators, sensors, etc., and is provided for each of the front, rear, left, and right wheels.

[0012] The coil spring and variable damping force damper are shock absorbers installed between the suspension arm and the vehicle body, supporting the suspension arm and the vehicle body so that they can move up and down elastically and damping the vertical stroke of the vehicle body. The variable damping force damper is equipped with an actuator that electrically controls the position of the variable damping force damper in response to control commands from the vehicle's ECU (electronic control unit). The actuator can also control the spring characteristics and damper characteristics by controlling thrust in response to control commands from the ECU. The sensors are installed on each wheel and detect the current state of the electronically controlled suspension system, such as the actuator position or suspension stroke speed. Note that active suspensions that can generate vertical forces using a drive source such as hydraulic pressure, air pressure, or an electric motor are used as shock absorbers.

[0013] The electronically controlled suspension system 30 is a mechanism for adjusting the height between the wheels and the vehicle body (suspension height) according to a vehicle height mode. The vehicle height modes include multiple adjustment modes that allow for gradual changes in vehicle height and an off mode that turns off the vehicle height adjustment function. Each vehicle height mode is indicated by the amount of change in vehicle height at the front wheel section and the amount of change in vehicle height at the rear wheel section. For example, in standard mode, the amount of change in vehicle height at the front wheel section and the rear wheel section is zero (reference value). In high mode, where the vehicle height is higher than in standard mode, the amount of change in vehicle height is positive. In low mode, where the vehicle height is lower than in standard mode, the amount of change in vehicle height is negative. The vehicle height mode is selected by the user and / or the system. The electronically controlled suspension system 30 then controls the suspension to achieve a vehicle height according to the vehicle height mode.

[0014] The first display 41 and the second display 42 are provided on the vehicle's instrument panel. The first display 41 is primarily intended for the driver, and the second display 42 is intended for the passenger in the front seat. FIG. 2 is a plan view showing a portion of the instrument panel. As shown in FIG. 2, the first display 41 is provided on the instrument panel in front of the driver. The second display 42 is provided in the center of the instrument panel. That is, the first display 41 is located within the driver's operable range but outside the passenger's operable range. The second display 42 is located at least within the passenger's operable range, and may be located within the driver's operable range. The first display 41 displays, for example, meters, a range indicator showing the current shift position, an icon indicating the seat belt status, and the like. The second display 42 displays, for example, a map for a navigation system, a menu screen for an entertainment system (such as a selection screen for various applications), content video, and the like. The first display 41 and / or the second display 42 display a display image generated by the image generation device 100. The display image is an image that displays the surroundings of the vehicle.

[0015] The first display 41 and the second display 42 have display screens that are long horizontally along the vehicle width direction (the left-right direction on the paper surface of FIG. 2 ). As shown in FIG. 2 , the first display 41 and the second display 42 are arranged side by side on the left and right along the vehicle width direction, with their display screens close to each other. In the following description, the first display 41 and the second display 42 will be collectively referred to as display 40. Furthermore, the vehicle does not need to be equipped with both the first display 41 and the second display 42, and may be equipped with only one of the displays.

[0016] The image generation device 100 includes a controller 10 and a memory 19. The image generation device 100 generates an image of the surroundings of a vehicle. The controller 10 has functions such as acquiring various data such as camera images and vehicle data, image processing functions, and display control functions. The controller 10 has, as functional blocks, an image acquisition unit 11, a vehicle data acquisition unit 12, an image processing unit 13, and a parameter determination unit 14. The controller 10 stores programs for realizing various functions in the memory, and executes the programs using a processor to perform the processing flow indicated in the programs.

[0017] The image acquisition unit 11 acquires camera images from the camera 20 or the like. The vehicle data acquisition unit 12 acquires vehicle data from sensors mounted on the vehicle via an in-vehicle communication network. The vehicle data is data indicating the current state of the vehicle, and includes, for example, vehicle speed, steering angle, the current position (current coordinates) of the vehicle, and attitude information. The vehicle data acquisition unit 12 acquires vehicle speed from a vehicle speed sensor and steering angle from a steering angle sensor. The vehicle data acquisition unit 12 acquires current position information from a navigation system. The vehicle data acquisition unit 12 also acquires attitude information from the electronically controlled suspension system 30 or an ECU that controls the electronically controlled suspension system 30. The attitude information is information indicating the attitude of the camera relative to the road surface, and corresponds to the vehicle height mode and the height between the wheels and the vehicle body adjusted by the electronically controlled suspension system 30. Note that the attitude information may be information for determining the height of a camera such as the camera 20, and may be, for example, a suspension control amount (adjustment amount).

[0018] The image processing unit 13 performs image processing on the camera image to display an image of the current surroundings of the vehicle and an image of the underfloor of the vehicle (under-vehicle image) on the display 40 while the vehicle is traveling. FIG. 3 shows a display screen of the display 40. FIG. 4 is a conceptual diagram in which the display screen shown in FIG. 3 is divided into a surrounding area of ​​the vehicle and an under-floor area. There are multiple display modes for displaying the image of the surroundings of the vehicle, and these modes are selected by, for example, the user. The display modes include a skeleton view as shown in FIG. 3 or FIG. 4, as well as an around view. In the skeleton view, a virtual image showing the state of the under-floor of the vehicle is displayed in addition to a current image (real image) showing the surroundings of the vehicle. In the around view, an image showing the surroundings of the vehicle is displayed as if looking down on the vehicle from directly above. An image showing the front of the vehicle, images showing the left and right sides of the vehicle, and an under-floor image are displayed on the display screen of the display 40. The image showing the front of the vehicle and the images showing the left and right sides of the vehicle correspond to the image of the surroundings of the vehicle. The image displayed on the display 40 is a composite image of a surrounding image showing the area around the vehicle as seen from a virtual viewpoint and an underfloor image of the underfloor area as seen from the virtual viewpoint. The virtual viewpoint is set at a predetermined position vertically away from the roof of the vehicle and outside the vehicle.

[0019] The image processing unit 13 performs viewpoint conversion processing on the camera images acquired from the front camera 21, right camera 22, and left camera 23, and generates viewpoint converted images that appear as if the area in front of and the areas to the left and right of the vehicle are being viewed from a virtual viewpoint. That is, the image processing unit 13 converts the camera images corresponding to images viewed from the camera positions into viewpoint converted images viewed from the virtual viewpoint. The viewpoint conversion from the camera positions to the virtual viewpoint is performed by calculation processing using parameters, and the parameters are stored in the memory 19. The parameters are determined by the parameter determination unit 14. As a result, the image processing unit 13 performs viewpoint conversion processing based on the camera images and the parameters stored in the memory 19.

[0020] The image processing unit 13 also performs viewpoint conversion processing on the camera images from the front camera 21, the right camera 22, and the left camera 23, and fits each image to the surrounding area of ​​the vehicle. In the example of FIG. 4 , the image processing unit 13 fits the camera image from the front camera 21, the camera image from the right camera 22, and the camera image from the left camera 23 to the front area 51, the right area 52, and the left area 53, respectively. At this time, the image processing unit 13 performs blending processing on the overlapping portions of the camera images from the front camera 21, the right camera 22, and the left camera 23. The blending processing is a process for maintaining continuity between two overlapping images, and a method known at the time of filing of this application is applied. For example, the image processing unit 13 may process one of the two camera images by making the overlapping portion transparent so that the two images become a single image. In the example of FIG. 4 , the boundary portion between the front area 51 and the right area 52 and the boundary portion between the front area 51 and the left area 53 are blending areas processed by the blending processing. As a result, a single image of the vehicle surroundings as seen from the virtual viewpoint is generated, as shown in Figures 3 and 4. Of the display images displayed on the display 40, a display image of the area surrounding the vehicle is generated.

[0021] The image processing unit 13 stores the image used for the front area 51 among the viewpoint converted images in the memory 19. In the following description, the viewpoint converted image of the front area 51 recorded in the memory 19 is also referred to as a past viewpoint converted image. The image processing unit 13 stores the past viewpoint converted image in the memory 19 at a predetermined cycle while the vehicle is traveling. In other words, the image processing unit 13 stores the past viewpoint converted image in the memory 19 every time the vehicle travels a predetermined distance.

[0022] The image processing unit 13 generates an underfloor image based on the past viewpoint converted image. The underfloor image is displayed as a skeleton image in the underfloor area of ​​the vehicle. That is, the underfloor area of ​​the vehicle is not normally visible, but is displayed on the display 40 as a virtual image. In the example of FIG. 3 , a manhole located under the vehicle's floor is not normally visible, but is displayed as a virtual image. While the vehicle is traveling forward, the viewpoint converted image displayed in the front area 51 moves relatively backward as the vehicle travels, becoming an image of the underfloor of the vehicle. The image processing unit 13 selects an image that can serve as an underfloor image from the past viewpoint converted images stored in the memory 19 based on the current position and speed of the vehicle. The image processing unit 13 cuts out the selected image with a frame corresponding to the underfloor area, and generates the image corresponding to the underfloor area as the underfloor image.

[0023] The image processing unit 13 may perform correction by enlarging or reducing the past-viewpoint converted image. As shown in FIG. 4 , the underfloor area 54 has a trapezoidal shape, and the upper side of the underfloor area 54 is shorter than the lower side. When generating an image corresponding to the underfloor area, the image processing unit 13 may enlarge or reduce the image based on the vehicle speed. Specifically, the image processing unit 13 enlarges or reduces the past-viewpoint converted image so that the image on the lower side of the underfloor area 54 is larger than the image on the upper side of the underfloor area 54, thereby generating an underfloor image with a sense of perspective. In this way, the image processing unit 13 enlarges or reduces the past-viewpoint converted image depending on the position of the image displayed on the display 40.

[0024] The image processing unit 13 also performs rotation correction on the selected image in accordance with the steering angle. For example, when the vehicle is traveling on a road that turns left, the viewpoint converted image in the front area 51 rotates counterclockwise relative to the vehicle. Therefore, when the viewpoint converted image displayed in the front area 51 is used for an underfloor image, it is advisable to perform rotation correction. For example, when the vehicle is traveling on a road that turns left, the image processing unit 13 rotates the past viewpoint converted image assigned to the front area 51 clockwise by the steering angle. In this way, the image processing unit 13 corrects the past viewpoint converted image so that the past viewpoint converted image rotates in accordance with the steering angle.

[0025] The image processing unit 13 applies the previous viewpoint converted image to the underfloor area 54, and combines the viewpoint converted image used for the vehicle's surrounding area with the previous viewpoint converted image used for the underfloor area (corresponding to the underfloor image) so that the vehicle surroundings image and the underfloor image form a single image. The combined image becomes the display image displayed on the display 40. The image processing unit 13 may also perform blending processing on the boundary portions between the vehicle surroundings image (images displayed in the front area 51, right area 52, and left area 53) and the underfloor image.

[0026] The image processing unit 13 may generate a display image for the display 40 by superimposing a contour line indicating the vehicle's outer shape and a tire image representing the tire on a composite image of the viewpoint conversion image and the previous viewpoint conversion image. As shown in FIG. 3 , the contour line 61 includes a contour line indicating the vehicle's outer shape and a contour line indicating the tire's outer shape. The contour line 61 is displayed in the underfloor area 54 and is drawn at a size correlated with the camera image. The contour line 61 is not limited to a dotted line as shown in FIG. 3 , but may be a solid line or a thick line, or may be a colored line such as orange or red. By checking the contour line 61 on the display screen displayed on the display 40, the user can understand the outer shape of the vehicle and the outer shape of the front tire. Note that the contour line 61 may be a line indicating the vehicle's outer shape or the tire's outer shape.

[0027] The image processing unit 13 may generate a display image by superimposing a tire image representing a tire on a composite image of the viewpoint conversion image and the past viewpoint conversion image. As shown in FIG. 3 , the tire image is represented by a tire icon 62, which represents a front wheel. The tire icon 62 is drawn so that the size and position of the tire can be confirmed on the screen of the display 40. The tire icon 62 is displayed as a transparent image. The image processing unit 13 may display the tire icon 62 on the display 40 by linking the steering angle with the angle (orientation) of the tire icon 62. The tire icon 62 includes multiple icons with different angles. The image processing unit 13 selects a tire icon 62 corresponding to the current steering angle from the multiple icons according to the steering angle when the vehicle is turning, and displays the tire icon 62 on the display 40. This allows the user to understand the current steering angle from the tire icon 62 displayed on the display 40.

[0028] The image processing unit 13 may superimpose the reference lines on a composite image of the viewpoint converted image and the past viewpoint converted image to generate a display image. The reference lines are guide information (driving support information) for supporting vehicle driving, and in the example of FIG. 3 include a predicted course line 63, a distance reference line 64, and a fixed reference line 65. The predicted course line 63 is a line representing the vehicle's travel path and is drawn as a line along the travel route. The image processing unit 13 calculates the predicted vehicle route from the current vehicle position and steering angle, and displays the travel trajectories of the left and right wheels as the predicted course lines 63 on the display 40.

[0029] The distance guide lines 64 are lines that indicate the distance from the front end of the vehicle and are represented by multiple line segments along the vehicle width direction. For example, the distance guide lines 64 are drawn at multiple positions (e.g., 0.5 m, 1.0 m, 2.0 m, and 3.0 m) away from the portion corresponding to the vehicle's bumper. The predicted course line 63 and the distance guide lines 64 may be represented in different colors.

[0030] The fixed reference line 65 indicates the route of the vehicle when the steering angle is 0 degrees. The image processing unit 13 calculates the route of the vehicle when the steering angle is 0 degrees from the current vehicle position and steering angle, and displays the calculated route on the display 40 as the fixed reference line 65. In the example of FIG. 3, since the vehicle is traveling on a road that turns left, the predicted course line 63 is drawn as a curved line that follows the curve, and the fixed reference line 65 is drawn as a straight line. This allows the user to understand the curvature of the road on which they are currently traveling.

[0031] The image processing unit 13 may generate a display image for the display 40 by superimposing characters or symbols indicating that the image of the underfloor area 54 (underfloor image) is not a current camera image on a composite image of the viewpoint converted image and the past viewpoint converted image. The image used for the underfloor area is a past viewpoint converted image and is not real-time. Therefore, the image processing unit 13 displays a NOT LIVE icon 66 on the display 40 to warn that the underfloor image is not a real-time image. The NOT LIVE icon 66 is displayed in the underfloor area 54. The NOT LIVE icon 66 is not limited to characters and may be a symbol.

[0032] The image processing unit 13 may generate a display image for the display 40 by superimposing a warning line 67 on a composite image of the viewpoint converted image and the past viewpoint converted image. The warning line 67 is displayed at the boundary between the viewpoint converted image and the underfloor image displayed in the front area 51. The warning line 67 represents the boundary between the current camera image and the past image. In the example of FIG. 3 , the warning line 67 is drawn as a line parallel to the distance reference line 64 at a position forward of the line ahead of the contour line 61 (a line corresponding to the bumper). The warning line 67 is displayed in a different color from the predicted course line 63, the distance reference line 64, and the fixed reference line 65, for example, in red.

[0033] If the display mode of the display 40 has just been switched to the skeleton view and a past viewpoint converted image cannot be acquired from the memory 19, the image processing unit 13 generates a display image for the display 40 by filling in the underfloor area with a black image. If a past viewpoint converted image can be acquired from the memory 19, the image processing unit 13 may generate a display image for the display 40 by superimposing the acquired image on a composite image of the viewpoint converted image and the past viewpoint converted image. In the skeleton view, an image in the traveling direction of the vehicle is displayed as a current image, so when the vehicle is moving backward, a camera image from the rear camera 24 of the vehicle may be displayed on the display 40. If a past viewpoint converted image cannot be acquired from the memory 19 immediately after the shift lever has been switched to the drive range or the reverse range, the image processing unit 13 may generate a display image for the display 40 by filling in the underfloor area with a black image.

[0034] Furthermore, the image processing unit 13 may generate the display image for the display 40 so that the NOT LIVE icon 66 and the warning line 67 are displayed on the display 40 immediately after the image of the underfloor area is switched from a blackened image to a past viewpoint converted image.

[0035] The image processing unit 13 may display the skeleton view display image generated in the above manner on either the display 41 or the display 42, or may combine the displays 41 and 42 into one large screen and display the image on the display. Furthermore, when the skeleton view display image is displayed on one large screen combining the displays 41 and 42, the image processing unit 13 may display the NOT LIVE icon 66 and the warning line 67 only on the display 41 for the driver.

[0036] When displaying the skeleton view, the image processing unit 13 may display, for example, a sonar indicator icon in addition to the contour line 61 shown in FIG. 3 . If the sonar detects an obstacle around the vehicle, the sonar indicator notifies the user of the detection of the obstacle by displaying an image on the display 40 or by issuing an alarm, for example. At this time, the vehicle data acquisition unit 12 may acquire sonar detection data, and the image processing unit 13 may superimpose a sonar indicator icon indicating the detection of the obstacle on a composite image of the viewpoint converted image and the past viewpoint converted image to generate a display image for the display 40. Furthermore, when displaying the skeleton view, the image processing unit 13 may display on the display 40 that the obstacle detection by the sonar indicator or the moving object detection function is on.

[0037] The parameter determination unit 14 determines parameters to be used for point conversion based on the attitude information acquired by the vehicle data acquisition unit 12. The parameters are stored in the memory 19 and are used in viewpoint conversion processing by the image processing unit 13. The parameters correspond to the vehicle height mode and are parameters for keeping the height of the virtual viewpoint constant even when the vehicle height changes depending on the vehicle height mode.

[0038] Next, the vehicle height (suspension height), the position of the virtual viewpoint, and the images used for the skeleton view will be described with reference to Figs. 5 to 7. Figs. 5 and 6 are conceptual diagrams illustrating the vehicle running state when the vehicle height (suspension height) is constant. Fig. 7 is a conceptual diagram illustrating the vehicle running state when the vehicle height (suspension height) changes. Fig. 5 shows the vehicle running state at time t а 6 and 7 show the vehicle running state at time t ｂ Indicates the vehicle's running condition at the time.

[0039] As shown in FIG. а At time point A, the front camera 21 captures an image of the area in front of the vehicle. １ is time t а The image processing unit 13 processes the camera image (A １ ) is subjected to viewpoint conversion processing. １) is the actual position of the front camera 21 (B １ ) to position (B ２ ) and the virtual viewpoint (C １ The image processing unit 13 generates a viewpoint converted image seen from a virtual viewpoint (C １ ) is displayed on the display 40, and the previous viewpoint converted image used for the front area 51 is stored in the memory 19.

[0040] As shown in FIG. 6, the vehicle ｂ When the camera moves to the position corresponding to １ ) is the past viewpoint converted image (A １ The image processing unit 13 converts the past viewpoint converted image (A １ 6, D represents the underfloor image, which corresponds to the underfloor area displayed in the skeleton image. ｂ At time point A, the front camera 21 captures an image of the area in front of the vehicle. ２ is time t ｂ The image processing unit 13 processes the camera image (A ２ ) is subjected to viewpoint conversion processing. ２ ) is the actual position of the front camera 21 (B ３ ) to position (B ４ ) and the virtual viewpoint (C ２ Then, the image processing unit 13 generates a viewpoint conversion image seen from the camera image (A １ ) is a past viewpoint converted image obtained by viewpoint conversion, and a camera image (A ２ ) is converted into a viewpoint-converted image, and the image displayed on the display 40 is generated. ２ ) is the height of the virtual viewpoint (C １ ) is the same height as

[0041] In the example of FIG. 7, the vehicle height is adjusted and at time t ｂ The position of the front camera 21 is adjusted by the vehicle height (E) at time t а Unlike the present embodiment, the image processing unit 13 processes the camera image (A１ When viewpoint conversion processing is performed using the same parameters as those used when viewpoint conversion processing was performed on the camera image (A ２ ) is the actual position of the front camera 21 (B ５ ) to position (B ６ The viewpoint is converted to the position (B ５ ) is time t а Position (B １ ) is higher than the position after viewpoint conversion (B ６ ) is time t а Position (B ２ ) and the virtual viewpoint position (C ３ ) also at time t а Position (C １ ) is higher than the virtual viewpoint. а At time t ｂ Also, the camera image (A １ ) is a past viewpoint converted image obtained by viewpoint conversion, and a camera image (A ２ When a viewpoint-converted image obtained by converting the viewpoint of the virtual viewpoint 100 is synthesized, viewpoint-converted images with different virtual viewpoint heights are synthesized, and when a display image generated from the synthesized image is displayed, the user feels uncomfortable.

[0042] In this embodiment, a plurality of parameters corresponding to the vehicle height mode are stored in the memory 19, and the parameter determination unit 14 selects parameters that match the current vehicle height (suspension height) based on the attitude information. For example, four vehicle height modes are set: off mode, standard mode, high mode, and low mode, and parameters corresponding to the mode are set. A table that associates each mode with the parameters is stored in the memory 19. The parameter determination unit 14 selects parameters that match the vehicle height mode by referring to the table. In the high mode, the camera position is higher than in the standard mode. The height length adjusted in the viewpoint conversion process is shorter in the high mode than in the standard mode. In other words, the parameters are set so that the height of the virtual viewpoint after viewpoint conversion remains constant even when the vehicle mode is changed.

[0043] In the example of FIG. 7, the image processing unit 13 processes the camera image (A １ ) and change the parameters used when performing viewpoint conversion processing. ｂ Using the parameters that match the vehicle height, the camera image (A ２ ) is subjected to viewpoint conversion processing. ２ ) is the actual position of the front camera 21 (B ５ ) to position (B ７ The viewpoint is converted to the position (B ７ ) is the position (B ６ ) and at position (B ２ ) and the virtual viewpoint position (C ４ ) at time t а Position (C １ ) and the camera image (A １ ) is a past viewpoint converted image obtained by viewpoint conversion, and a camera image (A ２ The height of the virtual viewpoint is kept constant between the previous viewpoint converted image and the viewpoint converted image obtained by viewpoint conversion of the previous viewpoint converted image. As a result, when the previous viewpoint converted image and the viewpoint converted image are combined, there is no change in the viewpoint, and the sense of discomfort felt by the user can be reduced.

[0044] Next, an image generation method by the controller 10 will be described with reference to Fig. 8. Fig. 8 is a flowchart of the image generation method by the controller 10. The controller 10 repeatedly executes the following control flow while an image of the surroundings of the vehicle is displayed on the display 40 using the skeleton view.

[0045] In step S1, the image acquisition unit 11 acquires a camera image from the camera 20. In step S2, the vehicle data acquisition unit 12 acquires vehicle data from an on-board device such as the electronically controlled suspension system 30. In step S3, the parameter determination unit 14 determines parameters to be used for viewpoint conversion based on posture information included in the vehicle data. In step S4, the image processing unit 13 generates a viewpoint conversion image based on the parameters determined by the parameter determination unit 14 and the camera image acquired by the image acquisition unit 11.

[0046] In step S5, the image processing unit 13 stores at least a portion of the image included in the viewpoint converted image in the memory 19 as a past viewpoint converted image. In step S6, the image processing unit 13 performs image processing on the current viewpoint converted image and the past viewpoint converted image. The current viewpoint converted image corresponds to a current camera image (real image) showing the surroundings of the vehicle. The image processing on the current viewpoint converted image and / or the past viewpoint converted image includes transparency processing to create a skeleton image, enlargement or reduction processing, rotation correction, and blending processing. Note that the image processing unit 13 does not need to perform all of these processes; it is sufficient to perform at least one of the processes. In step S7, the image processing unit 13 performs synthesis processing to synthesize the current viewpoint converted image and the past viewpoint converted image. In step S8, the image processing unit 13 generates a display image based on the synthesized image. The processing to generate the display image includes superimposition processing of contours, etc., and image processing for displaying as a single image. The image processing for displaying as a single image includes image processing, blending, rotation correction, etc. It should be noted that the image processing unit 13 does not need to perform all of these processes, but only needs to perform at least one of them. In step S9, the image processing unit 13 displays the generated display image on the display 40. Then, the controller 10 ends the control flow.

[0047] As described above, in the image generation method or image generation device according to this embodiment, the controller 10 acquires a camera image from the camera 20, acquires attitude information, determines parameters to be used for viewpoint conversion based on the attitude information, generates a viewpoint converted image showing the area around the vehicle as seen from a virtual viewpoint based on the parameters and the camera image, stores at least a portion of the image included in the viewpoint converted image as a previous viewpoint converted image in the memory 19, and generates a composite image by combining the viewpoint converted image and the previous viewpoint converted image. This suppresses changes in the viewpoint of the viewpoint converted image that accompany changes in the attitude of the vehicle, thereby suppressing any sense of discomfort felt by the user.

[0048] In this embodiment, the attitude information also includes information indicating the control state of the vehicle's suspension, which allows the parameters used for viewpoint conversion to be determined according to the suspension height.

[0049] In this embodiment, the display area displayed by the composite image includes the underfloor area of ​​the vehicle, and the past viewpoint converted image is used to display the underfloor area, allowing the user to check the condition of the underfloor area of ​​the vehicle in the skeleton view.

[0050] In this embodiment, the controller 10 superimposes an outline showing the vehicle's outer shape and / or a tire image showing the tire on the composite image to generate a display image for the display 40. This allows the user to check the vehicle's outer shape, the position of the tire, etc. in the skeleton view.

[0051] In this embodiment, the controller 10 generates a display image on the display by superimposing on the composite image a character or symbol indicating that the image of the underfloor area is not a current camera image, thereby enabling the user to confirm in the skeleton view that the image of the underfloor area is not a real image.

[0052] In this embodiment, the controller 10 also enlarges or reduces the past viewpoint converted image depending on the position of the image displayed on the display, allowing the user to check the condition of the underfloor area in a perspective image in the skeleton view.

[0053] In this embodiment, the controller 10 corrects the past viewpoint converted image so that the past viewpoint converted image rotates in accordance with the steering angle, thereby correcting the underfloor image in accordance with the steering angle of the vehicle.

[0054] As a modification of this embodiment, the image processing unit 13 may generate a display image for the display 40 by superimposing the tire icon 62 and the tire outline frame 68 on a composite image of the viewpoint converted image and the past viewpoint converted image so that the tire icon 62 and the tire outline frame 68 indicating the outline of the tire are interlocked. FIG. 9 is a conceptual diagram showing a portion of the contour line 61, the tire icon 62, and the tire outline frame 68 superimposed on the past viewpoint converted image. The image processing unit 13 calculates the angle (inclination) of the tire outline frame 68 using the same calculation method as for the predicted course line 63, and rotates the tire outline frame 68 around the center of the left and right front wheels. The tire outline frame 68 is drawn at a size that matches the diameter and width of the tire icon 62.

[0055] As a modification of this embodiment, the image processing unit 13 may change the positions of the contour line 61 and the tire icons 62 based on the attitude information and superimpose them on a composite image of the viewpoint converted image and the previous viewpoint converted image to generate a display image for the display 40. Fig. 10 is a diagram showing a display screen displayed on the display 40 when the vehicle is traveling on a flat surface. Fig. 11 is a diagram showing a display screen displayed on the display 40 when the vehicle is traveling on a road on which the left wheel is higher than the right wheel. As shown in Fig. 10, when the vehicle is traveling on a flat surface, the contour line 61 is drawn as a flat line, similar to a flat road surface, and the tire icons 62 are drawn so that the heights of the left and right tire icons 62 do not change.

[0056] When the road surface is inclined and the position of the left wheel is higher than the position of the right wheel, as shown in FIG. 11 , the contour line 61 is drawn at an inclination similar to the inclination of the road surface, and the tire icon 62 is drawn so that the position of the left tire icon 62 is higher than the position of the right tire icon 62. Note that the predicted course line 63, distance guide line 64, fixed guide line 65, and warning line 67 do not have to be inclined to match the road surface, but may be horizontal lines. Furthermore, the composite image of the current viewpoint converted image and the past converted image is not inclined, just like a flat road surface. When the road surface is inclined, the absolute position of the virtual viewpoint changes, but the virtual viewpoint of the composite image does not change relative to a flat road. As a result, when the vehicle is traveling on an inclined road surface, the user can check the inclination of the road surface and the roll angle from the display of the contour line 61 and the tire icon 62 in the skeleton view. Furthermore, even when the vehicle is traveling on an inclined road surface, the vehicle's posture can be confirmed from the display of the composite image and the predicted course line 63, etc. In order to prevent the position of the virtual viewpoint of the composite image from changing relative to a flat road when the road surface is inclined, the parameter used for viewpoint conversion may be a value corresponding to the inclination of the vehicle. For example, the parameter may be a parameter that keeps the position of the virtual viewpoint constant even when the inclination of the vehicle changes.

[0057] In this embodiment, the controller 10 does not need to execute all of the control flows shown in FIG. 8, and does not need to execute each control flow in the order shown in FIG.

[0058] REFERENCE SIGNS LIST 1 Image generation system 10 Controller 11 Image acquisition unit 12 Vehicle data acquisition unit 13 Image processing unit 14 Parameter determination unit 19 Memory 20 to 24 Camera 30 Electronically controlled suspension device 40 to 42 Display

Claims

1. An image generation method for generating a peripheral image of a vehicle, which is executed by a controller, wherein the controller: obtains a camera image from a camera that images the periphery of the vehicle; obtains attitude information indicating the attitude of the camera with respect to the road surface; determines parameters for perspective transformation based on the attitude information; generates a perspective transformation image showing a peripheral area of the vehicle as seen from a virtual viewpoint based on the parameters and the camera image; stores at least a part of the image included in the perspective transformation image in a memory as a past perspective transformation image; and generates a composite image by synthesizing the perspective transformation image and the past perspective transformation image.

2. The image generation method according to claim 1, wherein the attitude information includes information indicating a control state of a suspension of the vehicle.

3. The image generation method according to claim 1 or 2, wherein a display area to be displayed by the composite image includes an underfloor area of the vehicle, and the past perspective transformation image is used for an image that displays the underfloor area.

4. The image generation method according to any one of claims 1 to 3, wherein the controller superimposes a contour line indicating an outer shape of the vehicle and / or a tire image representing a tire on the composite image to generate a display image for a display.

5. The image generation method according to claim 3, wherein the controller superimposes a character or symbol indicating that the image of the underfloor area is not the current camera image on the composite image to generate a display image for a display.

6. The image generation method according to any one of claims 1 to 5, wherein the controller stores the past perspective transformation image in the memory every time the vehicle travels a predetermined distance.

7. The image generation method according to any one of claims 1 to 6, wherein the controller enlarges or reduces the past perspective transformation image according to a position of an image displayed on a display.

8. The image generation method according to any one of claims 1 to 7, wherein the controller corrects the past perspective transformation image so that the past perspective transformation image rotates according to a steering angle.

9. An image generation device comprising a controller that generates a peripheral image of a vehicle, the controller: obtaining a camera image from a camera that images the periphery of the vehicle; obtaining attitude information indicating the attitude of the camera with respect to the road surface; determining parameters for use in viewpoint conversion based on the attitude information; generating a viewpoint conversion image showing a peripheral area of the vehicle as seen from a virtual viewpoint based on the parameters and the camera image; storing at least a part of the image included in the viewpoint conversion image in a memory as a past viewpoint conversion image; and generating a composite image by synthesizing the viewpoint conversion image and the past viewpoint conversion image.

Images