Image generating device and image generating method

Abstract

To provide an image generating device and an image generating method that can output images that make it easy to understand the surroundings of a vehicle.SOLUTION: An image generating device according to the present disclosure includes a processor that acquires a first image that shows the surrounding environment of a vehicle captured by an imaging device provided on the vehicle, generates a plurality of second images on the basis of the first image, viewed from a virtual viewpoint provided above the vehicle in a plurality of line-of-sight directions each of which faces in a different direction with a horizontal component circling around the vehicle, and switches between and outputs the plurality of second images on a display device.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to an image generation device and an image generation method.

Background Art

[0002] Conventionally, there has been a technique of arranging a virtual viewpoint outside a vehicle, generating an image of the periphery of the vehicle as seen from the virtual viewpoint, and displaying the generated image on a display device in the vehicle interior (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present disclosure is to provide an image generation device and an image generation method capable of outputting an image that easily grasps the surrounding environment of a vehicle.

Means for Solving the Problems

[0005] The image generation device according to the present disclosure acquires a first image reflecting the surrounding environment of the vehicle imaged by an imaging device provided in the vehicle, and based on the first image, generates a plurality of second images seen in a plurality of viewing directions in which the horizontal components face different directions that go around the periphery of the vehicle from a virtual viewpoint provided above the vehicle, and includes a processor that switches and outputs the plurality of second images to a display device.

Effects of the Invention

[0006] According to the present disclosure, it is possible to provide an image generation device and an image generation method capable of outputting an image that easily grasps the periphery of a vehicle.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate. However, a more detailed description than necessary may be omitted. Note that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0009] (First Embodiment) FIG. 1 is a top view of a vehicle 1000 equipped with an image generation device 1 according to the first embodiment. In some of the figures after this figure, a coordinate axis is shown with the forward direction of the vehicle 1000 as the Y-axis direction, the right side direction of the vehicle 1000 as the X-axis direction, and the upward direction of the vehicle 1000 as the Z-axis direction. Also, for other figures showing a vehicle model 200 (described later), which is a 3D model of the vehicle 1000, the corresponding coordinate axes are displayed.

[0010] The vehicle 1000 is an automobile having a drive source such as an internal combustion engine or an electric motor. The vehicle 1000 includes a plurality of wheels 3 (here, four wheels 3), and one or more of the plurality of wheels 3 are driven by the drive source.

[0011] The vehicle 1000 includes a passenger compartment (not shown). The operation of the vehicle 1000 is controlled by a passenger in the passenger compartment. Note that part or all of the operation control of the vehicle 1000 may be automated.

[0012] The vehicle 1000 is provided with a plurality of imaging devices 2. In the first embodiment, four imaging devices 2a to 2d are provided in the vehicle 1000 as the plurality of imaging devices 2. Each imaging device 2 is an imaging device incorporating an image sensor such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor), for example. Each imaging device 2 can output images at a predetermined frame rate. Each imaging device 2 has a wide-angle lens or a fish-eye lens, respectively. Therefore, each imaging device 2 can image a wide range of areas in the direction of its optical axis. Further, the optical axis of each imaging device 2 is directed at least outward from the vehicle 1000 in plan view, whereby each imaging device 2 can image the surrounding environment including the road surface around the vehicle 1000.

[0013] Specifically, the imaging device 2a is provided at the front part of the vehicle 1000 (for example, the front grille). The optical axis of the imaging device 2a is directed forward of the vehicle or slightly downward from the forward direction of the vehicle. Therefore, the imaging device 2a can image a wide area in front of the vehicle.

[0014] The imaging device 2b is provided at the right side part of the vehicle 1000 (for example, the lower part of the right door mirror). The optical axis of the imaging device 2b is directed rightward of the vehicle or slightly downward from the rightward direction of the vehicle. Therefore, the imaging device 2b can image a wide area on the right side of the vehicle.

[0015] The imaging device 2c is provided at the rear part of the vehicle 1000 (for example, the rear gate). The optical axis of the imaging device 2c is directed rearward of the vehicle or slightly downward from the rearward direction of the vehicle. Therefore, the imaging device 2c can image a wide area behind the vehicle.

[0016] The imaging device 2d is provided at the left side part of the vehicle 1000 (for example, the lower part of the left door mirror). The optical axis of the imaging device 2d is directed leftward of the vehicle or slightly downward from the leftward direction of the vehicle. Therefore, the imaging device 2d can image a wide area on the left side of the vehicle.

[0017] Note that the position where each imaging device 2 is provided is not limited to the above example. As long as it is possible to image the surrounding environment of the vehicle 1000, the number of imaging devices 2 provided in the vehicle 1000 and the position of the imaging device 2 are arbitrary.

[0018] An image generation device 1 is provided in the vehicle 1000. The image generation device 1 may be, for example, one of a group of ECUs (Electronic Control Units). The image generation device 1 executes arithmetic processing and image processing based on an image showing the surrounding environment imaged by the four imaging devices 2, and outputs the image after the arithmetic processing and image processing to a display device (display device 5 described later) provided in the vehicle interior.

[0019] FIG. 2 is a diagram showing an example of the hardware configuration of the image generation device 1 according to the first embodiment.

[0020] The imaging devices 2a to 2d, the input device 4, and the display device 5 are connected to the image generation device 1. Some or all of the imaging devices 2a to 2d, the input device 4, and the display device 5 may be connected to the image generation device 1 via a network such as CAN (Controller Area Network).

[0021] The input device 4 is provided in the vehicle interior. The input device 4 is a touch panel, a switch, a dial, a joystick, or a push button, etc. The occupant of the vehicle 1000 can input various instructions to the image generation device 1 via the input device 4.

[0022] The display device 5 is provided in the vehicle interior. The display device 5 is, for example, an LCD (liquid crystal display) or an OELD (organic electroluminescent display), etc. The display device 5 displays the image output by the image generation device 1 so that the occupant can view it.

[0023] The image generation device 1 includes a CPU (Central Processing Unit) 10, a RAM (Random Access Memory) 11, a ROM (Read Only Memory) 12, an I / O (Input / Output) interface 13, and a bus 14. The CPU 10, the RAM 11, the ROM 12, and the I / O interface 13 are electrically connected to the bus 14.

[0024] The I / O interface 13 is an interface for the image generation device 1 to communicate data with an external device. Here, the external devices are the imaging devices 2a to 2d, the input device 4, and the display device 5. Note that the image generation device 1 may include a plurality of I / O interfaces 13, and the imaging devices 2a to 2d, the input device 4, and the display device 5 may be connected to different I / O interfaces 13 respectively. In FIG. 2, for simplicity of explanation, it is assumed that the image generation device 1 includes one I / O interface 13 to which these are connected.

[0025] The ROM 12 stores a computer program and parameters necessary for the execution of the computer program. The RAM 11 temporarily stores various data used in the operations by the CPU 10. The CPU 10 is a processor capable of executing a computer program.

[0026] Here, the ROM 12 stores an image generation program 100. The CPU 10 loads the image generation program 100 from the ROM 12 to the RAM 11 at a predetermined timing such as at startup. Then, the CPU 10 realizes the image generation method according to the first embodiment based on the image generation program 100 in the RAM 11.

[0027] In the first embodiment, the CPU 10 acquires, via the I / O interface 13, an image showing the surrounding environment of the vehicle 1000 captured by the imaging devices 2a to 2d. Then, based on the images acquired from the imaging devices 2a to 2d, the CPU 10 generates a plurality of images viewed in a plurality of line-of-sight directions each having a horizontal component directed in a different direction that circles around the vehicle 1000 from a virtual viewpoint provided above the vehicle 1000. Then, the CPU 10 switches and outputs the generated plurality of images to the display device 5 via the I / O interface 13. The images generated by the CPU 10 are referred to as surrounding images.

[0028] FIG. 3 is a diagram for explaining the virtual viewpoint for generating the surrounding image of the first embodiment and the line-of-sight directions from the virtual viewpoint.

[0029] When generating the surrounding image, the CPU 10 provides the vehicle model 200, which is a 3D model showing the vehicle 1000, on a virtual plane. Then, the CPU 10 provides a virtual viewpoint at a position L0 directly above the vehicle model 200.

[0030] The CPU 10 sets the line-of-sight direction from the virtual viewpoint so as to face outward from the vehicle model 200 in plan view. In other words, the CPU 10 sets the line-of-sight direction so that the horizontal component of the line-of-sight direction faces outward from the vehicle model 200 to the outside of the vehicle model 200.

[0031] For each of the captured images acquired from the imaging devices 2a to 2d, the CPU 10 performs a viewpoint conversion based on the relationship between the position of the imaging device 2 that captured the image and the position L0 of the virtual viewpoint. Thereby, the CPU 10 generates an image of the surrounding environment seen from the virtual viewpoint. Further, the CPU 10 superimposes an image of the vehicle model 200 seen from the virtual viewpoint (referred to as a vehicle model image) on the generated image. The CPU 10 outputs the image with the vehicle model image superimposed as the surrounding image.

[0032] Furthermore, the CPU 10 can rotate the line of sight direction about a straight line passing through position L0 and extending in the Z-axis direction. Then, the CPU 10 generates peripheral images for different lines of sight directions and outputs them to the display device 5. In other words, the CPU 10 generates a plurality of peripheral images seen in a plurality of lines of sight directions, each with a horizontal component facing a different direction that circles around the vehicle 1000, and switches and outputs the generated plurality of peripheral images to the display device 5.

[0033] As described above, the optical axes of the imaging devices 2a to 2d are directed outward from the vehicle 1000 in a plan view. Also, as described with reference to FIG. 3, the line of sight direction from the virtual viewpoint is set so that it faces outward from the vehicle model 200 in a plan view. That is, according to the first embodiment, the directions of the optical axes of the imaging devices 2a to 2d and the line of sight direction from the virtual viewpoint coincide in that they both face outward from the vehicle 1000 (vehicle model 200) in a plan view.

[0034] For comparison with the first embodiment, consider a case where the line of sight direction from the virtual viewpoint faces the vehicle model from outside the vehicle model in a plan view. This case is referred to as the comparison case. According to the comparison case, the line of sight direction from the virtual viewpoint is different from the direction of the optical axis of the imaging device in terms of whether it faces the vehicle (vehicle model) in a plan view. Therefore, when there is a three-dimensional object around the vehicle, in the peripheral image, the three-dimensional object appears to fall to the ground.

[0035] In contrast, according to the first embodiment, the directions of the optical axes of the imaging devices 2a to 2d and the line of sight direction from the virtual viewpoint coincide in that they both face outward from the vehicle 1000 (vehicle model 200) in a plan view. Therefore, when there is a three-dimensional object around the vehicle 1000, the CPU 10 can generate a peripheral image in which the three-dimensional object appears to stand upright from the ground. Thus, according to the first embodiment, unlike the comparison case, it is possible to generate a peripheral image that allows the occupant to perceive the three-dimensional object without a sense of discomfort.

[0036] As shown in FIG. 3, in this specification, the angle formed by the positive direction of the Y-axis and the line-of-sight direction in a plan view is denoted as the rotation angle. Further, the angle formed by the negative direction of the Z-axis and the line-of-sight direction is denoted as the depression angle. Note that the definitions of these rotation angle and depression angle are merely examples.

[0037] The trigger for the operation of rotating the line-of-sight direction is not limited to a specific event.

[0038] In one example, the CPU 10 may sequentially generate peripheral images while continuously changing the rotation angle in the positive or negative direction at a constant angular velocity, and sequentially output each generated peripheral image to the display device 5.

[0039] In another example, the CPU 10 may change the line-of-sight direction based on an input via an input device from the occupant, and generate and output a peripheral image each time the line-of-sight direction is changed.

[0040] The vehicle model 200 has a height corresponding to the vehicle 1000. Therefore, there is a possibility that a part of the area where the peripheral environment in the viewpoint-converted image is reflected is hidden by the vehicle model image. The portion of the area where the peripheral environment is reflected and is hidden by the vehicle model image is denoted as the road surface cut-off. Also, the area of the road surface cut-off is denoted as the road surface cut-off area.

[0041] When the road surface cut-off occurs, the occupant cannot confirm the situation of the road surface in the immediate vicinity of the vehicle 1000. Therefore, the CPU 10 restricts the height of the vehicle model 200 when generating the vehicle model image in order to suppress the road surface cut-off area.

[0042] FIG. 4 is a diagram for explaining the operation of restricting the height of the vehicle model 200 by the image generation device 1 according to the first embodiment. As shown in this figure, the CPU 10 generates the vehicle model 200 after height restriction by uniformly compressing the dimension of the original vehicle model 200 in the Z-axis direction.

[0043] Incidentally, the vehicle model 200 before the height limit is denoted as the first vehicle model 201. The vehicle model 200 after the height limit is denoted as the second vehicle model 202.

[0044] In order for the occupant to more easily grasp the surrounding environment, it is desirable that a part of the vehicle 1000 (more precisely, the vehicle model 200) is reflected in the surrounding image. For example, when there is an obstacle (person or object) around the vehicle 1000, if a part of the vehicle 1000 in addition to the obstacle is reflected in the surrounding image, it becomes easier for the occupant to grasp the positional relationship between the obstacle and the vehicle 1000.

[0045] However, if the area of the image of the vehicle model 200, that is, the vehicle model image, is too large, the area of the region reflecting the surrounding environment becomes relatively small, and the information on the surrounding environment obtained by the occupant from the surrounding image decreases. Therefore, it is desirable that the balance between the area of the vehicle model image and the area of the region reflecting the surrounding environment is appropriate. This balance is denoted as the display balance.

[0046] A normal vehicle has an elongated shape extending in the front-rear direction (in the example shown in FIG. 1, the Y-axis direction). Therefore, when the depression angle is fixed and the rotation angle is changed, the display balance changes according to the rotation angle. Specifically, when the rotation angle is 90 degrees or 270 degrees corresponding to the right side or the left side direction of the vehicle 1000, the area of the region where the image of the vehicle model 200 is displayed decreases compared to the case where the rotation angle is 0 degrees or 180 degrees corresponding to the front direction or the rear direction of the vehicle 1000.

[0047] Therefore, in the first embodiment, the CPU 10 changes the depression angle according to the rotation angle in the line-of-sight direction so that the above balance is maintained at an appropriate level as much as possible.

[0048] FIG. 5 is a diagram for explaining an example of a method for setting the depression angle in the first embodiment. In this figure, the horizontal axis represents the rotation angle, and the vertical axis represents the depression angle. In this specification, the numerical values of the rotation angle and the depression angle are expressed in degrees.

[0049] In the example shown in FIG. 5, when the rotation angle is 0 degrees or 180 degrees corresponding to the forward or rearward direction of the vehicle 1000, the depression angle is set to the maximum value of 30 degrees. And when the rotation angle is 90 degrees or 270 degrees corresponding to the rightward or leftward direction of the vehicle 1000, the depression angle is set to the minimum value of 20 degrees.

[0050] In this way, when the rotation angle is 90 degrees or 270 degrees corresponding to the rightward or leftward direction of the vehicle 1000 and the depression angle is reduced, a decrease in the area of the region where the image of the vehicle model 200 is displayed is suppressed. As a result, a change in the balance between the area of the region where the image of the vehicle model 200 is displayed and the area of the region reflecting the surrounding environment is suppressed.

[0051] Note that in the example shown in FIG. 5, the depression angle is smoothly changed in a sine wave shape with respect to the rotation angle. The change in the depression angle with respect to the rotation angle is not limited to a sine wave shape. The depression angle may be smoothly changed with respect to the rotation angle, or may be changed stepwise with respect to the rotation angle. Note that when the depression angle is smoothly changed with respect to the rotation angle, the visibility by the occupant is improved compared to the case where the depression angle is changed stepwise with respect to the rotation angle.

[0052] FIG. 6 is a diagram showing an example of a surrounding image displayed on the display device 5.

[0053] The surrounding image D101 is a surrounding image when the rotation angle is 0 degrees. The surrounding image D102 is a surrounding image when the rotation angle is 45 degrees. The surrounding image D103 is a surrounding image when the rotation angle is 90 degrees.

[0054] According to the example shown in FIG. 5, the set value of the depression angle when the rotation angle is 0 degrees is 30 degrees, the set value of the depression angle when the rotation angle is 45 degrees is 25 degrees, and the set value of the depression angle when the rotation angle is 90 degrees is 20 degrees. Therefore, the surrounding image D101 is generated at a depression angle of 30 degrees, the surrounding image D102 is generated at a depression angle of 25 degrees, and the surrounding image D103 is generated at a depression angle of 20 degrees.

[0055] As can be seen from FIG. 6, in each of the peripheral images D101, D102, and D103, it can be read that the display balance is maintained at an appropriate level.

[0056] Therefore, the occupant can easily grasp the positional relationship between the surrounding environment reflected in the peripheral image and the vehicle 1000 regardless of the rotation angle.

[0057] There are blind spots, that is, areas that cannot be imaged, on the vehicle 1000 side for each imaging device 2. Therefore, in the peripheral image, in the substantially rectangular area where the vehicle 1000 is present, the image obtained from the captured image cannot be projected. The non-imaging area 400 shown in FIG. 6 is an area that cannot project the image obtained from the captured image.

[0058] The above-described road surface chipping refers to the portion where the vehicle model image 300 protrudes from the non-imaging area 400. In the first embodiment, the vehicle model image 300 is generated based on the vehicle model 200 with height limitation, that is, the second vehicle model 202. Therefore, the area of the portion where the vehicle model image 300 protrudes from the non-imaging area 400, that is, the road surface chipping area, can be suppressed.

[0059] Subsequently, the operation of the image generation device 1 of the first embodiment will be described.

[0060] FIG. 7 is a flowchart showing the operation of outputting a peripheral image to the display device 5 by the image generation device 1 of the first embodiment. Here, as an example, the operation of outputting one frame of the peripheral image will be described. That is, the series of operations shown in this figure are repeatedly executed for each frame.

[0061] The CPU 10 first acquires captured images from the four imaging devices 2a to 2d via the I / O interface 13 (S101). The CPU 10 generates the second vehicle model 202 which is the vehicle model 200 with height limitation (S102).

[0062] The CPU 10 executes an image generation process for generating a peripheral image (S103). Then, the CPU 10 outputs the generated peripheral image to the display device 5 via the I / O interface 13 (S104). Then, the operation for one frame is completed.

[0063] FIG. 8 is a flowchart showing the image generation process of the first embodiment.

[0064] The CPU 10 sets a rotation angle for generating a peripheral image (S201).

[0065] The method of setting the rotation angle is not limited to a specific method. When continuously changing the rotation angle at a constant angular velocity, the CPU 10 obtains the rotation angle for generating the current peripheral image by adding a value of a predetermined step width to the rotation angle used when generating the peripheral image of the previous frame. When the rotation angle is specified by an instruction from the occupant, the CPU 10 sets the specified rotation angle as the rotation angle for generating the peripheral image.

[0066] Subsequent to S201, the CPU 10 sets a depression angle corresponding to the rotation angle set by the process of S201 (S202). For example, the CPU 10 obtains the depression angle corresponding to the rotation angle based on the correspondence shown in FIG. 5 and sets it as the depression angle for generating the peripheral image.

[0067] The CPU 10 generates a peripheral image using the four captured images acquired from the four imaging devices 2a to 2d by the process of S101, the second vehicle model 202 generated by the process of S102, and the line-of-sight direction determined by the rotation angle and the depression angle acquired by the processes of S201 and S202 (S203). Then, the image generation process ends.

[0068] Thus, according to the first embodiment, the CPU 10 acquires, via the I / O interface 13, an image showing the surrounding environment of the vehicle 1000 captured by the imaging devices 2a to 2d provided in the vehicle 1000. Then, based on the images acquired from the imaging devices 2a to 2d, the CPU 10 generates a plurality of peripheral images of the surrounding environment of the vehicle 1000 as seen in a plurality of line-of-sight directions in which the horizontal components face different directions that go around the vehicle 1000 from a virtual viewpoint provided above the vehicle 1000. Then, the CPU 10 outputs, via the I / O interface 13, the plurality of peripheral images to the display device 5 by switching them.

[0069] The directions of the optical axes of the imaging devices 2a to 2d and the line-of-sight directions from the virtual viewpoint coincide at points that face outward from the vehicle 1000 (vehicle model 200) in plan view. Therefore, the CPU 10 can generate peripheral images that allow the occupant to comfortably recognize the three-dimensional objects around the vehicle 1000. That is, the image generation device 1 can output an image that makes it easy to grasp the surrounding environment of the vehicle 1000.

[0070] Also, according to the first embodiment, the CPU 10 sets the depression angle so that a part of the vehicle model 200 is reflected in each peripheral image.

[0071] Therefore, when there are obstacles around the vehicle 1000, the occupant can easily grasp the positional relationship between the vehicle 1000 and the obstacles.

[0072] Also, the CPU 10 sets the depression angle to change in a sine wave shape with respect to the rotation angle.

[0073] Since the depression angle changes smoothly with respect to the rotation angle, the visibility for the occupant is improved.

[0074] Also, when generating the vehicle model image, the CPU 10 restricts the height of the vehicle model 200.

[0075] Therefore, it becomes possible to output a peripheral image that allows the occupant to confirm the condition of the road surface in the immediate vicinity of the vehicle 1000.

[0076] (Second Embodiment) In the second embodiment, the CPU 10 sets the depression angle so that the display balance, that is, the balance between the area of the vehicle model image and the area of the region reflecting the surrounding environment, becomes more precisely constant. Specifically, a target range of the area of the vehicle model image in the surrounding image is set in advance, and the CPU 10 sets the depression angle so that the area of the vehicle model image in the surrounding image is within the target range.

[0077] In the following description of the second embodiment, matters different from the first embodiment will be described. Regarding matters similar to the first embodiment, the description will be omitted or briefly described.

[0078] FIG. 9 is a flowchart showing the image generation process of the second embodiment.

[0079] First, the CPU 10 sets a rotation angle for generating a surrounding image (S301). In S301, the CPU 10 sets the rotation angle in the same manner as in S201.

[0080] Subsequently, the CPU 10 temporarily sets the depression angle (S302). In S302, the CPU 10 may use, as the temporary setting value, the depression angle used when generating the surrounding image of the previous frame. Alternatively, when an initial value of the depression angle is set in advance, the CPU 10 may use the initial value of the depression angle as the temporary setting value of the depression angle.

[0081] Subsequently, the CPU 10 calculates the area of the vehicle model image in the surrounding image when generating the surrounding image using the line-of-sight direction determined by the temporarily set rotation angle and depression angle (S303). Then, the CPU 10 determines whether the area of the vehicle model image obtained by the calculation is within a preset target range (S304). The target range of the area of the vehicle model image may be set in advance by the designer or may be settable by the occupant.

[0082] If the area of the vehicle model image is not within the target range (S304: No), the CPU 10 determines whether the area of the vehicle model image is less than the lower limit of the target range (S305).

[0083] If the area of the vehicle model image is less than the lower limit of the target range (S305: Yes), the CPU 10 decreases the provisional setting value of the depression angle by, for example, a predetermined amount (S306).

[0084] If the area of the vehicle model image is not less than the lower limit of the target range (S305: No), that is, if the area of the vehicle model image exceeds the upper limit of the target range, the CPU 10 increases the provisional setting value of the depression angle by, for example, a predetermined amount (S307).

[0085] After S306 or S307, the control transitions to S303, and the CPU 10 recalculates the area of the vehicle model image using the adjusted provisional setting value of the depression angle.

[0086] If it is determined in the determination process of S304 that the area of the vehicle model image is within the target range (S304: Yes), the CPU 10 sets the provisional setting value of the depression angle as the depression angle to be used for generating the peripheral image (S308).

[0087] The CPU 10 generates a peripheral image using the four captured images acquired from the four imaging devices 2a to 2d by the process of S101, the second vehicle model 202 generated by the process of S102, the rotation angle set by the process of S301, and the line-of-sight direction determined by the depression angle determined by the process of S308 (S309). Then, the image generation process according to the second embodiment ends.

[0088] In the above description, the CPU 10 adjusted the depression angle based on the area of the vehicle model 200 reflected in the peripheral image every time the image generation process was performed. A correspondence relationship between the rotation angle and the depression angle such that the area of the vehicle model image is within the target range may be stored in advance in a predetermined storage area of the image generation apparatus 1. Then, in the image generation process, the CPU 10 may set the depression angle used for generating the peripheral image based on the correspondence relationship. The CPU 10 may obtain the correspondence relationship in advance by executing the processes of S302 to S308 for each different set value of the rotation angle.

[0089] As described above, in the second embodiment, the CPU 10 sets the depression angle based on the area of the vehicle model 200 reflected in the peripheral image.

[0090] Therefore, since the display balance is controlled to be constant regardless of the rotation angle, the amount of information on the peripheral environment that the occupant can obtain from the peripheral image can be made constant regardless of the rotation angle.

[0091] (Third Embodiment) In the third embodiment, the CPU 10 sets the depression angle so that at least a part of the dimensions of the area in which the vehicle model image included in the peripheral image is reflected becomes constant, thereby making the display balance as constant as possible. Here, as an example, the CPU 10 sets the depression angle so that the length in the vertical direction in the peripheral image among the dimensions of the area in which the vehicle model image is reflected becomes constant.

[0092] In the following description of the third embodiment, matters different from those of the second embodiment will be described. Descriptions of matters the same as those of the second embodiment will be omitted or briefly described.

[0093] FIG. 10 is a flowchart showing the image generation process of the third embodiment.

[0094] The CPU 10 sets the rotation angle for generating the peripheral image (S401) and temporarily sets the depression angle (S402) in the same manner as in S301 and S302.

[0095] Subsequently, the CPU 10 calculates the display destination coordinates of the upper end point of the vehicle model image in the peripheral image when generating the peripheral image using the line-of-sight direction determined by the provisional setting values of the rotation angle and the depression angle (S403). Then, the CPU 10 determines whether the upper end point coordinates obtained by the calculation are within the target range (S404).

[0096] In the third embodiment, the range of the coordinates of the upper end point of the vehicle model image that can make the display balance appropriate is set in advance as the target range. The target range of the coordinates of the upper end point may be set by the designer or may be settable by the occupant.

[0097] When the upper end point coordinates are not within the target range (S404: No), the CPU 10 determines whether the upper end point coordinates are less than the lower limit value of the target range (S405).

[0098] When the upper end point coordinates are less than the lower limit value of the target range (S405: Yes), the CPU 10 decreases the provisional setting value of the depression angle by, for example, a predetermined amount (S406).

[0099] When the upper end point coordinates are not less than the lower limit value of the target range (S405: No), that is, when the upper end point coordinates exceed the upper limit value of the target range, the CPU 10 increases the provisional setting value of the depression angle by, for example, a predetermined amount (S407).

[0100] After S406 or S407, the control transitions to S403, and the CPU 10 recalculates the display destination coordinates of the upper end point of the vehicle model image using the adjusted provisional setting value of the depression angle.

[0101] When it is determined in the determination process of S404 that the on-vehicle upper end point coordinates are within the target range (S404: Yes), the CPU 10 determines the provisional setting value of the depression angle as the depression angle used for generating the peripheral image (S408).

[0102] The CPU 10 generates a peripheral image (S409) using the four captured images acquired from the four imaging devices 2a to 2d by the process of S101, the second vehicle model 202 generated by the process of S102, the rotation angle set by the process of S401, and the line-of-sight direction determined by the depression angle determined by the process of S408. Then, the image generation process according to the third embodiment ends.

[0103] In the above description, the CPU 10 adjusted the depression angle based on the dimensions of the vehicle model 200 reflected in the peripheral image. A correspondence relationship between the rotation angle and the depression angle such that the dimensions of the vehicle model 200 reflected in the peripheral image are within the target range may be stored in advance in a predetermined storage area of the image generation device 1. In the image generation process, the CPU 10 may acquire the depression angle used for generating the peripheral image based on the correspondence relationship. The CPU 10 may acquire the correspondence relationship in advance by executing the processes of S402 to S408 for each different set value of the rotation angle.

[0104] As described above, according to the third embodiment, the CPU 10 sets the depression angle based on the dimensions of the vehicle model 200 reflected in the peripheral image.

[0105] Therefore, similar to the second embodiment, since the display balance is controlled to be constant regardless of the rotation angle, the amount of information on the surrounding environment that the occupant can obtain from the peripheral image can be made constant regardless of the rotation angle.

[0106] (Fourth Embodiment) In the first to third embodiments, the CPU 10 set the depression angle so that a part of the vehicle model 200 is reflected in each peripheral image. The method for causing a part of the vehicle model 200 to be reflected in each peripheral image is not limited to this.

[0107] In the fourth embodiment, as an example of a variation of the method for causing a part of the vehicle model 200 to be reflected in each peripheral image, an example of moving the position of the virtual viewpoint will be described.

[0108] FIG. 11 is a diagram for explaining an operation of moving a virtual view point, which is executed by the image generation apparatus 1 according to the fourth embodiment. As shown in this figure, instead of changing the depression angle, the CPU 10 moves the position of the virtual view point.

[0109] The CPU 10 may move the virtual view point laterally with respect to the vehicle 1000 above the vehicle 1000 (more precisely, the vehicle model 200). The positions L1 and L2 are examples of the positions of the destinations when the virtual view point is moved laterally with respect to the vehicle 1000.

[0110] Alternatively, the CPU 10 may move the virtual view point vertically (in other words, in the X-axis direction) with respect to the vehicle 1000 above the vehicle 1000. The position L3 is an example of the position of the destination when the virtual view point is moved vertically with respect to the vehicle 1000.

[0111] The correspondence between the rotation angle and the position of the virtual view point is preset, and the CPU 10 may move the virtual view point based on the correspondence.

[0112] Alternatively, the CPU 10 may move the virtual view point based on the area of the vehicle model 200 reflected in the surrounding image.

[0113] Alternatively, the CPU 10 may move the virtual view point based on the dimensions of the vehicle model 200 reflected in the surrounding image.

[0114] FIG. 12 is a diagram showing an example of a surrounding image obtained by moving the virtual view point. Here, the surrounding images before and after the movement of the virtual view point are shown when the control for moving the virtual view point based on the area of the vehicle model 200 reflected in the surrounding image is executed.

[0115] The peripheral image D201 is a peripheral image generated based on the settings before the movement of the virtual viewpoint. Specifically, the peripheral image D201 is a peripheral image generated when the rotation angle is set to 90 degrees, the depression angle is set to 30 degrees, and the virtual viewpoint is set to L0. According to this peripheral image D201, since the area of the region where the vehicle model image 300 is reflected is extremely small, it is difficult for the occupant to grasp the positional relationship between the vehicle 1000 and the surrounding environment.

[0116] The peripheral image D211 is a peripheral image generated based on the settings after the movement of the virtual viewpoint. Specifically, the peripheral image D111 is a peripheral image generated when the rotation angle is set to 90 degrees, the depression angle is set to 30 degrees, and the virtual viewpoint is set to L2. According to this peripheral image D111, it can be seen that the area of the region where the vehicle model image 300 is reflected is larger than that of the peripheral image D201, and the display balance is significantly improved.

[0117] In this way, the CPU 10 may set the viewpoint position according to the line-of-sight direction so that a part of the vehicle model 200 is reflected in each peripheral image.

[0118] (Fifth Embodiment) In the fifth embodiment, variations in the operation when generating the vehicle model image will be described. The operation of the fifth embodiment can be used in combination with the operations of the first to fourth embodiments.

[0119] FIG. 13 is a flowchart showing the operation of generating the vehicle model image executed by the image generation device 1 of the fifth embodiment.

[0120] First, the CPU 10 calculates the color value of the first vehicle model 201 by performing shading processing based on the parameters of the first vehicle model 201 (S501). The shading processing is a process of shading considering the lighting conditions and the like. The parameters of the first vehicle model 201 include the coordinates of a large number of polygon vertices that define the shape of the first vehicle model 201. That is, in S501, the CPU 10 calculates the color value after shading based on the shape of the vehicle model 200 before the height limit.

[0121] The CPU 10 calculates the display position coordinates in the peripheral image based on the parameters of the second vehicle model 202 (S502).

[0122] The CPU 10 applies the color value obtained by the calculation in S501 to the display position coordinates of the second vehicle model 202 (S503), and the operation of generating the vehicle model image ends.

[0123] In this way, the CPU 10 applies the colored value after shading to the vehicle model 200 after height limitation based on the shape of the vehicle model 200 before height limitation.

[0124] Therefore, it is possible to suppress the loss of the three-dimensional effect due to the height limitation. Note that the colored value after shading based on the shape of the vehicle model 200 before height limitation may be calculated in advance and applied to the vehicle model 200 after height limitation.

[0125] (Sixth Embodiment) In the sixth embodiment, a variation of the operation of limiting the height of the vehicle model 200 will be described. The operation of the sixth embodiment is applicable to any of the first to fifth embodiments.

[0126] FIG. 14 is a flowchart showing the operation of limiting the height of the vehicle model 200, which is executed by the image generation device 1 according to the sixth embodiment.

[0127] First, when setting the rotation angle in the image generation process (S601), the CPU 10 temporarily sets the height of the vehicle model 200 (S602). In S602, the CPU 10 may use the height used when generating the peripheral image of the previous frame as the temporary setting value. Alternatively, when the initial value of the height is set in advance, the CPU 10 may use the initial value as the temporary setting value of the height.

[0128] Subsequently, the CPU 10 calculates the road surface chipping area when generating a peripheral image using the temporarily set values of the rotation angle and height (S603). Then, the CPU 10 determines whether the road surface chipping area is less than or equal to a threshold value (S604). The threshold value is set in advance. The threshold value may be set by a designer or may be settable by an occupant.

[0129] If the road surface chipping area is not less than or equal to the threshold value (S604: No), the temporarily set value of the height is decreased by, for example, a predetermined amount (S605). Then, the control transitions to S603, and the CPU 10 recalculates the area of the vehicle model image using the adjusted temporarily set value of the height.

[0130] If the road surface chipping area is less than or equal to the threshold value (S604: Yes), the CPU 10 determines the temporarily set value as the height to be used for the height limit (S606). Then, the CPU 10 generates the second vehicle model 202 using the determined height (S607), and the operation of limiting the height of the vehicle model 200 ends.

[0131] In the above description, the CPU 10 adjusted the height of the vehicle model 200 based on the road surface chipping area. A correspondence relationship between the rotation angle and the height such that the road surface chipping area is less than or equal to the threshold value may be stored in advance in a predetermined storage area of the image generation device 1. In the image generation process, the CPU 10 may acquire the height to be used for the height limit based on the correspondence relationship. The CPU 10 may acquire the correspondence relationship in advance by executing the processes of S602 to S606 for each different set value of the rotation angle.

[0132] In this way, the CPU 10 may set the height of the vehicle model 200 based on the road surface chipping area.

[0133] (Seventh Embodiment) In the seventh embodiment, another variation of the operation of limiting the height of the vehicle model 200 will be described. The operation of the seventh embodiment is applicable to any of the first to fifth embodiments.

[0134] FIG. 15 is a diagram for explaining a method of limiting the height of the vehicle model 200 executed by the image generation device 1 according to the seventh embodiment. The CPU 10 generates a second vehicle model 203, which is the vehicle model 200 after height limitation, by compressing the body above the wheels of the vehicle model 200. In the example shown in this figure, the CPU 10 compresses the portion of the body above a certain position H1 higher than the wheels.

[0135] The image generation device 1 may change and display the steering angle of the wheels included in the vehicle model 200 according to the actual steering angle so that the occupant can check the state of the wheels 3. Alternatively, the image generation device 1 may display the body of the vehicle model 200 semi-transparently or only display the outline, etc., so that the wheels hidden by the body can be visually recognized through the body. When those configurations are adopted, if all the bodies including the wheels are uniformly compressed, when the actual steering angle is not 0 degrees, the shape of the wheels reflected in the surrounding image becomes elliptical, which may make the occupant feel uncomfortable.

[0136] In the seventh embodiment, the CPU 10 compresses the body above the wheels and does not compress the wheels. Therefore, it is possible to prevent the shape of the wheels that can be reflected in the surrounding image from becoming elliptical.

[0137] (Eighth Embodiment) In the eighth embodiment, yet another variation of the operation of limiting the height of the vehicle model 200 will be described. The operation of the eighth embodiment is applicable to any of the first to fifth embodiments.

[0138] FIG. 16 is a diagram for explaining a method of limiting the height of the vehicle model 200 executed by the image generation device 1 according to the eighth embodiment.

[0139] The CPU 10 moves the polygon vertices of the vehicle model 200 along a straight line extending from the virtual viewpoint. In the example shown in FIG. 16, the line of sight direction is shown as an example of the straight line extending from the virtual viewpoint. Then, vertex F1, which is a polygon vertex of the vehicle model 200 on the straight line in the line of sight direction, is moved to the position of F2. The CPU 10 uniformly moves all the polygon vertices of the vehicle model 200 in the above-described manner to generate the vehicle model 200 after height restriction, that is, the second vehicle model 202.

[0140] In the first embodiment, the CPU 10 restricted the height of the vehicle model 200 by uniformly compressing the entire vehicle model 200. When the height is restricted in such a manner, the appearance of the vehicle model 200 as viewed from the virtual viewpoint may change before and after the height restriction, causing discomfort to the occupants.

[0141] In the eighth embodiment, the polygon vertices of the vehicle model 200 are moved along a straight line extending from the virtual viewpoint. Thereby, the appearance of the vehicle model 200 as viewed from the virtual viewpoint can be prevented from changing before and after the height restriction.

[0142] (Ninth Embodiment) In the ninth embodiment, yet another variation of the operation for restricting the height of the vehicle model 200 will be described. The operation of the ninth embodiment is applicable to any of the first to fifth embodiments.

[0143] FIG. 17 is a diagram for explaining a method of restricting the height of the vehicle model 200, which is executed by the image generation apparatus 1 according to the ninth embodiment.

[0144] The CPU 10 moves the polygon vertices of the vehicle model 200 along the body ridgeline of the vehicle model 200. For example, the CPU 10 moves a certain vertex F3 to the position of F4 along the body ridgeline. The CPU 10 uniformly moves all the polygon vertices of the vehicle model 200 in the above-described manner to generate the vehicle model 200 after height restriction, that is, the second vehicle model 202.

[0145] When performing height restriction by moving the polygon vertices of the vehicle model 200 along the body ridgeline, similar to the eighth embodiment, the appearance of the vehicle model 200 as viewed from the virtual viewpoint can be made not to change before and after the height restriction.

[0146] The image generation program 100, which is a computer program executed by the image generation apparatus 1 according to the first to eighth embodiments, is stored in advance in the ROM 12 of the image generation apparatus 1. The image generation program 100 may be provided by being recorded on a computer-readable recording medium such as a CD (Compact Disc)-ROM (Read Only Memory), a flexible disk (FD: Flexible Disc), a CD-R (Recordable), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc., in an installable format or an executable format file.

[0147] Furthermore, the image generation program 100 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0148] Part or all of the processing executed by the CPU 10 according to the image generation program 100 may be realized by an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), or the like.

[0149] Although some embodiments of the present disclosure have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, as well as in the invention described in the claims and the equivalent scope thereof.

Explanation of Signs

[0150] 1 Image generation device 2, 2a~2d Imaging device 3 Wheel 4 Input device 5 Display device 10 CPU 11 RAM 12 ROM 13 I / O interface 14 Bus 100 Image generation program 200 Vehicle model 201 First vehicle model 202 Second vehicle model 300 Vehicle model image 1000 Vehicle

Claims

1. Obtaining a first image showing the surrounding environment of the vehicle captured by an imaging device provided on the vehicle, and based on the first image, generating a plurality of second images viewed in a plurality of line-of-sight directions in which the horizontal component circulates around the vehicle from a virtual viewpoint provided above the vehicle, and a processor that switches and outputs the plurality of second images to a display device, An image generation device comprising:

2. The processor sets the depression angles of the plurality of line-of-sight directions so that a part of the vehicle model indicating the vehicle is reflected in the plurality of second images. The image generation device according to claim 1.

3. The processor sets the depression angles of the plurality of line-of-sight directions so as to change smoothly with respect to the directions of the horizontal components of the plurality of line-of-sight directions. The image generation device according to claim 1.

4. The processor sets the depression angles of the plurality of line-of-sight directions based on the area of the region in which the vehicle model is reflected in the plurality of second images. The image generation device according to claim 2.

5. The processor sets the depression angles of the plurality of line-of-sight directions based on the dimensions of the region in which the vehicle model is reflected in the plurality of second images. The image generation device according to claim 2.

6. The processor sets the position of the virtual viewpoint for each of the plurality of line-of-sight directions so that a part of the vehicle model indicating the vehicle is reflected in the plurality of second images. The image generation device according to claim 1.

7. The processor, generates a vehicle model image based on a three-dimensional vehicle model corresponding to the vehicle, superimposes and displays the vehicle model image on each of the plurality of second images, When generating the vehicle model image, restricting the height of the vehicle model The image generation device according to any one of claims 1 to 6.

8. The processor calculates the color value of the vehicle model by performing shading processing on the vehicle model before height restriction, and applies the color value to the vehicle model after height restriction. The image generation device according to claim 7.

9. The processor restricts the height of the vehicle model by compressing the body of the vehicle model above the wheels. The image generation device according to claim 7.

10. The processor restricts the height of the vehicle model by moving the polygon vertices of the vehicle model along a straight line extending from the virtual viewpoint. The image generation device according to claim 7.

11. Obtaining a first image showing the surrounding environment of the vehicle captured by an imaging device provided on the vehicle; Based on the first image, generating a plurality of second images viewed in a plurality of line-of-sight directions with horizontal components directed in different directions that circle around the vehicle from a virtual viewpoint provided above the vehicle; Switching and outputting the plurality of second images to a display device; An image generation method including:

12. Setting the depression angles of the plurality of line-of-sight directions such that a part of the vehicle model indicating the vehicle is reflected in the plurality of second images; The image generation method according to claim 11, further including:

13. Setting the depression angles of the plurality of line-of-sight directions to change smoothly with respect to the directions of the horizontal components of the plurality of line-of-sight directions; The image generation method according to claim 11, further including:

14. Setting the depression angles of the plurality of line-of-sight directions based on the area of the region in which the vehicle model is reflected in the plurality of second images. The image generation method according to claim 12, further comprising this.

15. Setting the depression angles of the plurality of line-of-sight directions based on the dimensions of the region in which the vehicle model is reflected in the plurality of second images. The image generation method according to claim 12, further comprising this.

16. Setting the position of the virtual viewpoint for each of the plurality of line-of-sight directions such that a part of the vehicle model showing the vehicle is reflected in the plurality of second images. The image generation method according to claim 11, further comprising this.

17. Generating a vehicle model image based on a three-dimensional vehicle model corresponding to the vehicle; Superimposing and displaying the vehicle model image on each of the plurality of second images; Limiting the height of the vehicle model when generating the vehicle model image; The image generation method according to any one of claims 11 to 16, further comprising this.

18. Calculating the color value of the vehicle model by performing shading processing on the vehicle model before height limitation, and applying the color value to the vehicle model after height limitation. The image generation method according to claim 17, further comprising this.

19. Limiting the height of the vehicle model means limiting the height of the vehicle model by compressing the body above the wheels of the vehicle model. The image generation method according to claim 17, which is this.

20. Limiting the height of the vehicle model means limiting the height of the vehicle model by moving the polygon vertices of the vehicle model along a straight line extending from the virtual viewpoint. The image generation method according to claim 17, which is this.

Images