Image generating device and image generating method

Abstract

To provide an image generating device and an image generating method that can output images that make it easy to understand the surroundings of a vehicle.SOLUTION: An image generating device according to the present disclosure includes a processor that acquires a first image that shows the surrounding environment of a vehicle captured by an imaging device provided on the vehicle, generates a plurality of second images on the basis of the first image, viewed from a virtual viewpoint provided above the vehicle in a plurality of line-of-sight directions each of which faces in a different direction with a horizontal component circling around the vehicle, and switches between and outputs the plurality of second images on a display device.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image generation device and an image generation method. [Background technology]

[0002] BACKGROUND ART There is a conventional technique in which a virtual viewpoint is placed outside a vehicle, an image of the surroundings of the vehicle viewed from the virtual viewpoint is generated, and the generated image is displayed on a display device inside the vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-8762 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to provide an image generation device and an image generation method that can output an image that makes it easy to grasp the surrounding environment of a vehicle. [Means for solving the problem]

[0005] The image generating device according to the present disclosure includes a processor that acquires a first image of the surrounding environment of the vehicle captured by an imaging device installed in the vehicle, generates, based on the first image, a plurality of second images viewed from a virtual viewpoint installed above the vehicle in a plurality of line-of-sight directions each facing in a different direction so that a horizontal component circles around the vehicle, and switches between the plurality of second images and outputs them to a display device. [Effects of the Invention]

[0006] According to the present disclosure, it is possible to provide an image generating device and an image generating method that can output an image that makes it easy to grasp the surroundings of a vehicle. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an external view of a vehicle equipped with an image generating device according to a first embodiment, as viewed from above. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the image generating apparatus according to the first embodiment. [Figure 3] FIG. 3 is a diagram for explaining a virtual viewpoint and a line-of-sight direction from the virtual viewpoint for generating a peripheral image according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a method for limiting the height of a vehicle model by the image generating device according to the first embodiment. [Figure 5] FIG. 5 is a diagram for explaining an example of a method for setting the depression angle in the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a peripheral image displayed on a display device by the image generating device of the first embodiment. [Figure 7] FIG. 7 is a flowchart showing the operation of the image generating device of the first embodiment to output a peripheral image to the display device. [Figure 8] FIG. 8 is a flowchart showing the image generation process of the first embodiment. [Figure 9] FIG. 9 is a flowchart showing the image generation process of the second embodiment. [Figure 10] FIG. 10 is a flowchart showing the image generation process of the third embodiment. [Figure 11] FIG. 11 is a diagram for explaining the operation of moving the virtual world, which is executed by the image generating device of the fourth embodiment. [Figure 12] FIG. 12 is a diagram showing an example of a peripheral image obtained by moving the virtual viewpoint. [Figure 13] FIG. 13 is a flowchart showing the operation of generating a vehicle model image, which is executed by the image generating device of the fifth embodiment. [Figure 14] FIG. 14 is a flowchart showing the operation of limiting the height of a vehicle model, which is executed by the image generating device of the sixth embodiment. [Figure 15]FIG. 15 is a diagram for explaining a method for limiting the height of a vehicle model, which is executed by the image generating device according to the seventh embodiment. [Figure 16] FIG. 16 is a diagram for explaining a method for limiting the height of a vehicle model, which is executed by the image generating device according to the eighth embodiment. [Figure 17] FIG. 17 is a diagram for explaining a method for limiting the height of a vehicle model, which is executed by the image generating device of the ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings as appropriate. However, more detailed description than necessary may be omitted. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0009] (First embodiment) 1 is an external view of a vehicle 1000 equipped with an image generating device 1 of the first embodiment, as seen from above. Note that in this and subsequent figures, coordinate axes are shown with the Y-axis direction extending forward of the vehicle 1000, the X-axis direction extending to the right of the vehicle 1000, and the Z-axis direction extending upward of the vehicle 1000. Corresponding coordinate axes are also shown in other figures showing a vehicle model 200 (described later), which is a 3D model showing the vehicle 1000.

[0010] The vehicle 1000 is an automobile having a drive source such as an internal combustion engine or an electric motor. The vehicle 1000 has a plurality of wheels 3 (four wheels 3 in this example), and one or more of the plurality of wheels 3 are driven by the drive source.

[0011] The vehicle 1000 includes a cabin (not shown). An occupant in the cabin controls the driving of the vehicle 1000. Note that part or all of the control of the driving of the vehicle 1000 may be automated.

[0012] A plurality of imaging devices 2 are provided on the vehicle 1000. In the first embodiment, four imaging devices 2a to 2d are provided on the vehicle 1000 as the plurality of imaging devices 2. Each imaging device 2 is an imaging device incorporating an imaging element such as a charge coupled device (CCD) or a CMOS image sensor (CIS). Each imaging device 2 can output images at a predetermined frame rate. Each imaging device 2 has a wide-angle lens or a fisheye lens. Therefore, each imaging device 2 can capture an image of a wide area toward which its optical axis is directed. Furthermore, the optical axis of each imaging device 2 is directed outward from the vehicle 1000 at least in a plan view, which allows each imaging device 2 to capture an image of the surrounding environment, including the road surface around the vehicle 1000.

[0013] Specifically, the imaging device 2a is provided at the front (for example, the front grille) of the vehicle 1000. The optical axis of the imaging device 2a is directed toward the front of the vehicle or slightly downward from the front of the vehicle. Therefore, the imaging device 2a can capture an image of a wide area in front of the vehicle.

[0014] The imaging device 2b is provided on the right side of the vehicle 1000 (for example, below the right door mirror). The optical axis of the imaging device 2b is directed toward the right side of the vehicle or slightly downward from the right side of the vehicle. Therefore, the imaging device 2b can capture an image of a wide area on the right side of the vehicle.

[0015] The imaging device 2c is provided at the rear (for example, at the rear gate) of the vehicle 1000. The optical axis of the imaging device 2c is directed toward the rear of the vehicle or slightly downward from the rear of the vehicle. Therefore, the imaging device 2c can capture an image of a wide area behind the vehicle.

[0016] The imaging device 2d is provided on the left side of the vehicle 1000 (for example, below the left door mirror). The optical axis of the imaging device 2d is directed toward the left side of the vehicle or slightly downward from the left side of the vehicle. Therefore, the imaging device 2d can capture an image of a wide area on the left side of the vehicle.

[0017] The locations at which the imaging devices 2 are installed are not limited to the above examples. The number of imaging devices 2 installed on the vehicle 1000 and the locations of the imaging devices 2 are arbitrary as long as they are capable of capturing images of the environment surrounding the vehicle 1000.

[0018] An image generating device 1 is provided in a vehicle 1000. The image generating device 1 may be, for example, one of a group of ECUs (Electronic Control Units). The image generating device 1 performs arithmetic processing and image processing based on images of the surrounding environment captured by four imaging devices 2, and outputs the images after the arithmetic processing and image processing to a display device (display device 5 described later) provided in the vehicle interior.

[0019] FIG. 2 is a diagram illustrating an example of the hardware configuration of the image generating device 1 according to the first embodiment.

[0020] Image generation device 1 is connected to imaging devices 2a to 2d, an input device 4, and a display device 5. Some or all of image generation devices 2a to 2d, input device 4, and display device 5 may be connected to image generation device 1 via a network such as a CAN (Controller Area Network).

[0021] The input device 4 is provided inside the vehicle cabin. The input device 4 is a touch panel, a switch, a dial, a joystick, a push button, or the like. An occupant of the vehicle 1000 can input various instructions to the image generation device 1 via the input device 4.

[0022] The display device 5 is provided in the vehicle interior. The display device 5 is, for example, an LCD (liquid crystal display) or an OLED (organic electroluminescent display). The display device 5 displays the image output by the image generation device 1 so that it can be viewed by the occupants.

[0023] The image generating device 1 includes a CPU (Central Processing Unit) 10, a RAM (Random Access Memory) 11, a ROM (Read Only Memory) 12, an I / O (Input / Output) interface 13, and a bus 14. The CPU 10, the RAM 11, the ROM 12, and the I / O interface 13 are electrically connected to the bus 14.

[0024] I / O interface 13 is an interface through which image generation device 1 communicates data with external devices. Here, the external devices are image capture devices 2a-2d, input device 4, and display device 5. Note that image generation device 1 may include multiple I / O interfaces 13, and image capture devices 2a-2d, input device 4, and display device 5 may each be connected to a different I / O interface 13. For ease of explanation, FIG. 2 shows image generation device 1 as including a single I / O interface 13 to which these are connected.

[0025] The ROM 12 stores computer programs and parameters necessary for executing the computer programs. The RAM 11 temporarily stores various data used in the calculations performed by the CPU 10. The CPU 10 is a processor capable of executing computer programs.

[0026] Here, an image generation program 100 is stored in the ROM 12. The CPU 10 loads the image generation program 100 from the ROM 12 into the RAM 11 at a predetermined timing, such as at the time of startup. Then, the CPU 10 realizes the image generation method according to the first embodiment based on the image generation program 100 in the RAM 11.

[0027] In the first embodiment, the CPU 10 acquires images of the surrounding environment of the vehicle 1000 captured by the imaging devices 2a to 2d via the I / O interface 13. Then, based on the images acquired from the imaging devices 2a to 2d, the CPU 10 generates a plurality of images viewed from a virtual viewpoint provided above the vehicle 1000 in a plurality of line-of-sight directions, each of which has a horizontal component facing in a different direction around the vehicle 1000. The CPU 10 then switches between the generated images and outputs them to the display device 5 via the I / O interface 13. The images generated by the CPU 10 are referred to as surrounding images.

[0028] FIG. 3 is a diagram for explaining a virtual viewpoint and a line-of-sight direction from the virtual viewpoint for generating a peripheral image according to the first embodiment.

[0029] When generating the surrounding image, the CPU 10 places a vehicle model 200, which is a 3D model representing the vehicle 1000, on a virtual plane. The CPU 10 then sets a virtual viewpoint at a position L0 directly above the vehicle model 200.

[0030] CPU 10 sets the line of sight direction from the virtual viewpoint so that it faces outward from vehicle model 200 in a planar view. In other words, CPU 10 sets the line of sight direction so that the horizontal component of the line of sight faces outward from vehicle model 200.

[0031] The CPU 10 performs viewpoint conversion on each of the captured images acquired from the imaging devices 2a to 2d based on the relationship between the position of the imaging device 2 that captured the image and the virtual viewpoint position L0. As a result, the CPU 10 generates an image of the surrounding environment viewed from the virtual viewpoint. Furthermore, the CPU 10 superimposes an image of the vehicle model 200 viewed from the virtual viewpoint (referred to as a vehicle model image) on the generated image. The CPU 10 outputs the image with the vehicle model image superimposed as a surrounding image.

[0032] Furthermore, CPU 10 can rotate the line of sight around a line that passes through position L0 and extends in the Z-axis direction as the rotation axis. CPU 10 then generates a peripheral image for each different line of sight direction and outputs it to display device 5. In other words, CPU 10 generates a plurality of peripheral images viewed from a plurality of line of sight directions in which the horizontal component faces in different directions around vehicle 1000, and switches between the generated peripheral images and outputs them to display device 5.

[0033] As described above, the optical axes of the image capture devices 2a to 2d are directed outward from the vehicle 1000 in a planar view. Also, as described with reference to Fig. 3, the line of sight direction from the virtual viewpoint is set so that the line of sight direction from the virtual viewpoint is directed outward from the vehicle model 200 in a planar view. That is, according to the first embodiment, the directions of the optical axes of the image capture devices 2a to 2d and the line of sight direction from the virtual viewpoint coincide in that they are directed outward from the vehicle 1000 (vehicle model 200) in a planar view.

[0034] For comparison with the first embodiment, consider a case in which the line of sight from the virtual viewpoint is directed toward the vehicle model from outside the vehicle model in a planar view. This case is referred to as a comparative case. According to the comparative case, the line of sight from the virtual viewpoint differs from the direction of the optical axis of the imaging device in terms of whether or not it is directed toward the vehicle (vehicle model) in a planar view. Therefore, if there is a three-dimensional object around the vehicle, the three-dimensional object will appear to have fallen to the ground in the peripheral image.

[0035] In contrast, according to the first embodiment, the direction of the optical axes of the image capture devices 2a to 2d and the line of sight from the virtual viewpoint coincide in that they are directed outward from the vehicle 1000 (vehicle model 200) in a plan view. Therefore, when there is a three-dimensional object around the vehicle 1000, the CPU 10 can generate a peripheral image in which the three-dimensional object appears to be standing upright from the ground. Therefore, according to the first embodiment, unlike the comparative case, it is possible to generate a peripheral image that allows the occupant to naturally recognize the three-dimensional object.

[0036] 3, in this specification, the angle formed between the positive Y-axis direction and the line of sight in a plan view is referred to as the rotation angle. The angle formed between the negative Z-axis direction and the line of sight is referred to as the depression angle. These definitions of the rotation angle and depression angle are merely examples.

[0037] The trigger for rotating the line of sight direction is not limited to a specific event.

[0038] In one example, the CPU 10 may generate peripheral images one after another while continuously changing the rotation angle in the positive or negative direction at a constant angular velocity, and output each generated peripheral image to the display device 5 one after another.

[0039] In another example, the CPU 10 may change the line of sight based on an input from an occupant via an input device, and generate and output a peripheral image each time the line of sight direction is changed.

[0040] The vehicle model 200 has a height corresponding to the vehicle 1000. Therefore, there is a risk that part of the area of ​​the viewpoint-converted image showing the surrounding environment will be hidden by the vehicle model image. The part of the area showing the surrounding environment that is hidden by the vehicle model image will be referred to as road surface vignetting. The area of ​​the road surface vignetting will be referred to as the road surface vignetting area.

[0041] If road surface vignetting occurs, the occupant will be unable to check the condition of the road surface in the immediate vicinity of the vehicle 1000. Therefore, in order to suppress the area of ​​road surface vignetting, the CPU 10 limits the height of the vehicle model 200 when generating a vehicle model image.

[0042] 4 is a diagram illustrating the operation of the image generating device 1 of the first embodiment to limit the height of the vehicle model 200. As shown in the diagram, the CPU 10 generates the height-limited vehicle model 200 by uniformly compressing the dimension of the original vehicle model 200 in the Z-axis direction.

[0043] The vehicle model 200 before the height restriction is referred to as a first vehicle model 201. The vehicle model 200 after the height restriction is referred to as a second vehicle model 202.

[0044] In order for the occupant to more easily grasp the surrounding environment, it is desirable that a part of the vehicle 1000 (more precisely, the vehicle model 200) is reflected in the surrounding image. For example, if an obstacle (a person or an object) exists around the vehicle 1000, if a part of the vehicle 1000 is reflected in the surrounding image in addition to the obstacle, the occupant can easily grasp the positional relationship between the obstacle and the vehicle 1000.

[0045] However, if the area of ​​the image of the vehicle model 200, i.e., the vehicle model image, is too large, the area of ​​the region showing the surrounding environment becomes relatively small, and the amount of information about the surrounding environment that the occupant can obtain from the surrounding image decreases. Therefore, it is desirable to have an appropriate balance between the area of ​​the vehicle model image and the area showing the surrounding environment. This balance is referred to as display balance.

[0046] A typical vehicle has an elongated shape that extends in the front-to-rear direction (the Y-axis direction in the example shown in FIG. 1). Therefore, when the depression angle is fixed and the rotation angle is changed, the display balance changes according to the rotation angle. Specifically, when the rotation angle is 90 degrees or 270 degrees, which correspond to the right or left direction of the vehicle 1000, the area of ​​the region where the image of the vehicle model 200 is displayed is smaller than when the rotation angle is 0 degrees or 180 degrees, which correspond to the forward or backward direction of the vehicle 1000.

[0047] Therefore, in the first embodiment, the CPU 10 changes the depression angle in accordance with the rotation angle of the line of sight direction so that the above balance is maintained at as appropriate a level as possible.

[0048] 5 is a diagram illustrating an example of a method for setting the depression angle in the first embodiment. In this diagram, the horizontal axis represents the rotation angle, and the vertical axis represents the depression angle. Note that in this specification, the rotation angle and depression angle values ​​are expressed in degrees.

[0049] 5, when the rotation angle is 0 degrees or 180 degrees, which correspond to the forward or backward direction of the vehicle 1000, the depression angle is set to the maximum value of 30 degrees. When the rotation angle is 90 degrees or 270 degrees, which correspond to the right or left direction of the vehicle 1000, the depression angle is set to the minimum value of 20 degrees.

[0050] In this way, the depression angle is reduced when the rotation angle is 90 degrees or 270 degrees, which corresponds to the right or left direction of vehicle 1000, thereby suppressing a decrease in the area of ​​the region where the image of vehicle model 200 is displayed. As a result, changes in the balance between the area of ​​the region where the image of vehicle model 200 is displayed and the area of ​​the region showing the surrounding environment are suppressed.

[0051] In the example shown in Figure 5, the depression angle is changed smoothly in a sinusoidal manner with respect to the rotation angle. The change in depression angle with respect to the rotation angle is not limited to a sinusoidal manner. The depression angle may be changed smoothly with respect to the rotation angle, or may be changed in a step-like manner with respect to the rotation angle. When the depression angle is changed smoothly with respect to the rotation angle, the visibility for the occupant is improved compared to when the depression angle is changed in a step-like manner with respect to the rotation angle.

[0052] FIG. 6 is a diagram showing an example of a peripheral image displayed on the display device 5. As shown in FIG.

[0053] Peripheral image D101 is a peripheral image when the rotation angle is 0 degrees. Peripheral image D102 is a peripheral image when the rotation angle is 45 degrees. Peripheral image D103 is a peripheral image when the rotation angle is 90 degrees.

[0054] 5, when the rotation angle is 0 degrees, the set value of the depression angle is 30 degrees, when the rotation angle is 45 degrees, the set value of the depression angle is 25 degrees, and when the rotation angle is 90 degrees, the set value of the depression angle is 20 degrees. Therefore, peripheral image D101 is generated at a depression angle of 30 degrees, peripheral image D102 is generated at a depression angle of 25 degrees, and peripheral image D103 is generated at a depression angle of 20 degrees.

[0055] It can be seen from FIG. 6 that the display balance is maintained at an appropriate level in each of the peripheral image D101, the peripheral image D102, and the peripheral image D103.

[0056] Therefore, the occupant can easily grasp the positional relationship between the vehicle 1000 and the surrounding environment reflected in the surrounding image, regardless of the rotation angle.

[0057] Each imaging device 2 has a blind spot on the vehicle 1000 side, i.e., an area that cannot be imaged. Therefore, in the peripheral image, an image obtained from the captured image cannot be captured in the substantially rectangular area where the vehicle 1000 is located. The non-image capture area 400 shown in Fig. 6 is an area where an image obtained from the captured image cannot be captured.

[0058] The road surface vignetting described above refers to the portion of the vehicle model image 300 that extends beyond the non-captured area 400. In the first embodiment, the vehicle model image 300 is generated based on the vehicle model 200 after the height restriction, i.e., the second vehicle model 202. Therefore, the area of ​​the portion of the vehicle model image 300 that extends beyond the non-captured area 400, i.e., the road surface vignetting area, can be reduced.

[0059] Next, the operation of the image generating device 1 of the first embodiment will be described.

[0060] 7 is a flowchart showing the operation of the image generating device 1 of the first embodiment to output a peripheral image to the display device 5. Here, as an example, the operation of outputting one frame of a peripheral image will be described. In other words, the series of operations shown in this figure is repeatedly executed for each frame.

[0061] First, the CPU 10 acquires captured images from the four imaging devices 2a to 2d via the I / O interface 13 (S101). The CPU 10 generates a second vehicle model 202, which is a vehicle model 200 with a restricted height (S102).

[0062] The CPU 10 executes an image generation process to generate a peripheral image (S103). Then, the CPU 10 outputs the generated peripheral image to the display device 5 via the I / O interface 13 (S104). Then, the operation for one frame is completed.

[0063] FIG. 8 is a flowchart showing the image generation process of the first embodiment.

[0064] The CPU 10 sets a rotation angle for generating a peripheral image (S201).

[0065] The method for setting the rotation angle is not limited to a specific method. When the rotation angle is changed continuously at a constant angular velocity, the CPU 10 obtains the rotation angle for generating the current peripheral image by adding a predetermined step size value to the rotation angle used when generating the peripheral image of the previous frame. When a rotation angle is specified by an instruction from the occupant, the CPU 10 sets the specified rotation angle as the rotation angle for generating the peripheral image.

[0066] Following S201, the CPU 10 sets a depression angle corresponding to the rotation angle set by the processing of S201 (S202). The CPU 10 acquires the depression angle corresponding to the rotation angle based on the correspondence shown in Fig. 5, for example, and sets it as the depression angle for generating a peripheral image.

[0067] CPU 10 generates a peripheral image (S203) using the four captured images acquired from four imaging devices 2a to 2d in the process of S101, second vehicle model 202 generated in the process of S102, and the line of sight direction determined by the rotation angle and depression angle acquired in the processes of S201 and S202. Then, the image generation process ends.

[0068] As described above, according to the first embodiment, the CPU 10 acquires, via the I / O interface 13, images of the surrounding environment of the vehicle 1000 captured by the imaging devices 2a to 2d provided on the vehicle 1000. Then, based on the images acquired from the imaging devices 2a to 2d, the CPU 10 generates a plurality of surrounding images viewed from a virtual viewpoint provided above the vehicle 1000 in a plurality of line-of-sight directions, each of which has a horizontal component facing in a different direction around the vehicle 1000. Then, the CPU 10 switches between the plurality of surrounding images and outputs them to the display device 5 via the I / O interface 13.

[0069] The direction of the optical axes of the imaging devices 2a to 2d and the line of sight from the virtual viewpoint coincide in that they are directed outward from the vehicle 1000 (vehicle model 200) in a plan view. Therefore, the CPU 10 can generate a surrounding image that allows the occupant to naturally grasp three-dimensional objects around the vehicle 1000. In other words, the image generation device 1 can output an image that makes it easy to grasp the surrounding environment of the vehicle 1000.

[0070] Furthermore, according to the first embodiment, the CPU 10 sets the depression angle so that part of the vehicle model 200 appears in each peripheral image.

[0071] Therefore, when there is an obstacle around the vehicle 1000, the occupant can easily grasp the positional relationship between the vehicle 1000 and the obstacle.

[0072] Furthermore, the CPU 10 sets the depression angle so that it varies sinusoidally with respect to the rotation angle.

[0073] The depression angle changes smoothly with the rotation angle, improving visibility for the occupant.

[0074] Furthermore, the CPU 10 limits the height of the vehicle model 200 when generating the vehicle model image.

[0075] Therefore, it is possible to output a surrounding image that allows the occupant to check the condition of the road surface in the immediate vicinity of the vehicle 1000.

[0076] (Second embodiment) In the second embodiment, the CPU 10 sets the depression angle so that the display balance, that is, the balance between the area of ​​the vehicle model image and the area of ​​the region showing the surrounding environment, is more precisely constant. Specifically, a target range for the area of ​​the vehicle model image in the surrounding image is set in advance, and the CPU 10 sets the depression angle so that the area of ​​the vehicle model image in the surrounding image falls within the target range.

[0077] In the following description of the second embodiment, differences from the first embodiment will be described, and descriptions of the same matters as those in the first embodiment will be omitted or will be given in a simplified manner.

[0078] FIG. 9 is a flowchart showing the image generation process of the second embodiment.

[0079] First, the CPU 10 sets a rotation angle for generating a peripheral image (S301). In S301, the CPU 10 sets the rotation angle in the same manner as in S201.

[0080] Next, CPU 10 temporarily sets the depression angle (S302). In S302, CPU 10 may set the depression angle used when generating the peripheral image of the immediately previous frame as the temporarily set value. Alternatively, if an initial value of the depression angle is set in advance, CPU 10 may set the initial value of the depression angle as the temporarily set value of the depression angle.

[0081] Next, CPU 10 calculates the area of ​​the vehicle model image that occupies the peripheral image when the peripheral image is generated using the line of sight direction determined by the provisional settings of the rotation angle and depression angle (S303). Then, CPU 10 determines whether the area of ​​the vehicle model image obtained by the calculation is within a preset target range (S304). The target range of the area of ​​the vehicle model image may be set in advance by a designer or may be set by a passenger.

[0082] If the area of ​​the vehicle model image is not within the target range (S304: No), the CPU 10 determines whether the area of ​​the vehicle model image is less than the lower limit of the target range (S305).

[0083] If the area of ​​the vehicle model image is less than the lower limit of the target range (S305: Yes), the CPU 10 reduces the provisional setting value of the depression angle by, for example, a predetermined amount (S306).

[0084] If the area of ​​the vehicle model image is not less than the lower limit of the target range (S305: No), that is, if the area of ​​the vehicle model image exceeds the upper limit of the target range, the CPU 10 increases the provisional setting value of the depression angle by, for example, a predetermined amount (S307).

[0085] After S306 or S307, the control shifts to S303, and the CPU 10 recalculates the area of ​​the vehicle model image using the provisionally set value of the depression angle after adjustment.

[0086] If it is determined in the determination process of S304 that the area of ​​the vehicle model image is within the target range (S304: Yes), the CPU 10 sets the provisional setting value of the depression angle as the depression angle used to generate the peripheral image (S308).

[0087] CPU 10 generates a peripheral image (S309) using the four captured images acquired from four imaging devices 2a to 2d in the process of S101, second vehicle model 202 generated in the process of S102, and the line of sight determined by the rotation angle set in the process of S301 and the depression angle determined in the process of S308. Then, the image generation process according to the second embodiment ends.

[0088] In the above description, CPU 10 adjusted the depression angle based on the area of ​​vehicle model 200 shown in the peripheral image each time image generation processing was performed. A correspondence relationship between the rotation angle and depression angle that would bring the area of ​​the vehicle model image within a target range may be stored in advance in a predetermined storage area of ​​image generation device 1. Then, in the image generation processing, CPU 10 may set the depression angle used to generate the peripheral image based on this correspondence relationship. CPU 10 may acquire this correspondence relationship in advance by executing the processes of S302 to S308 for each different setting value of the rotation angle.

[0089] As described above, in the second embodiment, the CPU 10 sets the depression angle based on the area of ​​the vehicle model 200 reflected in the peripheral image.

[0090] Therefore, the display balance is controlled to a constant value regardless of the rotation angle, so that the amount of information about the surrounding environment that the occupant obtains from the surrounding image can be kept constant regardless of the rotation angle.

[0091] (Third embodiment) In the third embodiment, the CPU 10 sets the depression angle so that at least a part of the dimensions of the area in which the vehicle model image is displayed in the peripheral image is constant, thereby making the display balance as constant as possible. Here, as an example, the CPU 10 sets the depression angle so that the vertical length of the area in which the vehicle model image is displayed in the peripheral image is constant.

[0092] In the following description of the third embodiment, differences from the second embodiment will be described, and descriptions of the same matters as those in the second embodiment will be omitted or will be given in a simplified manner.

[0093] FIG. 10 is a flowchart showing the image generation process of the third embodiment.

[0094] As in S301 and S302, the CPU 10 sets a rotation angle for generating a peripheral image (S401) and temporarily sets a depression angle (S402).

[0095] Next, the CPU 10 calculates the display destination coordinates of the upper end point of the vehicle model image in the peripheral image when the peripheral image is generated using the line of sight direction determined by the provisional settings of the rotation angle and depression angle (S403).Then, the CPU 10 determines whether the calculated upper end point coordinates are within the target range (S404).

[0096] In the third embodiment, a range of coordinates of the upper end points of the vehicle model image that can achieve an appropriate level of display balance is set in advance as a target range. The target range of coordinates of the upper end points may be set by the designer or may be set by the occupant.

[0097] If the upper end point coordinate is not within the target range (S404: No), the CPU 10 determines whether the upper end point coordinate is less than the lower limit of the target range (S405).

[0098] If the upper end point coordinate is less than the lower limit of the target range (S405: Yes), the CPU 10 reduces the provisional setting value of the depression angle by, for example, a predetermined amount (S406).

[0099] If the upper end point coordinate is not less than the lower limit of the target range (S405: No), that is, if the upper end point coordinate exceeds the upper limit of the target range, the CPU 10 increases the provisional depression angle setting by, for example, a predetermined amount (S407).

[0100] After S406 or S407, the control shifts to S403, and the CPU 10 recalculates the display destination coordinates of the upper end point of the vehicle model image using the provisionally set value of the depression angle after adjustment.

[0101] If it is determined in the determination process of S404 that the vehicle-side end point coordinates are within the target range (S404: Yes), the CPU 10 determines the provisional setting value of the depression angle as the depression angle to be used for generating the peripheral image (S408).

[0102] CPU 10 generates a peripheral image (S409) using the four captured images acquired from four imaging devices 2a to 2d in the process of S101, second vehicle model 202 generated in the process of S102, and the line of sight determined by the rotation angle set in the process of S401 and the depression angle determined in the process of S408. Then, the image generation process according to the third embodiment ends.

[0103] In the above description, CPU 10 adjusted the depression angle based on the dimensions of vehicle model 200 shown in the peripheral image. A correspondence relationship between the rotation angle and depression angle that causes the dimensions of vehicle model 200 shown in the peripheral image to fall within a target range may be stored in advance in a predetermined storage area of ​​image generation device 1. Then, in the image generation process, CPU 10 may obtain the depression angle to be used in generating the peripheral image based on this correspondence relationship. CPU 10 may obtain this correspondence relationship in advance by executing the processes of S402 to S408 for each different setting value of the rotation angle.

[0104] As described above, according to the third embodiment, the CPU 10 sets the depression angle based on the dimensions of the vehicle model 200 shown in the peripheral image.

[0105] Therefore, as in the second embodiment, the display balance is controlled to a constant value regardless of the rotation angle, so that the amount of information about the surrounding environment that the occupant obtains from the surrounding image can be kept constant regardless of the rotation angle.

[0106] (Fourth embodiment) In the first to third embodiments, the CPU 10 sets the depression angle so that part of the vehicle model 200 is reflected in each peripheral image. The method for reflecting part of the vehicle model 200 in each peripheral image is not limited to this.

[0107] In the fourth embodiment, an example in which the position of the virtual viewpoint is moved will be described as a variation of the method for making a part of the vehicle model 200 appear in each peripheral image.

[0108] 11 is a diagram for explaining the operation of moving the virtual world executed by the image generating device 1 of the fourth embodiment. As shown in this diagram, the CPU 10 moves the position of the virtual viewpoint instead of changing the depression angle.

[0109] The CPU 10 may move the virtual viewpoint above the vehicle 1000 (more precisely, the vehicle model 200) laterally relative to the vehicle 1000. Positions L1 and L2 are examples of destination positions when the virtual viewpoint is moved laterally relative to the vehicle 1000.

[0110] Alternatively, CPU 10 may move the virtual viewpoint above vehicle 1000 in the vertical direction (in other words, the X-axis direction) relative to vehicle 1000. Position L3 is an example of a destination position when the virtual viewpoint is moved in the vertical direction relative to vehicle 1000.

[0111] A correspondence relationship between the rotation angle and the position of the virtual viewpoint may be set in advance, and the CPU 10 may move the virtual viewpoint based on this correspondence relationship.

[0112] Alternatively, the CPU 10 may move the virtual viewpoint based on the area of ​​the vehicle model 200 that appears in the peripheral image.

[0113] Alternatively, the CPU 10 may move the virtual viewpoint based on the dimensions of the vehicle model 200 shown in the peripheral image.

[0114] 12 is a diagram showing an example of a peripheral image obtained by moving the virtual viewpoint, showing peripheral images before and after the movement of the virtual viewpoint when control is executed to move the virtual viewpoint based on the area of ​​vehicle model 200 shown in the peripheral image.

[0115] The peripheral image D201 is a peripheral image generated based on the settings before the virtual viewpoint is moved. Specifically, the peripheral image D201 is a peripheral image generated when the rotation angle is set to 90 degrees, the depression angle is set to 30 degrees, and the virtual viewpoint is set to L0. In this peripheral image D201, the area in which the vehicle model image 300 is displayed is significantly small, making it difficult for the occupant to grasp the positional relationship between the vehicle 1000 and the surrounding environment.

[0116] The peripheral image D211 is a peripheral image generated based on the settings after the virtual viewpoint is moved. Specifically, the peripheral image D111 is a peripheral image generated when the rotation angle is set to 90 degrees, the depression angle is set to 30 degrees, and the virtual viewpoint is set to L2. According to this peripheral image D111, the area in which the vehicle model image 300 is displayed is larger than that of the peripheral image D201, and it can be seen that the display balance has been greatly improved.

[0117] In this way, the CPU 10 may set the viewpoint position according to the line of sight so that part of the vehicle model 200 appears in each peripheral image.

[0118] (Fifth embodiment) In the fifth embodiment, variations of the operation when generating a vehicle model image will be described. The operation of the fifth embodiment can be used in combination with the operations of the first to fourth embodiments.

[0119] FIG. 13 is a flowchart showing the operation of generating a vehicle model image, which is executed by the image generating device 1 of the fifth embodiment.

[0120] First, the CPU 10 calculates color values ​​of the first vehicle model 201 by performing shading processing based on the parameters of the first vehicle model 201 (S501). The shading processing is processing that performs shading taking into account the amount of light hitting the vehicle. The parameters of the first vehicle model 201 include the coordinates of a large number of polygon vertices that define the shape of the first vehicle model 201. That is, in S501, the CPU 10 calculates color values ​​after shading based on the shape of the vehicle model 200 before the height restriction was imposed.

[0121] The CPU 10 calculates the display position coordinates in the surrounding image based on the parameters of the second vehicle model 202 (S502).

[0122] The CPU 10 applies the color value obtained by the calculation in S501 to the display position coordinates of the second vehicle model 202 (S503), and the operation of generating the vehicle model image is completed.

[0123] In this way, the CPU 10 applies the post-shading color values ​​to the vehicle model 200 after the height restriction based on the shape of the vehicle model 200 before the height restriction.

[0124] Therefore, it is possible to prevent the loss of three-dimensional effect due to height restrictions. Note that the color values ​​after shading may be calculated in advance based on the shape of the vehicle model 200 before the height restriction, and may be applied to the vehicle model 200 after the height restriction.

[0125] (Sixth embodiment) The sixth embodiment describes a variation of the operation for limiting the height of the vehicle model 200. The operation of the sixth embodiment is applicable to any of the first to fifth embodiments.

[0126] FIG. 14 is a flowchart showing the operation of limiting the height of the vehicle model 200, which is executed by the image generating device 1 of the sixth embodiment.

[0127] First, the CPU 10 sets a rotation angle in the image generation process (S601), and then provisionally sets the height of the vehicle model 200 (S602). In S602, the CPU 10 may use the height used when generating the peripheral image of the immediately previous frame as the provisional setting value. Alternatively, if an initial value for the height has been set in advance, the CPU 10 may use the initial value as the provisional setting value for the height.

[0128] Next, the CPU 10 calculates the road surface vignetting area when the peripheral image is generated using the provisionally set values ​​of the rotation angle and height (S603). Then, the CPU 10 determines whether the road surface vignetting area is equal to or smaller than a threshold value (S604). The threshold value is set in advance. The threshold value may be set by a designer or may be set by a passenger.

[0129] If the road surface vignetting area is not equal to or less than the threshold value (S604: No), the provisional height setting value is lowered, for example, by a predetermined amount (S605). Then, the control transitions to S603, and the CPU 10 recalculates the area of ​​the vehicle model image using the provisional height setting value after adjustment.

[0130] If the road surface vignetting area is equal to or smaller than the threshold value (S604: Yes), CPU 10 determines the provisional setting value as the height to be used for height restriction (S606). CPU 10 then generates second vehicle model 202 using the determined height (S607), and the operation of restricting the height of vehicle model 200 ends.

[0131] In the above description, CPU 10 adjusted the height of vehicle model 200 based on the road surface vignetting area. A correspondence relationship between the rotation angle and height such that the road surface vignetting area is equal to or smaller than a threshold value may be stored in advance in a predetermined storage area of ​​image generation device 1. Then, in the image generation process, CPU 10 may obtain the height to be used for the height restriction based on this correspondence relationship. CPU 10 may obtain this correspondence relationship in advance by executing the processes of S602 to S606 for each different set value of the rotation angle.

[0132] In this way, the CPU 10 may set the height of the vehicle model 200 based on the road surface vignetting area.

[0133] (Seventh embodiment) The seventh embodiment will describe another variation of the operation for limiting the height of the vehicle model 200. The operation of the seventh embodiment can be applied to any of the first to fifth embodiments.

[0134] 15 is a diagram illustrating a method for restricting the height of vehicle model 200 executed by image generating device 1 of the seventh embodiment. CPU 10 compresses the body above the wheels of vehicle model 200 to generate second vehicle model 203, which is vehicle model 200 after height restriction. In the example shown in this diagram, CPU 10 compresses the portion of the body above a certain position H1 that is higher than the wheels.

[0135] Image generation device 1 may change and display the steering angle of the wheels included in vehicle model 200 according to the actual steering angle so that the occupant can check the state of wheels 3. Alternatively, image generation device 1 may display the body of vehicle model 200 semi-transparently or by displaying only the outline, so that the wheels hidden by the body can be seen through the body. When such a configuration is adopted, if all the bodies including the wheels are uniformly compressed, the shapes of the wheels displayed in the peripheral image when the actual steering angle is not 0 degrees may become oval, which may cause the occupant to feel uncomfortable.

[0136] In the seventh embodiment, the CPU 10 compresses the body above the wheels but does not compress the wheels, thereby preventing the wheels from appearing oval in the peripheral image.

[0137] (Eighth embodiment) The eighth embodiment will describe yet another variation of the operation for limiting the height of the vehicle model 200. The operation of the eighth embodiment can be applied to any of the first to fifth embodiments.

[0138] FIG. 16 is a diagram for explaining a method for limiting the height of the vehicle model 200, which is executed by the image generating device 1 of the eighth embodiment.

[0139] The CPU 10 moves the polygon vertices of the vehicle model 200 along a straight line extending from the virtual viewpoint. In the example shown in FIG. 16, the line of sight is shown as an example of a straight line extending from the virtual viewpoint. Then, vertex F1, which is a polygon vertex of the vehicle model 200 on the line in the line of sight, is moved to position F2. The CPU 10 uniformly moves all polygon vertices of the vehicle model 200 using the above method, thereby generating the vehicle model 200 after the height restriction, i.e., the second vehicle model 202.

[0140] In the first embodiment, the CPU 10 limited the height of the vehicle model 200 by uniformly compressing the entire vehicle model 200. If the height is limited in this way, the appearance of the vehicle model 200 as seen from the virtual viewpoint changes before and after the height limit, which may cause a feeling of discomfort to the occupants.

[0141] In the eighth embodiment, the polygon vertices of the vehicle model 200 are moved along a straight line extending from the virtual viewpoint, thereby preventing the appearance of the vehicle model 200 as seen from the virtual viewpoint from changing before and after the height limit.

[0142] (Ninth embodiment) The ninth embodiment will describe yet another variation of the operation for limiting the height of the vehicle model 200. The operation of the ninth embodiment can be applied to any of the first to fifth embodiments.

[0143] FIG. 17 is a diagram for explaining a method for limiting the height of the vehicle model 200, which is executed by the image generating device 1 of the ninth embodiment.

[0144] The CPU 10 moves the polygon vertices of the vehicle model 200 along the body ridgeline of the vehicle model 200. For example, the CPU 10 moves a certain vertex F3 along the body ridgeline to a position F4. The CPU 10 uniformly moves all of the polygon vertices of the vehicle model 200 using the above method, thereby generating the vehicle model 200 after the height restriction, i.e., the second vehicle model 202.

[0145] When height restrictions are imposed by moving the polygon vertices of the vehicle model 200 along the body ridgelines, the appearance of the vehicle model 200 as seen from the virtual viewpoint can be made to remain unchanged before and after the height restrictions, as in the eighth embodiment.

[0146] An image generation program 100, which is a computer program executed by the image generation device 1 of the first to eighth embodiments, is stored in advance in the ROM 12 of the image generation device 1. The image generation program 100 may be provided by being recorded in the form of an installable or executable file on a computer-readable recording medium such as a CD (Compact Disc)-ROM (Read Only Memory), a flexible disk (FD), a CD-R (Recordable), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, or an SD (Secure Digital) card.

[0147] Furthermore, the image generating program 100 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0148] A part or all of the processing executed by the CPU 10 in accordance with the image generating program 100 may be realized by a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or the like.

[0149] Although several embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0150] 1. Image generation device 2,2a~2d Imaging device 3 wheels 4 Input Devices 5 Display device 10 CPU 11 RAM 12 ROM 13 I / O Interface 14 Bus 100 Image Generation Program 200 vehicle models 201 First Vehicle Model 202 Second Vehicle Model 300 vehicle model images 1000 vehicles

Claims

1. a processor that acquires a first image of the surrounding environment of the vehicle captured by an imaging device provided in the vehicle, generates a plurality of second images based on the first image, the second images being viewed from a virtual viewpoint provided above the vehicle in a plurality of line-of-sight directions with horizontal components facing in different directions around the vehicle, and outputs the plurality of second images to a display device by switching between them; the processor sets depression angles of the plurality of line-of-sight directions so that a part of a vehicle model representing the vehicle is reflected in the plurality of second images; The processor further sets depression angles of the plurality of line-of-sight directions based on areas of regions in which the vehicle model is reflected in the plurality of second images. Image generating device.

2. the processor further sets depression angles of the plurality of gaze directions so as to smoothly change with respect to directions of horizontal components of the plurality of gaze directions. The image generating device of claim 1 .

3. The processor further sets depression angles of the plurality of gaze directions based on dimensions of areas in which the vehicle model is reflected in the plurality of second images. The image generating device of claim 1 .

4. the processor sets the position of the virtual viewpoint for each of the plurality of line-of-sight directions so that a part of a vehicle model representing the vehicle is reflected in the plurality of second images. The image generating device of claim 1 .

5. The processor further comprises: generating a vehicle model image based on a three-dimensional vehicle model corresponding to the vehicle; superimposing and displaying the vehicle model image on each of the plurality of second images; limiting the height of the vehicle model when generating the vehicle model image; The image generating device of claim 1 .

6. The processor further calculates color values ​​of the vehicle model by performing a shading process on the vehicle model before the height restriction, and applies the color values ​​to the vehicle model after the height restriction. The image generating device according to claim 5 .

7. the processor further restricts the height of the vehicle model by compressing a body above the wheels relative to the vehicle model. The image generating device according to claim 5 .

8. the processor further limits a height of the vehicle model by moving polygon vertices of the vehicle model along a straight line extending from the virtual viewpoint. The image generating device according to claim 5 .

9. acquiring a first image of a surrounding environment of the vehicle captured by an imaging device provided in the vehicle; generating, based on the first image, a plurality of second images viewed from a virtual viewpoint provided above the vehicle in a plurality of line-of-sight directions, each of which has a horizontal component facing in a different direction around the vehicle; and outputting the plurality of second images to a display device by switching between the plurality of second images, setting depression angles of the plurality of line-of-sight directions so that a part of a vehicle model representing the vehicle is reflected in the plurality of second images; setting depression angles of the plurality of line-of-sight directions based on areas of regions in which the vehicle model is reflected in the plurality of second images; The image generating method further comprises:

10. setting depression angles of the plurality of line-of-sight directions so as to smoothly change with respect to the directions of horizontal components of the plurality of line-of-sight directions; The image generating method of claim 9 further comprising:

11. setting depression angles of the plurality of line-of-sight directions based on dimensions of areas in which the vehicle model is shown in the plurality of second images; The image generating method of claim 9 further comprising:

12. setting the position of the virtual viewpoint for each of the plurality of line-of-sight directions so that a part of a vehicle model representing the vehicle is reflected in the plurality of second images; The image generating method of claim 9 further comprising:

13. generating a vehicle model image based on a three-dimensional vehicle model corresponding to the vehicle; superimposing and displaying the vehicle model image on each of the plurality of second images; limiting the height of the vehicle model when generating the vehicle model image; The image generating method of claim 9 further comprising:

14. calculating color values ​​of the vehicle model by performing a shading process on the vehicle model before height restriction, and applying the color values ​​to the vehicle model after height restriction; The image generating method of claim 13 further comprising:

15. limiting the height of the vehicle model by compressing a body above wheels of the vehicle model; The image generating method according to claim 13, wherein

16. limiting the height of the vehicle model by moving polygon vertices of the vehicle model along a straight line extending from the virtual viewpoint; The image generating method according to claim 13, wherein

Images