Image generation apparatus and image generation method

Abstract

To enable a user to more appropriately understand that an obstacle is located in an area not photographed by a side camera and a rear camera.SOLUTION: An image generation device 20 generates a composite image obtained by compositing a rear image obtained by photographing the rear of a vehicle with a rear side image obtained by photographing the rear side of the vehicle, and includes an obstacle detection part 23, a position determination part 24, and an image processing part 25. The image processing unit 25 superimposes an image indicating the presence of the obstacle on the synthesized image when the obstacle is located outside the area where the side camera 11,12 photographs the body of the vehicle and the area defined by the angle of view of the rear camera 13 in the area defined by the angle of view of the side camera 11,12.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image generation device and an image generation method. [Background technology]

[0002] Image generation devices are known that generate composite images by combining rearward images of the rear side of the vehicle captured by a rear camera with rearward images of the rear of the vehicle captured by a side camera. Patent Document 1 discloses a technology for combining a rearward image of a blind spot captured by a side camera with a rearward image when a blind spot that cannot be captured by the rear camera occurs due to the approach of a following vehicle behind the vehicle. Patent Document 2 discloses an image generation device that includes a rear camera and a side camera that capture images of the rear of the vehicle, and a display unit that displays a composite image that combines images captured by each camera and a blind spot warning image that warns of the blind spot in the composite image.

[0003] Patent Document 3 discloses a technology that includes a side camera and two rear cameras with different angles of view, synthesizes a rearward image taken by a wide-angle rear camera or a narrow-angle rear camera with a rearward image taken by the side camera, and complements the blind spot area of ​​the rearward image taken by the narrow-angle rear camera with the rearward image taken by the wide-angle rear camera.

[0004] Patent Document 4 discloses a technology for detecting obstacles in which, when an obstacle is detected around the vehicle, a substitute image corresponding to the obstacle is obtained from a storage unit, and the orientation and tilt of the substitute image are changed based on a virtual viewpoint and superimposed on an overhead image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5347257 specification [Patent Document 2] Japanese Patent Application Publication No. 2019-034692 [Patent Document 3] Patent No. 7280006 specification [Patent Document 4] Patent No. 5165631 specification Summary of the Invention [Problem to be solved by the invention]

[0006] In the inventions described in Patent Documents 1 to 4, it was sometimes difficult for the user to recognize an obstacle located in an area that was not captured by either the side camera or the rear camera. For example, in the technology disclosed in Patent Document 3, the image synthesized by complex image processing that combines images from three different cameras into one image may not be appropriate.

[0007] The present disclosure aims to allow a user to more appropriately understand that an obstacle is located in an area that is not captured by either the side camera or the rear camera. [Means for solving the problem]

[0008] In order to achieve the above object, an image generation device according to the present disclosure generates a composite image by combining a rear image captured behind a vehicle with a rearward image captured behind the vehicle, the image generation device including: an obstacle detection unit that detects the position of an obstacle present around the vehicle; a position determination unit that determines whether the obstacle is located outside a region defined by the angle of view of a side camera that captures the rearward side of the vehicle, the region defined by the angle of view of the side camera that captures the rearward side of the vehicle; and an image processing unit that superimposes an image indicating the presence of the obstacle on the composite image based on the position of the obstacle. When the obstacle is located outside a region defined by the angle of view of the side camera that captures the vehicle body, the region defined by the angle of view of the rear camera, the image processing unit superimposes the image indicating the presence of the obstacle on the composite image. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to make a user more appropriately understand that an obstacle is located in an area that is not captured by either the side camera or the rear camera. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram showing an example of installation of first, second, and third display units in a vehicle equipped with an image generation device according to a first embodiment. [Figure 2] 1 is a block diagram showing the functional configuration of an image generation system including an image generation device according to a first embodiment. [Figure 3] 3A and 3B are diagrams illustrating the angle of view of a side camera, the angle of view of a rear camera, a first mask area, and a second mask area. [Figure 4] 10A and 10B are diagrams illustrating images displayed on a first display unit. [Figure 5] FIG. 2 is a diagram schematically illustrating a map. [Figure 6] 10A and 10B are diagrams illustrating an example of a right composite image displayed on the first display unit. [Figure 7] 10A to 10C are diagrams illustrating various examples of a right composite image displayed on the first display unit. [Figure 8] 5 is a flowchart showing an example of the operation of the image generating device according to the first embodiment. [Figure 9] 10A and 10B are diagrams illustrating an example of a right composite image displayed on the first display unit in the image generation system according to the second embodiment. [Figure 10] 10 is a flowchart showing an example of the operation of the image generating device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] (First embodiment) An image generation system 100 including an image generation device 20 according to the first embodiment will be described below with reference to the drawings. With reference to Fig. 1, the image generation system 100 is mounted on a vehicle 1. With reference to Fig. 2, the image generation system 100 is mainly configured to include an information acquisition device 10, an image generation device 20, a display device 40, and an operation unit 45.

[0012] The information acquisition device 10 is a device that acquires various information about the surroundings of the vehicle 1 and about the vehicle 1. The information acquisition device 10 includes a right side camera 11, a left side camera 12, a rear camera 13, a surrounding information detection sensor 14, a vehicle information detection sensor 15, and the like.

[0013] The right side camera 11 is a camera that photographs the right rear side of the vehicle 1 to acquire a right rear side image. As shown in FIG. 1(a), the right side camera 11 is provided on or near the right side mirror 2 of the vehicle 1, or as shown in FIG. 1(b), it replaces the right side mirror 2 of the vehicle 1. The left side camera 12 is a camera that photographs the left rear side of the vehicle 1 to acquire a left rear side image. As shown in FIG. 1(a), the left side camera 12 is provided on or near the left side mirror 3 of the vehicle 1, or as shown in FIG. 1(b), it replaces the left side mirror 3 of the vehicle 1.

[0014] Hereinafter, the right side camera 11 and the left side camera 12 may be referred to as side cameras 11, 12. The side cameras 11, 12 have imaging lenses, imaging elements such as CCD sensors, etc. The angle of view θ1 of the imaging lens of the side camera 11 shown in FIG. 3 is adjusted to an angle of view that captures a part of the right side of the vehicle body, which is the body of the vehicle 1, and the surroundings of the vehicle 1. The angle of view θ2 of the imaging lens of the side camera 12 shown in FIG. 3 is adjusted to an angle of view that captures a part of the left side of the vehicle body, which is the body of the vehicle 1, and the surroundings of the vehicle 1. Image signals of the right rear side image and the left rear side image captured by the side cameras 11, 12 are output to the image generation device 20, and are displayed by the image generation device 20 on the first display unit 41 or the second display unit 42 of the display device 40, respectively. The body of the vehicle 1 is also referred to as the vehicle body.

[0015] The rear camera 13 is a camera that photographs the area behind the vehicle 1 to obtain a rearward image. As shown in FIG. 3, the rear camera 13 is installed below the rear window 4 at the rear of the vehicle 1 and approximately in the center in the left-right direction, but is not limited to this location. The rear camera 13 includes an imaging lens, an imaging element such as a CCD sensor, etc. The angle of view θ3 of the imaging lens of the rear camera 13 shown in FIG. 3 is adjusted to capture an area that allows the user to check the rearward view through the rearview mirror. The angle of view θ3 is smaller than 180 degrees and narrower than the angle of view of the rearview camera described below. The image captured by the rear camera 13 has less distortion than the image captured by the rearview camera. An image signal of the rearward image captured by the rear camera 13 is output to the image generation device 20, which then displays it on the third display unit 43 of the display device 40.

[0016] The surrounding information detection sensor 14 is installed outside the vehicle 1 and is a device that detects obstacles 5 around the vehicle 1. The surrounding information detection sensor 14 in this embodiment is, for example, an ultrasonic sonar. An ultrasonic sonar is a device that detects obstacles 5 by emitting ultrasonic waves into the surroundings and receiving waves reflected by objects such as obstacles around the vehicle. One or more surrounding information detection sensors 14 are installed in the front and / or rear of the vehicle 1 and output measurement information measuring the obstacle 5 to the image generation device 20. The surrounding information detection sensor 14 calculates the distance from the vehicle 1 to the obstacle 5 based on the elapsed time between emitting the ultrasonic waves and detecting the reflected waves, and outputs measurement information including the calculated distance, the intensity of the reflected waves, the relative position, etc. to the image generation device 20.

[0017] The surrounding information detection sensor 14 may be any sensor capable of detecting obstacles 5 around the vehicle 1, and may be a 3D LiDAR, a camera, a stereo camera, or a front-view camera, rear-view camera, or side-view camera installed on the vehicle 1 to generate an image of the vehicle 1 seen from a virtual viewpoint. Hereinafter, the front-view camera, rear-view camera, and side-view camera will be referred to as view cameras. The view camera has a fisheye lens and captures the surroundings with a wider angle of view than the side cameras 11, 12 and the rear camera 13.

[0018] The vehicle information detection sensor 15 is composed of a plurality of sensors that detect the operation state of the vehicle 1 by the user, the state of the vehicle 1, etc. The vehicle information detection sensor 15 includes a shift lever detection sensor that detects the operation state of the shift lever, an accelerator detection sensor that detects the operation state of the accelerator, a brake detection sensor that detects the operation state of the brake, a steering angle detection sensor that detects the operation state of the steering wheel, a wheel speed detection sensor that detects the speed, etc., and outputs the detection results to the image generation device 20 as vehicle information.

[0019] The display device 40 is a device that displays various information such as images captured by various cameras and warnings under the control of the image generating device 20. The display device 40 includes a first display unit 41, a second display unit 42, and a third display unit 43. The first, second, and third display units 41, 42, and 43 are made up of liquid crystal displays or the like. As shown in FIG. 1, the first display unit 41 is installed on the right side of the front of the vehicle interior, the second display unit 42 is installed on the left side of the front of the vehicle interior, and the third display unit 43 is installed in a position where a rearview mirror would be attached at the top center of the front of the vehicle interior.

[0020] The first display unit 41 displays a right rear side image captured by the right side camera 11 or a right composite image obtained by combining the right rear side image and a rear image. The second display unit 42 displays a left rear side image captured by the left side camera 12 or a left composite image obtained by combining the left rear side image and a rear image. The third display unit 43 displays a rear image captured by the rear camera 13.

[0021] The operation unit 45 is a device that accepts operation inputs from a user such as a driver. The operation unit 45 can be a touch panel display surface provided on any of the first, second, and third display units 41, 42, and 43 of the display device 40. The operation unit 45 can also be provided with operation buttons that are pressed or turned to input user operations. By operating the operation unit 45, the user can select whether to display a rear side image on the first display unit 41 and the second display unit 42, or to display a composite image that combines a rear side image and a rear image.

[0022] The image generating device 20 displays a right rear side image based on image signals output from the side cameras 11 and 12 and the rear camera 13 on the first display unit 41, a left rear side image on the second display unit 42, and a rear image on the third display unit 43. The image generating device 20 combines the right rear side image and the rear image to generate a right composite image in which the vehicle body appears to be see-through, and displays this on the first display unit 41. The image generating device 20 combines the left rear side image and the rear image to generate a left composite image in which the vehicle body appears to be see-through, and displays this on the second display unit 42. Hereinafter, when no distinction is made between left and right, the right rear side image and the left rear side image will be referred to as rear side images, and the right composite image and the left composite image will be referred to as composite images. "Appearing to be see-through of the vehicle body" means replacing the part of the rear side image in which the vehicle body is shown with part or all of the rear image, or superimposing part or all of the rear image on the part of the rear side image in which the vehicle body is shown.

[0023] When an obstacle 5 is located in an area within the field of view of the side cameras 11, 12 that is not captured by either the side cameras 11, 12 or the rear camera 13, the image generation device 20 superimposes an image indicating the presence of the obstacle 5 on the composite image. Parts of the areas 54 and 53 that are not captured by either the side cameras 11, 12 or the rear camera 13 are areas where the side cameras 11, 12 cannot capture anything behind the vehicle body because the vehicle body is present within the field of view θ1 or θ2 around the side cameras 11, 12 and the side cameras 11, 12 capture the vehicle body. The areas where the side cameras 11, 12 capture the vehicle body can be considered areas where visualization is obstructed by the vehicle body. The first mask area 54 and the second mask area 55 are areas within the field of view θ1 or θ2 where visualization is obstructed by the vehicle body, excluding the area defined by the field of view θ3 of the rear camera. For example, within the viewing angles θ1 and θ2 of the side cameras 11 and 12, the areas that are not captured by either the side cameras 11 and 12 or the rear camera 13 include part or all of the first mask area 54, the second mask area 55, and the third capturing area 53.

[0024] The reason for superimposing an image showing the obstacle 5 on the composite image will be explained below with reference to FIGS. 3 and 4. As shown in FIG. 3(a), the right side camera 11 and the left side camera 12 can capture images within the range of field angles θ1 and θ2. The left and right isosceles triangles in a plan view shown in FIG. 3(a), whose vertices are the angles that make up the field angles θ1 and θ2, are the areas defined by the field angles θ1 and θ2. Of the areas defined by the field angles θ1 and θ2, a first capturing area 51 and a second capturing area 52, shown by dashed hatching, are areas in which the right side camera 11 or the left side camera 12 can capture images of the periphery of the vehicle 1 without including the vehicle body. The first capturing area 51 and the second capturing area 52 do not include a first mask area 54 and a second mask area 55, which will be described later. The first mask area 54 and the second mask area 55 are areas where the side cameras 11 and 12 photograph the vehicle body, but objects behind the vehicle 1 are not visualized in the right rear side image or the left rear side image, due to the vehicle body being positioned within a portion of the angles of view θ1 and θ2. The first photographing area 51 is an area photographed at the angle of view θ1 that can photograph the periphery of the vehicle 1 without including the vehicle body, and the second photographing area 52 is an area photographed at the angle of view θ2 that can photograph the periphery of the vehicle 1 without including the vehicle body. The rear camera 13 can photograph the periphery of the vehicle 1 at the angle of view θ3. The third photographing area is an area defined by the angle of view θ3, and is an isosceles triangle in plan view, with the angle constituting the angle of view θ3 as its vertex. The third photographing area 53 is an area indicated by a dotted triangle.

[0025] 3(a) are areas that are not captured by any of the right side camera 11, the left side camera 12, and the rear camera 13. The first mask area 54 and the second mask area 55 are areas within the angles of view of the side cameras 11 and 12 that are blocked by the vehicle body, and are areas outside the angle of view of the rear camera 13. For this reason, if an obstacle 5 exists in the first mask area 54 or the second mask area 55, the obstacle 5 may not be displayed in the right composite image in which the right rear side image and the rear image are composited, or in the left composite image in which the left rear side image and the rear image are composited.

[0026] FIG. 4 is a diagram illustrating an image displayed on the first display unit 41. FIG. 4(a) shows a state in which the right rear side image Gr captured by the right side camera 11 is displayed on the first display unit 41. FIG. 4(b) shows a state in which a right composite image Gm, which is a composite of the right rear side image Gr and the rear image Gb captured by the rear camera 13, is displayed on the first display unit 41. The right composite image Gm, in which the vehicle body appears to be transparent, is displayed on the first display unit 41, but the right composite image Gm does not display an image of the obstacle 5 present in the first mask area 54. A user viewing the right composite image Gm may be under the illusion that the first mask area 54, which is not captured by the camera, does not exist, and may not notice the presence of the obstacle 5. The image generating device 20 of this embodiment superimposes an image indicating the presence of the obstacle 5 present in the first mask area 54 or the second mask area 55 on the composite image.

[0027] The image generating device 20 is composed of an ECU having processing devices such as a CPU and a GPU, and storage devices such as a RAM and a ROM. The image generating device 20 can be configured with one ECU, or can be configured with multiple ECUs to distribute the functions of the control unit 21 and to distribute the data to be stored. The image generating device 20 can also be configured with an FPGA, an ASIC, etc.

[0028] 2, the image generating device 20 functions as a control unit 21 and a storage unit 30. The control unit 21 is mainly composed of a processing device such as a CPU included in an ECU, and controls the overall operation of the image generating device 20 by loading a predetermined program stored in a ROM into a RAM and executing it. The storage unit 30 is mainly composed of a storage device such as a RAM included in the ECU, but may also include an external server or database.

[0029] The memory unit 30 stores a control program for operating the image generation device 20, an application program including an image generation program, etc. The memory unit 30 temporarily or non-temporarily stores thresholds, data, parameters, etc. used by the control unit 21. The memory unit 30 stores an abstract image Gs to be superimposed on a composite image. The memory unit 30 stores a map. The map is a driving history in which the time when the side cameras 11, 12, the rear camera 13, or the surrounding information detection sensor 14 detected an obstacle 5 and the coordinates of the detected obstacle 5 are stored, for example, in a vehicle coordinate system described below. For example, the driving history for a predetermined period, such as while driving or during parking, is recorded in the memory unit 30. The coordinate system used in the map is not limited to the vehicle coordinate system, and a world coordinate system based on latitude and longitude, a relative coordinate system whose origin is the vehicle position at the time when the vehicle 1 starts creating the map, etc., can be used. The memory unit 30 may record right rear side images, left rear side images, and rear images acquired by the side cameras 11, 12 and the rear camera 13, view images acquired by the view camera, etc. The memory unit 30 stores a plurality of abstract images Gs of a size according to the distance from the vehicle 1 to the obstacle 5.

[0030] The control unit 21 functions as an image acquisition unit 22, an obstacle detection unit 23, a position determination unit 24, an image processing unit 25, a vehicle information acquisition unit 26, and a map generation unit 27. The image acquisition unit 22 acquires image signals of a right rear side image, a left rear side image, and a rear image input from the side cameras 11 and 12 and the rear camera 13, and outputs them to the image processing unit 25. The image acquisition unit 22 may record each acquired image signal in the storage unit 30 for a predetermined period of time. The obstacle detection unit 23 acquires measurement information input from the surrounding information detection sensor 14, and acquires position information of an obstacle 5 present around the vehicle 1 based on the acquired measurement information. The position information is coordinate data indicating the distance and direction between the obstacle 5 and the surrounding information detection sensor 14. The obstacle detection unit 23 outputs the position information of the obstacle 5 to the position determination unit 24 and the map generation unit 27.

[0031] Based on the position information of the obstacle 5 acquired by the obstacle detection unit 23, the position determination unit 24 determines whether the obstacle 5 is located outside of an area 53, which is an area in which the side cameras 11, 12 capture images of the body of the vehicle 1 and is defined by the angle of view θ3 of the rear camera 13, within the area defined by the angles of view θ1, θ2 of the side cameras 11, 12. In other words, based on the position information of the obstacle 5 acquired by the obstacle detection unit 23, the position determination unit 24 determines whether the obstacle 5 is located in the first mask area 54 or the second mask area 55. The determination by the position determination unit 24 will be described as follows with reference to FIG. 5, which schematically shows a map M.

[0032] V shown in FIG. 5 is an image simulating the vehicle 1. The map M is a three-dimensional map having a vehicle coordinate system in which the origin O is the center of the central axis of the rear wheel axle, the axis passing through the origin O and the central axis is the x-axis, the axis passing through the origin O and perpendicular to the x-axis and extending in the longitudinal direction of the vehicle 1 is the y-axis, and the axis perpendicular to the x-axis and y-axis and extending in the vertical direction is the z-axis. The position determination unit 24 converts the coordinates of the obstacle 5 acquired by the obstacle detection unit 23 into coordinates in the vehicle coordinate system and virtually places the obstacle 5 on the map M. 5a, 5b, and 5c shown in FIG. 5 are schematic diagrams of three obstacles placed on the map M. Hereinafter, when the obstacles 5a, 5b, and 5c are not to be distinguished from one another, the obstacles 5a, 5b, and 5c may be referred to as obstacles 5.

[0033] The position determination unit 24 determines whether the obstacle 5 is located in the first mask area 54 or the second mask area 55 based on the coordinates of the obstacle 5 in the vehicle coordinate system. The position determination unit 24 determines whether the coordinates of the obstacle 5 are included in the first mask area 54 or the second mask area 55 in the vehicle coordinate system. The installation positions, orientations, and viewing angles θ1, θ2, and θ3 of the side cameras 11 and 12 and the rear camera 13 are determined at the time of design. The position information in real space of the first shooting area 51, the second shooting area 52, the third shooting area 53, the first mask area 54, and the second mask area 55 shown in FIG. 3(a) is also known in advance at the design stage. For this reason, the positions of the side cameras 11 and 12 and the rear camera 13, the first shooting area 51, the second shooting area 52, the third shooting area 53, the first mask area 54, and the second mask area 55 are set on the map M. The areas on the map M in FIG. 5 that correspond to the first photographing area 51, the second photographing area 52, the third photographing area 53, the first mask area 54, and the second mask area 55 are given the same reference numerals.

[0034] 5, the position determination unit 24 determines that the obstacles 5a and 5b are located in the first photographing area 51 and the second photographing area 52, respectively, and determines that the obstacle 5c is located in the first mask area 54. The position determination unit 24 outputs the determination result and position information of the obstacle 5 determined to be located in the first mask area 54 or the second mask area 55 to the image processing unit 25, but the determination results and position information of all the obstacles 5 may be output to the image processing unit 25.

[0035] The image processing unit 25 displays the right rear side image or the right composite image on the first display unit 41, the left rear side image or the left composite image on the second display unit 42, and the rear image on the third display unit 43. The timing at which the image processing unit 25 switches between displaying the rear side image or the composite image on the first and second display units 41, 42 is not limited. The image processing unit 25 may always display the composite image on the first and second display units 41, 42. The image processing unit 25 can normally display the rear side image on the first and second display units 41, 42, and display the composite image when the vehicle information acquisition unit 26 detects a predetermined operation including an operation to park the vehicle 1, or when the user detects an operation to instruct the operation unit 45 to display the composite image.

[0036] The image processing unit 25 superimposes an image indicating the presence of the obstacle 5 on the composite image based on the position information of the obstacle 5. An example of an image indicating the presence of the obstacle 5 is an abstract image Gs that depicts the presence of the obstacle 5. The image processing unit 25 superimposes the abstract image Gs on the composite image based on whether the obstacle 5 is located outside of an area 53 that is defined by the angle of view θ1 and θ2 of the side cameras 11 and 12 and in which the side cameras 11 and 12 capture images of the body of the vehicle 1 and that is also defined by the angle of view θ3 of the rear camera 13, among the areas defined by the angle of view θ1 and θ2 of the side cameras 11 and 12. When the obstacle 5 is located outside of an area 53 that is defined by the angle of view θ1 and θ2 of the side cameras 11 and 12 and in which the side cameras 11 and 12 capture images of the body of the vehicle 1 and that is also defined by the angle of view θ3 of the rear camera 13, the image processing unit 25 superimposes the abstract image Gs on the composite image. The image processing unit 25 does not superimpose the abstract image Gs onto the composite image when the obstacle 5 is located within the area defined by the viewing angles θ1 and θ2 of the side cameras 11 and 12, where the side cameras 11 and 12 photograph the body of the vehicle 1, and outside the area 53 defined by the viewing angle θ3 of the rear camera 13.

[0037] The image processing unit 25 calculates the position where the abstract image Gs is to be displayed on the composite image based on the position information of the obstacle 5, and superimposes the abstract image Gs on the composite image at a size based on the position information of the obstacle 5. The image processing unit 25 determines the size of the abstract image Gs based on the distance from the coordinates of the obstacle 5 in the y-axis direction of the vehicle coordinate system to the origin O or the rear end of the vehicle, and acquires the abstract image Gs based on the size from the storage unit 30. The image processing unit 25 displays the abstract image Gs larger the closer the obstacle 5 is to the vehicle 1 in the y-axis direction of the vehicle coordinate system, and smaller the farther the obstacle 5 is from the vehicle 1 in the y-axis direction of the vehicle coordinate system. The image processing unit 25 can also select a large abstract image Gs1 when the distance from the vehicle 1 is equal to or less than a threshold, and select a small abstract image Gs2 when the distance exceeds the threshold. The image processing unit 25 calculates the horizontal rendering position of Gs based on the horizontal position of the obstacle 5 relative to the vehicle 1 in the x-axis direction of the vehicle coordinate system, and superimposes the abstract image Gs on the composite image based on the calculated rendering position. The image processing unit 25 superimposes the selected abstract image Gs at the calculated drawing position on the composite image, and displays it on the first display unit 41 or the second display unit 42.

[0038] The image processing unit 25 may also not display a portion of the abstract image Gs on the first display unit 41 or the second display unit 42. For example, the portion of the abstract image Gs may be the lower portion of the abstract image Gs. When the distance from the vehicle 1 is equal to or less than a threshold, the image processing unit 25 may not display a portion of the large abstract image Gs1, i.e., the lower portion of the large abstract image Gs1, on the first display unit 41 or the second display unit 42. When the distance exceeds the threshold, the image processing unit 25 may display the entire small abstract image Gs2 on the first display unit 41 or the second display unit 42. The closer the obstacle 5 is to the vehicle 1 in the y-axis direction, the larger the area of ​​the portion of the abstract image Gs that is not displayed on the first display unit 41 or the second display unit 42. Depending on the size of the obstacle, both the upper and lower portions of the large abstract image Gs1 may not be displayed. This allows the user to intuitively understand whether the obstacle 5 is close to or far from the vehicle 1.

[0039] The type, shape, color, etc. of the abstract image Gs are not particularly limited as long as they can inform the user that an obstacle 5 exists at the position where the abstract image Gs is displayed in the composite image. When the abstract images Gs1 and Gs2 cannot be distinguished, they are referred to as the abstract image Gs.

[0040] FIG. 6 illustrates an example of a right composite image displayed on the first display unit 41. FIG. 6 shows an example of a right composite image displayed on the first display unit 41. In FIG. 6, a right composite image Gm obtained by combining a right rear side image Gr and a rear image Gb is displayed on the first display unit 41, and an abstract image Gs is superimposed on the right composite image Gm, which shows a line drawing of a vehicle body. In FIG. 6, a three-dimensional abstract image Gs represented by an exclamation mark is superimposed as the abstract image Gs to warn the user and call their attention. The abstract image Gs is not limited to an exclamation mark.

[0041] Fig. 7 illustrates various examples of the right composite image displayed on the first display unit 41. In Fig. 7, the closer the obstacle 5 is to the vehicle 1, the larger the abstract image Gs displayed in the right composite image Gm, and the farther the obstacle 5 is, the smaller the abstract image Gs displayed. The horizontal drawing position of the abstract image Gs is calculated according to the horizontal position of the obstacle 5 relative to the vehicle 1, and the abstract image Gs is superimposed on the right composite image Gm according to the calculated drawing position.

[0042] The vehicle information acquisition unit 26 acquires vehicle information such as the shift lever operation state, accelerator operation state, brake operation state, steering operation state, speed, etc. from the vehicle information detection sensor 15 and outputs it to the map generation unit 27.

[0043] The map generation unit 27 generates a map M based on the vehicle information input from the vehicle information acquisition unit 26 and records it in chronological order in the storage unit 30. The period to be recorded may be the period from when the engine starts to when it stops, or the period from the start to the end of a parking operation. Based on the position information of the obstacle 5, the map generation unit 27 places the obstacle on the map M in the vehicle coordinate system, that is, records the coordinates of the obstacle 5 on the map M.

[0044] An example of the operation of the image generation system 100 according to the first embodiment will be described below with reference to the flowchart in Fig. 8. The flowchart in Fig. 8 shows an example of the operation of the image generation device 20. The operation of the image generation device 20 is not limited to the operation shown in Fig. 8. The operation of the image generation system 100 is also not limited to the operation described below.

[0045] 8 is repeated while the engine of the vehicle 1 is running. When the engine of the vehicle 1 is started, the surrounding information detection sensor 14 starts detecting the obstacle 5, the vehicle information detection sensor 15 starts detecting the vehicle information, and the side cameras 11 and 12 and the rear camera 13 start capturing images of the right rear side, the left rear side, and the rear.

[0046] The image acquisition unit 22 of the image generation device 20 starts acquiring image signals from the side cameras 11, 12 and the rear camera 13, and outputs the acquired image signals to the image processing unit 25. The image processing unit 25 displays a right composite image, which is a composite of the right rear side image taken by the right side camera 11 and the rear image taken by the rear camera 13, on the first display unit 41. The image processing unit 25 displays a left composite image, which is a composite of the left rear side image taken by the left side camera 12 and the rear image taken by the rear camera 13, on the second display unit 42. The image processing unit 25 displays the rear image taken by the rear camera 13 on the third display unit 43.

[0047] In step S1, the obstacle detection unit 23 detects obstacles 5 around the vehicle 1 based on measurement information acquired from the surrounding information detection sensor 14, and acquires position information of the obstacles 5. In step S2, the position determination unit 24 determines whether one or more obstacles 5 are located in the first mask area 54 or the second mask area 55 based on the position information of the obstacles 5.

[0048] If it is determined in step S2 that one or more obstacles 5 are located in the first mask area 54 or the second mask area 55 (Yes), the program proceeds to step S3. If it is determined in step S2 that one or more obstacles 5 are not located in the first mask area 54 or the second mask area 55, the program proceeds to END. If the determination in step S2 is No, even if an obstacle 5 is detected, the obstacle 5 is photographed by the side cameras 11, 12 or the rear camera 13 and is shown in the right rear side image, left rear side image, or rear image, so there is no need to superimpose the abstract image Gs.

[0049] In step S3, the image processing unit 25 calculates the drawing position of the abstract image Gs on the composite image according to the position information of the obstacle 5. In step S4, the image processing unit 25 determines the size of the abstract image Gs according to the distance to the obstacle 5, and obtains the abstract image Gs according to the determined size from the storage unit 30. In step S5, the image processing unit 25 superimposes the selected abstract image Gs at the calculated drawing position on the composite image and displays it on the first display unit 41 or the second display unit 42. The program then proceeds to END.

[0050] With the above, the operation of the image generating device 20 shown in the flowchart of FIG. 8 is completed, but the process from step S1 is repeated while the engine is running.

[0051] (Second embodiment) In an image generation system 100 including an image generation device 20 according to a second embodiment of the present disclosure, the image generation device 20 further includes an image determination unit 28, indicated by a dashed line in FIG. 2 . Except for the image determination unit 28, the image generation system 100 according to the second embodiment has the same basic configuration as the first embodiment. In addition to the same functions as the first embodiment, the image generation system 100 according to the second embodiment also has the following functions: a function for selecting a type of abstract image Gs based on height information of the obstacle 5 and superimposing it on the composite image; a function for superimposing a captured image Gp of the obstacle 5 captured from a photographed image when the obstacle 5 was photographed at a past time or with another camera on the composite image; and a function for superimposing an image indicating the presence of the obstacle 5 on the composite image when the obstacle 5 was not photographed at a past time or with another camera. The following mainly describes the configuration and functions of the second embodiment that are different from the first embodiment.

[0052] "Capture" means cutting out an image of an area in which an obstacle 5 is believed to be captured from an image captured by a camera and temporarily or non-temporarily storing the image in the memory unit 30. A "captured image" refers to an image cut out from an image captured by a camera.

[0053] As with the first embodiment, the image generation system 100 according to the second embodiment is mainly configured to include an information acquisition device 10, an image generation device 20, a display device 40, and an operation unit 45. The information acquisition device 10, the display device 40, and the operation unit 45 have the same configurations and functions as those in the first embodiment. The surrounding information detection sensor 14 of the information acquisition device 10 includes a plurality of view cameras that capture images of the surroundings of the vehicle 1.

[0054] The image generating device 20 includes a control unit 21 and a storage unit 30. The storage unit 30 stores the same information as in the first embodiment, an abstract image Gs of the obstacle 5 according to the height information, and a threshold value for determining whether the obstacle 5 is high or low. The threshold value can be the height of the bumper according to the vehicle model.

[0055] The control unit 21 of the second embodiment functions as an image determination unit 28 in addition to the functions disclosed in the first embodiment. The obstacle detection unit 23 acquires height information in addition to the measurement information and position information of the obstacle 5 disclosed in the first embodiment. The height information is height data indicating the height of the obstacle 5 from the road surface. The obstacle detection unit 23 outputs the position information and height information of the obstacle 5 to the position determination unit 24 and the map generation unit 27.

[0056] As in the first embodiment, when the position determination unit 24 determines that the obstacle 5 is located in the first mask area 54 or the second mask area 55 based on the position information of the obstacle 5 acquired by the obstacle detection unit 23, the position determination unit 24 outputs height information to the image processing unit 25 in addition to the determination result and the position information of the obstacle 5.

[0057] As in the first embodiment, the image processing unit 25 displays the rear-side image or the composite image on the first display unit, the second display unit, or the third display unit, and selects an abstract image Gs from the storage unit 30 to superimpose on the composite image. The image processing unit 25 determines whether the height of the obstacle 5 is equal to or less than a threshold based on the height information. When the height of the obstacle 5 is equal to or less than the threshold, the image processing unit 25 determines that the obstacle 5 is a low obstacle, and selects an additional abstract image Gu from the storage unit 30. The additional abstract image Gu is an image that indicates the presence of a low obstacle.

[0058] An obstacle 5 whose height exceeds the threshold is called a high obstacle. A high obstacle means an obstacle whose height is higher than the threshold and which the user can see through the rear window 4. A low obstacle means an obstacle whose height is lower than the threshold and which the user cannot see through the rear window 4, and which the user cannot see even if the user opens the door. A low obstacle is an obstacle whose height is less than a predetermined height defined by the threshold.

[0059] When an obstacle 5 is located in the first mask area 54 or the second mask area 55, the image determination unit 28 determines, based on the map, whether the view camera is capturing an image of the obstacle 5 at the current time. When the position of the obstacle 5 at the current time is located within the angle of view of the view camera, the image determination unit 28 determines that the view camera is capturing an image of the obstacle 5 at the current time.

[0060] When an obstacle 5 is located in the first mask area 54 or the second mask area 55, the image determination unit 28 determines, based on the map M, whether the rear camera 13 or the side cameras 11, 12 photographed the obstacle 5 at a time in the past. When the position of the obstacle 5 was located within the angle of view of the rear camera 13 or the side cameras 11, 12 at a past time, the image determination unit 28 determines that the rear camera 13 or the side cameras 11, 12 photographed the obstacle 5 at a past time. When it is determined that the obstacle 5 has not been photographed by the view camera at the current time, the image determination unit 28 may determine, based on the map M, whether the rear camera 13 or the side cameras 11, 12 photographed the obstacle 5 at a past time. The image determination unit 28 extracts the past time when the obstacle 5 was detected by the side cameras 11, 12 or the rear camera 13, and the position of the obstacle 5 at that time, from the traveling history of the vehicle 1 calculated based on the obstacle position information and vehicle information. If the position of the obstacle 5 extracted from the driving history overlaps or is close to the position of the obstacle 5 at the current time detected by the surrounding information detection sensor 14, the image determination unit 28 determines that the obstacle 5 was photographed at a past time.

[0061] Based on the determination result of image determination unit 28, image processing unit 25 cuts out an area recognized as an obstacle 5 from a past frame of the rear side image or rear image to generate a captured image Gp. When image processing unit 25 obtains a determination result from image determination unit 28 that the obstacle 5 has been photographed by the view camera, image processing unit 25 acquires a captured image of the obstacle 5 photographed by the view camera. Image processing unit 25 compares the angle of view of the view camera with the coordinates of the obstacle 5 in the vehicle coordinate system, and cuts out an area including the obstacle 5 from the image photographed by the view camera. When image processing unit 25 obtains a determination result from image determination unit 28 that the side cameras 11, 12 or the rear camera 13 photographed the obstacle 5 at a past time, image processing unit 25 acquires a captured image of the obstacle 5 based on the photographed image stored in memory unit 30. The image processing unit 25 compares the angles of view of the side cameras 11, 12 and rear camera 13 with the coordinates of the obstacle 5 in the vehicle coordinate system at a past time, and cuts out the area containing the obstacle 5 from the image captured by either the side cameras 11, 12 or rear camera 13.

[0062] The image processing unit 25 superimposes the acquired capture image Gp on the abstract image Gs, and displays the superimposed image on the composite image on the first display unit 41 or the second display unit 42. If the image processing unit 25 determines that the obstacle 5 was not photographed by another camera or at a past time, it does not display the capture image Gp, but superimposes the abstract image Gs or the abstract image Gs and the additional abstract image Gu on the composite image and displays them on the first display unit 41 or the second display unit 42.

[0063] Fig. 9 illustrates an example of the right composite image Gm displayed on the first display unit 41. Fig. 9(a) is an example of the right composite image Gm when a low obstacle 5 is detected, in which an additional abstract image Gu is superimposed on the right composite image Gm together with the abstract image Gs. Fig. 9(b) is an example of the right composite image Gm when a high obstacle 5 is detected, in which the abstract image Gs is superimposed on the right composite image Gm.

[0064] 9(c) and 9(d) are examples of the right composite image Gm when multiple obstacles 5 are detected. As in FIG. 9(c), in the right composite image Gm when multiple low obstacles 5 are detected at different distances, an additional abstract image Gu is superimposed on an abstract image Gs of a size corresponding to the distance from the vehicle 1. As in FIG. 9(d), in the right composite image Gm when low obstacles 5 and tall obstacles 5 are detected at different distances, an abstract image Gs of a size corresponding to the distance from the vehicle 1 is superimposed, and an additional abstract image Gu is superimposed on the abstract image Gs indicating the presence of the low obstacle 5. FIG. 9(e) is an example of the right composite image Gm when an obstacle 5 is photographed by a side-view camera or at a past time, in which a captured image Gp is ​​superimposed on the abstract image Gs. In the case of a low obstacle 5, an additional abstract image Gu may be superimposed along with the captured image Gp.

[0065] In a modified example of the second embodiment, the map generation unit 27 can store, in addition to the coordinates of the obstacle 5, one frame of a rearward image or a rearward image taken by the side cameras 11, 12 or the rear camera 13 when the obstacle 5 is detected, or multiple frames of images before and after the detection, on the map M, or can store these in the storage unit 30 in association with the position information of the obstacle 5 on the map M. A captured image Gp can be easily generated from these frames.

[0066] In a modification of the second embodiment, the storage unit 30 in the second embodiment does not store the angle of view of the view camera on the map M, and the image determination unit 28 acquires the coordinates of the obstacle 5 based on the view camera image taken by the view camera. If the coordinates of the detected obstacle 5 overlap or are close to the coordinates of the obstacle 5 placed on the map M, the image determination unit 28 can determine that the obstacle 5 has been photographed by the view camera. The image processing unit 25 cuts out the area recognized as the obstacle 5 from the current frame of the view camera image to generate a captured image Gp.

[0067] An example of the operation of the image generation system 100 according to the second embodiment will be described below with reference to the flowchart of Fig. 10. The operation of the image generation system 100 is not limited to the operation described below.

[0068] As in the first embodiment, when the engine of the vehicle 1 starts, the surrounding information detection sensor 14 starts detecting the obstacle 5, the vehicle information detection sensor 15 starts detecting vehicle information, and the side cameras 11, 12 and the rear camera 13 start capturing images of the rear side and rear. The image generation device 20 displays a right rear side image or a right composite image on the first display unit 41, a left rear side image or a left composite image on the second display unit 42, and a rear image on the third display unit 43.

[0069] In step S11, the obstacle detection unit 23 detects obstacles 5 around the vehicle 1 based on the measurement information acquired from the surrounding information detection sensor 14, and acquires position information of the obstacles 5. In step S12, the position determination unit 24 determines whether one or more obstacles 5 are located in the first mask area 54 or the second mask area 55 based on the position information of the obstacles 5.

[0070] If it is determined in step S12 that one or more obstacles 5 are located in the first mask region 54 or the second mask region 55 (Yes), the program proceeds to step S13. If it is determined in step S12 that one or more obstacles 5 are not located in the first mask region 54 or the second mask region 55, the program proceeds to END.

[0071] In step S13, the image processing unit 25 calculates the drawing position of the abstract image on the composite image according to the position information of the obstacle 5. In step S14, the image processing unit 25 determines the size of the abstract image according to the distance to the obstacle 5, and acquires an abstract image according to the determined size from the storage unit 30. In step S15, the image processing unit 25 determines whether the obstacle 5 is a low obstacle 5 by determining whether the height of the obstacle 5 is equal to or less than a threshold. If the image processing unit 25 determines that the obstacle 5 is a low obstacle (Yes), the program proceeds to step S16, where the image processing unit 25 selects an additional abstract image Gu and superimposes it on the abstract image. If the image processing unit 25 determines that the obstacle 5 is a high obstacle (No), the image processing unit 25 does not select an additional superimposed image. The program then proceeds to step S17.

[0072] In step S17, the image determination unit 28 determines whether or not the obstacle 5 has been photographed by the view camera of the surrounding information detection sensor 14 at the current time. If it is determined in step S17 that the obstacle 5 has been photographed by the view camera (Yes), the program proceeds to step S18, where the image determination unit 28 generates a captured image Gp from the image of the obstacle 5 taken from the view camera image. The program then proceeds to step S21.

[0073] If it is determined in step S17 that the obstacle 5 has not been photographed by the view camera (No), the program proceeds to step S19, where the image determination unit 28 determines whether the obstacle 5 was photographed by the side cameras 11, 12 or the rear camera 13 at a past time. If it is determined in step S19 that the obstacle 5 was photographed at a past time (Yes), the program proceeds to step S20, where the image determination unit 28 generates a captured image Gp from the image of the obstacle 5 in the rear-side image or the rear image. Thereafter, the program proceeds to step S21. If it is determined in step S19 that the obstacle 5 was not photographed at a past time (No), the program proceeds to step S22.

[0074] In step S21, the image processing unit 25 superimposes the captured image Gp on the abstract image. Then, the program proceeds to step S22. In step S22, the image processing unit 25 superimposes the abstract image on the composite image and displays it on the first display unit 41 or the second display unit 42. Then, the program proceeds to END.

[0075] With the above, the operation of the image generating device 20 shown in the flowchart of FIG. 10 is completed, but the process from step S11 onwards is repeated while the engine is running.

[0076] As described above, the image generation device 20 according to each of the above embodiments is an image generation device that generates a composite image by combining a rear image captured behind the vehicle 1 with a rear-side image captured behind the vehicle 1. The image generation device 20 includes an obstacle detection unit 23 that detects the position of an obstacle 5 present around the vehicle 1, a position determination unit 24 that determines whether the obstacle 5 is located outside the area defined by the angles of view θ1 and θ2 of the side cameras 11 and 12 that capture images of the rear sides of the vehicle 1, the area where the side cameras 11 and 12 capture images of the body of the vehicle 1 and the area defined by the angle of view θ3 of the rear camera 13 that captures images behind the vehicle 1, and an image processing unit 25 that superimposes an image indicating the presence of the obstacle 5 on the composite image based on the position of the obstacle 5. When the obstacle 5 is located outside the area defined by the viewing angles θ1 and θ2 of the side cameras 11 and 12 where the side cameras 11 and 12 photograph the body of the vehicle 1 and outside the area defined by the viewing angle θ3 of the rear camera 13, the image processing unit 25 superimposes an image indicating the presence of the obstacle 5 on the composite image.

[0077] The image generation method executed by the control unit 21 of the image generation device 20 according to the above embodiment is an image generation method for generating a composite image by combining a rear image captured behind the vehicle 1 with a rear-side image captured behind the vehicle 1. The image generation method includes an obstacle detection step for detecting the position of an obstacle 5 present around the vehicle 1, a position determination step for determining whether the obstacle 5 is located outside the area defined by the angles of view θ1 and θ2 of the side cameras 11 and 12 capturing images of the rear sides of the vehicle 1, the area defined by the angle of view θ3 of the rear camera 13 capturing images behind the vehicle 1, and an image processing step for superimposing an image indicating the presence of the obstacle 5 on the composite image based on the position of the obstacle 5. The image processing unit step superimposes an image indicating the presence of the obstacle 5 on the composite image when the obstacle 5 is located within the area defined by the angles of view θ1 and θ2 of the side cameras 11 and 12 where the side cameras 11 and 12 capture images of the body of the vehicle 1 and outside the area defined by the angle of view θ3 of the rear camera 13. The program according to the above embodiment is a program that causes a computer to execute each step of the image generation method.

[0078] The image generation device 20, image generation system 100, image processing method and program according to the above embodiments can more appropriately allow the user to understand that the obstacle 5 is located in an area that is not photographed by either the side cameras 11, 12 or the rear camera 13, and in particular that the obstacle 5 is located outside the area defined by the viewing angles θ1, θ2 of the side cameras 11, 12 in which the side cameras 11, 12 photograph the body of the vehicle 1 and outside the area defined by the viewing angle θ3 of the rear camera 13.

[0079] When the obstacle 5 is located outside the area defined by the angle of view θ1, θ2 of the side cameras 11, 12 where the side cameras 11, 12 capture the body of the vehicle 1 and outside the area defined by the angle of view θ3 of the rear camera 13, the image processing unit 25 of the image generation device 20 or the image processing step of the image processing method in each of the above embodiments calculates the position where an image indicating the presence of the obstacle 5 will be displayed on the composite image based on the position of the obstacle 5, and superimposes the image indicating the presence of the obstacle 5 on the composite image with a size based on the position of the obstacle 5. This allows the image generation device 20 or the image generation method to more appropriately let the user understand the presence of the obstacle 5.

[0080] The image generation device 20 or the image generation method of each of the above embodiments further includes an image determination unit 28 or an image determination step that determines whether an obstacle 5 was captured at a time in the past in a rear-side image captured from the rear of the vehicle 1 or a rear image captured from the rear of the vehicle 1. When an obstacle was captured at a time in the past in a rear-side image captured from the rear of the vehicle 1 or a rear image captured from the rear of the vehicle, the image processing unit 25 or the image processing step superimposes a captured image Gp, which is an image of the obstacle 5 captured at a time in the past, onto the composite image. This allows the image generation device 20 or the image generation method of each embodiment to allow the user to specifically understand not only the location of the obstacle but also what the obstacle is. Furthermore, a captured image Gp, which is a more accurate image of the obstacle with less distortion, is acquired from the image captured by the side cameras 11, 12 or the rear camera 13.

[0081] The image processing unit 25 of the image generation device 20 or the image processing step of the image generation method of each of the above embodiments superimposes an image indicating the presence of the obstacle 5 on the composite image when the obstacle 5 was not photographed at a past time. In this way, the image generation device 20 or the image generation method allows the user to more appropriately understand the position of the obstacle.

[0082] Although the embodiments of the present disclosure have been described in detail above with reference to the drawings, the specific configurations are not limited to these embodiments, and design changes that do not deviate from the gist of the present disclosure are included in the present disclosure.

[0083] Depending on the angles of view of the side cameras 11, 12 and the rear camera 13, as shown in FIG. 3B, areas 56, 57 of the obstacle 5 that are not captured by either the side cameras 11, 12 or the rear camera 13 may exist between the angles of view θ1, θ2 of the side cameras 11, 12 that capture the rearward and lateral sides of the vehicle 1 and the angle of view θ3 of the rear camera 13 that captures the area behind the vehicle 1. Hereinafter, the area 56 outside the angles of view will be referred to as the first non-captured area 56, and the area 57 outside the angles of view will be referred to as the second non-captured area 57. The first and second non-captured areas 56, 57 are not included in the "area defined by the angles of view of the side cameras that capture the rearward and lateral sides of the vehicle, that is, the area where the side cameras capture the vehicle body and that is outside the area defined by the angle of view of the rear camera that captures the area behind the vehicle." Obstacles 5 located in the first and second non-captured areas 56, 57 may not be subject to determination by the position determination unit 24 in each of the above-described embodiments. For this reason, in another different embodiment, the position determination unit 24 can be configured to "determine whether an obstacle is located in an area defined by the angle of view of the side camera that photographs the rear side of the vehicle, outside the area where the side camera photographs the body of the vehicle and the area defined by the angle of view of the rear camera that photographs the rear of the vehicle, or whether an obstacle is located in an area between the angle of view of the side camera and the angle of view of the rear camera, outside the angle of view of the side camera and the angle of view of the rear camera."

[0084] The image processing unit 25 can be configured to superimpose an image indicating the presence of the obstacle 5 on the composite image when the obstacle 5 is located in the area defined by the angles of view θ1, θ2 of the side cameras 11, 12, outside the area where the side cameras 11, 12 photograph the body of the vehicle 1 and outside the area defined by the angle of view θ3 of the rear camera 13, or when the obstacle 5 is located in an area between the angles of view θ1, θ2 of the side cameras 11, 12 and the angle of view θ3 of the rear camera 13, outside the angles of view θ1, θ2 of the side cameras 11, 12 and outside the angle of view θ3 of the rear camera 13.

[0085] This configuration allows the user to more appropriately understand the presence of an obstacle 5 that is not captured by the side cameras 11, 12 or the rear camera 13. By widening the angle of view of the side cameras 11, 12 or the rear camera 13, the first and second non-captured areas 56, 57 are eliminated or become very small. Therefore, each part of the image generating device 20 only needs to be aware that "the obstacle is in the area defined by the angle of view of the side cameras capturing images of the rear side of the vehicle, in the area where the side cameras capture images of the vehicle body and outside the area defined by the angle of view of the rear camera capturing images of the rear of the vehicle." [Explanation of symbols]

[0086] 1: vehicle, 5, 5a, 5b, 5c: obstacle, 11: right side camera, 12: left side camera, 13: rear camera, 20: image generating device, 21: control unit, 23: obstacle detection unit, 24: position determination unit, 25: image processing unit, 28: image determination unit, 54: first mask area, 55: second mask area, Gb: rear image, Gm: right composite image, Gp: captured image, Gr: right rear side image, Gs: abstract image, Gu: additional abstract image, θ1, θ2, θ3: angle of view

Claims

1. An image generating device that generates a composite image by combining a rear image obtained by photographing a rear side of a vehicle with a rear side image obtained by photographing a rear of the vehicle, an obstacle detection unit that detects the position of an obstacle present around the vehicle; a position determination unit that determines whether the obstacle is located outside an area that is defined by the angle of view of a side camera that photographs the rear side of the vehicle, and that is outside an area that is defined by the angle of view of a rear camera that photographs the rear of the vehicle and that is defined by the angle of view of the side camera that photographs the rear of the vehicle; an image processing unit that superimposes an image indicating the presence of the obstacle on the composite image based on the position of the obstacle, The image processing unit is characterized in that, when the obstacle is located within an area defined by the angle of view of the side camera, but outside the area defined by the angle of view of the rear camera where the side camera photographs the vehicle body, an image indicating the presence of the obstacle is superimposed on the composite image.

2. 2. The image generating device according to claim 1, wherein, when the obstacle is located outside the area defined by the angle of view of the side camera, where the side camera captures an image of the vehicle body, and outside the area defined by the angle of view of the rear camera, the image processing unit calculates a position on the composite image where an image indicating the presence of the obstacle will be displayed based on the position of the obstacle, and superimposes the image indicating the presence of the obstacle on the composite image with a size based on the position of the obstacle.

3. an image determination unit that determines whether the obstacle was photographed at a time earlier than the present time in a rear-side image photographed from the rear side of the vehicle or a rear image photographed from the rear of the vehicle; 2. The image generating device according to claim 1, wherein when the obstacle was photographed at the past time in a rear-side image photographed from the rear side of the vehicle or a rear image photographed from the rear of the vehicle, the image processing unit superimposes an image of the obstacle photographed at the past time on the composite image.

4. 4. The image generating device according to claim 3, wherein the image processing unit superimposes an image indicating the presence of the obstacle on the composite image when the obstacle was not photographed at the past time.

5. 1. An image generating method executed by a control unit of an image generating device that generates a composite image by combining a rear image captured behind a vehicle with a rear side image captured behind the vehicle, the method comprising: an obstacle detection step of detecting the position of an obstacle present around the vehicle; a position determination step of determining whether the obstacle is located outside an area defined by the angle of view of a side camera that photographs the rear side of the vehicle, the area being defined by the angle of view of the side camera that photographs the vehicle body and the rear camera that photographs the vehicle; an image processing step of superimposing an image showing the obstacle on the composite image based on the position of the obstacle, The image processing step is an image generation method characterized in that, when the obstacle is located within an area defined by the angle of view of the side camera, but outside the area defined by the angle of view of the rear camera where the side camera photographs the vehicle body, an image indicating the presence of the obstacle is superimposed on the composite image.

6. 6. The image generation method according to claim 5, wherein the image processing step, when the obstacle is located outside the area defined by the angle of view of the side camera, where the side camera captures an image of the vehicle body, and outside the area defined by the angle of view of the rear camera, calculates a position on the composite image where an image indicating the presence of the obstacle will be displayed based on the position of the obstacle, and superimposes the image on the composite image at a size based on the position of the obstacle.

7. The method further includes an image determination step of determining whether the obstacle was photographed at a time earlier than the present time in a rear-side image photographed behind the vehicle or a rear image photographed behind the vehicle, 6. The image generating method according to claim 5, wherein the image processing step includes superimposing an image of the obstacle photographed at the past time on the composite image when the obstacle was photographed at the past time in a rear-side image photographed from the rear side of the vehicle or a rear image photographed from the rear of the vehicle.

8. 8. The image generating method according to claim 7, wherein the image processing step includes superimposing an image of the obstacle on the composite image when the obstacle was not photographed at the past time.

Images