Image generation apparatus and image generation method

Abstract

Even if obstacles or other objects are present around the vehicle, the system must accurately convey information about the object to the user. [Solution] The image generation device comprises an acquisition unit that acquires an image of the area around a vehicle, an identification unit that identifies a reference point which is the location of the vehicle and a target point which is the location of an object to be presented to the user, a detection unit that detects an object present around the vehicle, and an image generation unit that superimposes marks on the image or composite image that associate a reference position corresponding to the reference point in the image or composite image with a target position corresponding to the target point in the image or composite image, wherein when the object is located between the reference point and the target point, the image generation unit superimposes an arc connecting the reference position and the target position on the image or composite image.

Description

Technical Field

[0001] The present invention relates to an image generation apparatus and an image generation method.

Background Art

[0002] Conventionally, as a technique for superimposing and displaying information on an image showing the periphery of a vehicle, there is Patent Document 1. Patent Document 1 describes that "a head-up display includes a self-vehicle position acquisition means, a display means, a first display control means, and a second display control means. The self-vehicle position acquisition means acquires the position of a moving body. The display means displays a virtual image by superimposing it on the scenery in front of the moving body. The first display control means superimposes a first guidance image indicating a route from the current position of the moving body to a destination or a stopover on the scenery and displays it on the display means. The second display control means superimposes a second guidance image indicating the distance from a guidance point existing on the route indicated by the first guidance image on the first guidance image and displays it on the display means."

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art, information on other objects and geographical information can be superimposed on an object viewed by a user through a head-up display, and the relative position and direction from a vehicle can be indicated.

[0005] Images using footage captured by an in-vehicle camera around a vehicle may include obstacles located between the object and the vehicle. These images may be composite images displaying the vehicle from a virtual viewpoint outside the vehicle, or they may be forward-facing images captured by the in-vehicle camera in front of the vehicle. If an object is present between the object and the vehicle, and a display indicating the object is superimposed, the user may mistakenly believe that the display is pointing to the obstacle rather than the object.

[0006] The present invention aims to accurately transmit information about an object to the user, even when obstacles or other objects are present around the vehicle. [Means for solving the problem]

[0007] To achieve the above objective, one representative image generation apparatus of the present invention comprises: an acquisition unit that acquires an image of the area around a vehicle; an identification unit that identifies a reference point which is the position of the vehicle and a target point which is the position of an object to be presented to the user; a detection unit that detects objects present around the vehicle; and an image generation unit that superimposes marks on the image or composite image that associate a reference position corresponding to the reference point in the image or composite image with a target position corresponding to the target point in the image or composite image, wherein when the object is located between the reference point and the target point, the image generation unit superimposes an arc connecting the reference position and the target position on the image or composite image. [Effects of the Invention]

[0008] According to the present invention, even when objects are present around the vehicle, information about those objects can be accurately transmitted to the user. [Brief explanation of the drawing]

[0009] [Figure 1] Specific examples of representations in the examples [Figure 2] Configuration diagram of the image generation device [Figure 3] Flowchart showing the processing procedure of the image generation device [Figure 4] An example of displaying arc heights that change according to distance. [Figure 5] An example of displaying arc widths that change according to distance. [Figure 6] Example of display with changed arc slope [Modes for carrying out the invention]

[0010] The following describes an example using drawings. [Examples]

[0011] Figure 1 shows a specific example of the display in this embodiment. The image generation device 60 in this embodiment can superimpose and display a mark indicating a predetermined object on a composite image or a forward-facing image, etc., as described later. The image generation device 60 may be mounted on the vehicle 1 or it may be a portable device. Figure 1(a) is an example of a display in which a mark indicating an object is superimposed on a composite image 10. Figure 1(b) is an example of a display in which a mark indicating an object is superimposed on a forward-facing image 40.

[0012] The composite image 10 is generated from images taken by an in-vehicle camera around vehicle 1, and displays vehicle 1 and its surroundings as seen from a virtual viewpoint outside the vehicle. The composite image 10 includes a graphic 11 representing vehicle 1 and images taken by the in-vehicle camera of objects present around vehicle 1. The objects present around vehicle 1 are, for example, vehicle 12. In this embodiment, the image generation device 60 superimposes marks on the composite image 10 that indicate objects designated by the user as objects to be presented to the user. The objects can be arbitrarily designated, such as entrances to commercial facilities or approaching pedestrians.

[0013] The image generation device 60 identifies a reference point, which is the position of vehicle 1, and a target point, which is the position of an object to be presented to the user. The image generation device 60 superimposes marks onto the composite image 10 that associate a reference position corresponding to the reference point in the composite image 10 with a target position corresponding to the target point in the composite image 10. The marks in Figure 1(a) are arcs extending from the reference position to the target position.

[0014] The image generation device 60, when an object is located between a reference point and a target point, makes a mark that is an arc connecting the reference point and the target point. By making the mark an arc, the user can easily recognize that the target object is on the other side of the object. It is preferable to draw the mark so that the height of its vertex exceeds the height of the object. By drawing the mark's vertex higher than the object, the mark does not obstruct the visibility of the object. The end of the arc on the target point side may also be an arrow.

[0015] The image generation device 60 superimposes panels that display information such as the name of the object as text near the marks. In Figure 1(a), the image generation device 60 indicates two objects using marks 21 and 22. The image generation device 60 displays panel 31 near mark 21 and panel 32 near mark 22. The image generation device 60 displays perspective clearly by varying the size of the panels to the extent that it does not affect the legibility of the text. Differences may be made in the display manner of marks 21 and 22. For example, the closest object may be indicated with a solid line mark, and the others with dotted line marks. Also, the thickness of the marks may be varied according to the distance from vehicle 1 to the object. Since it is preferable that the marks do not overlap the road surface, the starting point of the marks may be the ceiling portion of vehicle 1 in graphic 11. The ending point of the marks may be the vanishing point on the horizontal line in the direction of the object.

[0016] The front image 40 is an image captured by the front camera 51 of the vehicle 1 in front. The image generation device 60 superimposes a mark indicating the object specified by the user on the front image 40. The image generation device 60 superimposes a mark on the front image 40 that associates the reference position corresponding to the reference point in the front image 40 with the target position corresponding to the target point in the front image 40. The mark in Fig. 1(b) is an arc extending from the reference position to the target position.

[0017] When an object is located between the reference point and the target point, the image generation device 60 makes the mark an arc connecting the reference position and the target position. By making the mark an arc, the user can easily recognize that the object is on the other side of the object. It is preferable to draw the height of the apex of the mark so as to exceed the height of the object. By drawing the apex of the mark higher than the object, the mark does not obstruct the visibility of the object. The end on the target point side of the arc may be an arrow.

[0018] The image generation device 60 superimposes and displays a panel that displays information such as the name of the object as characters in the vicinity of the mark. In Fig. 1(b), the image generation device 60 indicates two objects by the marks 23 and 24. The image generation device 60 displays the panel 33 in the vicinity of the mark 23 and the panel 34 in the vicinity of the mark 24. The image generation device 60 displays the sense of depth clearly by making a difference in the size of the panel within a range that does not affect the visibility of the characters. A difference may be provided in the display mode of the marks 23 to 24. For example, the closest object may be indicated by a solid line mark, and the others may be indicated by a dotted line mark. Also, the thickness of the mark may be varied according to the distance from the vehicle 1 to the object. Since it is preferable that the mark does not overlap the road surface, the starting point of the mark may be shifted upward perpendicular to the road surface from the reference point so as not to overlap the road surface. The end point of the mark may be the vanishing point on the horizontal line in the direction of the object.

[0019] In Fig. 1(b), a display example in which a mark indicating an object is superimposed on the front image 40 is shown. However, a mark indicating an object may be superimposed and displayed on the rear image. The rear image is an image captured by the rear camera 54 behind the vehicle 1.

[0020] Fig. 2 is a configuration diagram of the image generation device 60. The image generation device 60 is mounted on the vehicle 1 and is connected to an in-vehicle camera or the like, a display unit 71, and an input unit 72. The front camera 51, the right side camera 52, the left side camera 53, the rear camera 54, the right rear side camera 55, the left rear side camera 56, and the optical sensor 57 are mounted on the vehicle 1. The front camera 51, the right side camera 52, the left side camera 53, the rear camera 54, the right rear side camera 55, and the left rear side camera 56 may be called in-vehicle cameras when they are not distinguished.

[0021] The front camera 51 captures an image in front of the vehicle 1. The right side camera 52 captures an image on the right side of the vehicle 1. The left side camera 53 captures an image on the left side of the vehicle 1. The rear camera 54 captures an image behind the vehicle 1. The right rear side camera 55 captures an image in the right rear of the vehicle 1. The left rear side camera 56 captures an image in the left rear of the vehicle 1. The optical sensor 57 is a sensor that detects objects around the vehicle 1, for example, LiDAR (Light Detection and Ranging). In addition, any remote sensing device such as a sonic radar can be used. The display unit 71 is a liquid crystal display, a head-up display, or the like. The input unit 72 is a touch panel or the like. The user can specify an object by inputting a predetermined instruction to the input unit 72.

[0022] The image generation device 60 includes, as functional units, an acquisition unit 61, a specification unit 62, a communication unit 63, a detection unit 64, and an image generation unit 65. The image generation device 60 is, for example, a computer and includes a processing unit 60a that is a processor and a memory 60b that stores a predetermined program for realizing each functional unit. By the processing unit 60a reading and executing the predetermined program stored in the memory 60b, the processing unit 60a realizes the functions as the acquisition unit 61, the specification unit 62, the communication unit 63, the detection unit 64, and the image generation unit 65.

[0023] The acquisition unit 61 acquires images of the area around the vehicle 1 from the in-vehicle camera. The identification unit 62 identifies a reference point, which is the current position of the vehicle 1, and a target point, which is the position of the object to be presented to the user. In other words, the identification unit 62 identifies the relative position of the target point, where the object is located, with respect to the reference point. For example, the coordinates of the object are identified in a vehicle coordinate system where the center of the vehicle 1 is the origin O, the width direction of the vehicle 1 is the X-axis, and the length direction of the vehicle 1 is the Y-axis. A world coordinate system based on latitude and longitude may be used instead of the vehicle coordinate system. When displaying a mark indicating the object superimposed on the front or rear image, the reference point may be the position of the camera lens as the origin O, rather than the current position of the vehicle 1.

[0024] The target object is an object or sign that the user is interested in. For example, it could be an entrance or exit to a facility corresponding to the parking lot where vehicle 1 is currently located, an empty parking space, or a pedestrian with a UWB (Ultra Wide Band) tag. The target location where the target object is located can be identified by any method. For example, the image generation device 60 may be equipped with map data and used to identify the location of the target object when it is an entrance or exit to a facility. If the target object is an empty parking space, the acquisition unit 61 can perform image recognition processing on the acquired image to detect the parking space and identify its location. The communication unit 63 acquires the location of the target object through communication with an external source. If the target object is an entrance or exit to a facility, the communication unit 63 can communicate with an external server and receive the target location where the target object is located. If the target object is a pedestrian with a UWB tag, the communication unit 63 can identify the target location where the target object is located using the UWB's location measurement function.

[0025] The detection unit 64 detects the position and height of objects located around the vehicle 1 based on images captured by each camera or data acquired by optical sensors. Alternatively, a recognition unit (not shown) may detect the position and height of objects located around the vehicle 1 based on images captured by each camera or data acquired by optical sensors, and the detection unit 64 may detect the position and height of objects located around the vehicle 1 by receiving the position and height of objects located around the vehicle 1 from the recognition unit (not shown). Objects include obstacles that may obstruct the movement of the vehicle 1 and signs drawn on the road surface. Examples of signs drawn on the road surface include parking space lines, which are lines that define the area where the vehicle 1 should park.

[0026] The image generation unit 65 superimposes marks on the image that correspond to a reference position in the image and a predetermined position in the image. The image generation unit 65 also superimposes marks on the composite image that correspond to a reference position in the composite image and a predetermined position in the composite image. The reference position is the position in the image or composite image that corresponds to the reference point. The target position is the position in the image or composite image that corresponds to the target point. The positions of the reference position and the target position are defined on an image coordinate system that defines the position of a point on the image or composite image. The image coordinate system is not particularly limited; for example, the origin of the image coordinate system is the center of the image, and the X-axis can be defined in the left-right direction from the origin, and the Y-axis in the up-down direction from the origin. When an object is located between the reference point and the target point, the image generation unit 65 superimposes an arc connecting the reference point and the target point onto the image or composite image. In other words, when an object is located between the reference point and the target point, the image generation unit 65 superimposes an arc as a marker. The image generation unit 65 determines whether an object is located between the reference point and the target point based on the position of the object present around the vehicle 1 detected by the detection unit 64. For example, the image generation unit 65 determines whether an object is located between the reference point and the target point by inputting the position of the object detected by the detection unit 64 into the coordinate system in which the identification unit 62 has identified the target point and the reference point.

[0027] The image generation unit 65 draws the reference point above the road surface. In the case of a composite image, the reference point is the roof portion of vehicle 1 in the graphic 11 in which vehicle 1 is drawn. In the case of a front image or a rear image, the reference point is set to be shifted upward perpendicular to the road surface and drawn in a position that does not overlap with the road surface.

[0028] When an object detected by the detection unit 64 is located between the reference point and the target point, the image generation unit 65 draws an arc such that the height of the vertex of the arc in the image or composite image exceeds the height of the object detected by the detection unit 64.

[0029] The image generation unit 65 may superimpose an arc onto the image when the detection unit 64 detects an object around the vehicle 1. In this case, the height of the arc in the image is set to be greater than the height of the object. The position and size of the object detected by the detection unit 64 are placed on the coordinate system where the point identified by the identification unit 62 is located. This makes it possible to determine whether there is an object between the reference point and the target point.

[0030] The image generation unit 65 displays multiple marks when there are multiple objects. The image generation unit 65 can indicate the positional relationship of multiple objects by appropriately changing the display characteristics such as the color, thickness, and line type of the marks.

[0031] The image generation unit 65 overlays a panel displaying information such as the name of the object as text near the mark. If there are multiple marks, the image generation unit 65 displays multiple panels. By appropriately changing the size, color, and other display characteristics of the panels, the image generation unit 65 can indicate the positional relationship of multiple object parts corresponding to multiple marks.

[0032] Figure 3 is a flowchart showing the processing procedure of the image generation device 60. The image generation device 60 sequentially executes the following steps S101 to S106. Step S101: The acquisition unit 61 acquires the image captured by the in-vehicle camera and the position of the in-vehicle camera from the in-vehicle camera. Then, the process proceeds to step S102. Step S102: The detection unit 64 detects the position and height of an object existing around the vehicle 1 based on the images captured by each camera or the data acquired by the optical sensor. Then, it proceeds to step S103. Step S103: The specifying unit 62 specifies the relative position of the target object with respect to the vehicle 1. Then, it proceeds to step S104. Step S104: The image generation unit 65 determines whether an object such as an obstacle is located between the target object and the camera. If the object is located between the target object and the camera, it proceeds to step S105. If the object is not located between the target object and the camera, it proceeds to step S106. Step S105: The image generation unit 65 draws an arc as a mark indicating the position of the target object. Then, the process ends. Step S106: The image generation unit 65 draws a straight line as a mark indicating the position of the target object. Then, the process ends.

[0033] Referring to FIGS. 4 to 6, a modified example of the mark display will be described. In FIG. 4, the image generation unit 65 draws arcs at different heights in the composite image according to the distance from the reference position to the target position. The height of the mark 25 is h 1, the height of the mark 26 is h 2, and the height of the mark 27 is h 3. The higher the height of the arc, the farther the target object is indicated. In the example of FIG. 4, since h 1 < h 2 < h 3, among the three target objects, the mark 25 points to the closest target object, and the mark 27 points to the farthest target object. By varying the height of the arc, the user can recognize the distance to the target object. As shown in FIGS. 4 and 6, the arc may be drawn such that the end on the target position side of the arc is an arrow and the end on the reference point side is not an arrow.

[0034] In Figure 5, the image generation unit 65 draws arcs of different thicknesses in the composite image, from the end on the reference position side to the end on the target position side. The image generation unit 65 draws the end of the arc on the reference position side with a different thickness depending on the distance from the reference position to the target position. In the example in Figure 5, the image generation unit 65 draws the end of the arc mark 28 on the reference position side thicker, and the end of another arc mark 29, which indicates a different object from mark 28, on the reference position side thinner. This allows the user to intuitively understand that mark 28 is pointing to the object closer to vehicle 1.

[0035] In Figure 6, the inclination of the arc, which is the mark, is changed to match the position of other objects in the image. By drawing the mark from a position higher than the road surface, signs drawn on the road surface are not obscured by the mark. It is also preferable to avoid a decrease in visibility due to the mark for signs, traffic lights, etc., that are positioned above the road surface. In Figure 6(a), mark 23 overlaps with traffic light 80. The image generation unit 65 identifies signs, traffic lights, etc., through image recognition, changes the inclination of mark 23, i.e., the arc, to avoid signs, traffic lights, etc., and draws Figure 6(b). In this way, by changing the inclination of the arc while maintaining the reference position and target position, the visibility of signs and traffic lights can be ensured. The modifications described in Figures 4 to 6 are applicable to both images and composite images.

[0036] As described above, the image generation device 60 includes an acquisition unit 61 that acquires an image of the area around the vehicle 1, an identification unit 62 that identifies a reference point which is the position of the vehicle 1 and a target point which is the position of an object to be presented to the user, a detection unit 64 that detects objects present around the vehicle 1, and an image generation unit 65 that superimposes marks on the image or composite image that associate a reference position corresponding to the reference point in the image or composite image with a target position corresponding to the target point in the image or composite image. When the object is located between the reference point and the target point, the image generation unit 65 superimposes an arc connecting the reference point and the target point on the image or composite image. This configuration and operation allows the image generation device 60 to accurately convey information about an object to the user, even if an object is present around the vehicle 1. Since the arc connecting the position of the vehicle 1 and the object is drawn to cross over the obstacle, the user can understand that the arc is pointing to an object beyond the obstacle.

[0037] Furthermore, when the object detected by the detection unit 64 is located between the reference point and the target point, the image generation unit 65 draws the arc in the image such that the height of the vertex of the arc exceeds the height of the object. Therefore, users can easily understand that the arc is pointing to an object behind the object, rather than an object located in front of the target point.

[0038] Furthermore, the image generation unit 65 draws the arc at different heights depending on the distance from the reference position to the target position. With this configuration, users can intuitively understand the distance to an object from the height of the arc.

[0039] Furthermore, the image generation unit 65 draws the arc with different thicknesses from the end on the reference position side to the end on the target position side, and draws the end of the arc on the reference position side with different thicknesses according to the distance from the reference position to the target position. With this configuration, users can intuitively understand the distance of an object from the change in the thickness of the arc.

[0040] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace or add configurations, not just delete them. [Explanation of symbols]

[0041] 10: Composite image, 11: Graphic, 12: Vehicle, 21-29: Mark, 31-34: Panel, 40: Front image, 51: Front camera, 52: Right side camera, 53: Left side camera, 54: Rear camera, 55: Right rear side camera, 56: Left rear side camera, 57: Optical sensor, 60: Image generation device, 60a: Processing unit, 60b: Memory, 61: Acquisition unit, 62: Identification unit, 63: Communication unit, 64: Detection unit, 65: Image generation unit, 71: Display unit, 72: Input unit, 80: Traffic light

Claims

1. An acquisition unit that acquires images of the area around the vehicle, A identifying unit that identifies a reference point which is the location of the vehicle and a target point which is the location of the object to be presented to the user, A detection unit for detecting objects present around the vehicle, The system includes an image generation unit that superimposes marks on the image or composite image that associate a reference position corresponding to the reference point in the image or composite image with a target position corresponding to the target point in the image or composite image. The image generation device is characterized in that, when the object is located between the reference point and the target point, it superimposes an arc connecting the reference point and the target point onto the image or the composite image.

2. An image generation apparatus according to claim 1, wherein the image generation unit, when the object detected by the detection unit is located between the reference point and the target point, draws the arc in the image or the composite image such that the height of the vertex of the arc exceeds the height of the object.

3. An image generation apparatus according to claim 1, wherein the image generation unit draws the arc at different heights according to the distance from the reference position to the target position.

4. An image generation apparatus according to claim 1, wherein the image generation unit draws an arc having different thicknesses from the end on the reference position side to the end on the target position side, and draws the end of the arc on the reference position side with different thicknesses according to the distance from the reference position to the target position.

5. The acquisition step involves obtaining images taken of the area around the vehicle, A selection step to identify a reference point which is the location of the vehicle and a target point which is the location of the object to be presented to the user, A detection step for detecting objects present around the vehicle, The image generation step includes superimposing a mark onto the image or composite image that associates a reference position corresponding to the reference point in the image or composite image with a target position corresponding to the target point in the image or composite image. The image generation step is an image generation method characterized in that, when the object is located between the reference point and the target point, an arc connecting the reference point and the target point is superimposed on the image or the composite image.

6. An image generation method according to claim 5, wherein the image generation step is characterized in that, when the object detected in the detection step is located between the reference point and the target point, the arc is drawn in the image or the composite image such that the height of the vertex of the arc exceeds the height of the object.

7. An image generation method according to claim 5, wherein the image generation step is characterized by drawing the arc at different heights according to the distance from the reference position to the target position.

8. An image generation method according to claim 5, wherein the image generation step is characterized by drawing an arc with different thicknesses from the end on the reference position side to the end on the target position side, and drawing the end of the arc on the reference position side with different thicknesses according to the distance from the reference position to the target position.

Images