Image generation device and program

Abstract

To enable a driver to more accurately understand a region of a vehicle where there is a possibility of contact with an obstruction.SOLUTION: An image generation device 30 comprises: an acquisition unit 33 that acquires a position of an obstruction relative to a vehicle and a three-dimensional shape of the obstruction; a specifying unit 36 that identifies a region of the vehicle which may be in contact with the obstruction based on the shape of the vehicle and the position and shape of the obstruction; and a drawing unit 37 that generates a vehicle image as if the vehicle were viewed from a viewpoint positioned outside the vehicle, and that highlights and draws a region of the vehicle image identified by the specifying unit 36. The specifying unit 36 that specifies a possibility of a region which contacts the obstruction among a plurality of regions which divide the vehicle at different heights in a height direction based on the shape of the vehicle and the position and shape of the obstruction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image generating device and a program. [Background technology]

[0002] Conventionally, techniques for generating and displaying images of the surroundings of a vehicle are known (see, for example, Patent Documents 1 and 2). Patent Document 1 discloses a technique for generating and displaying an overhead image of the surroundings of the vehicle from a virtual viewpoint based on images acquired by a camera that captures the surroundings of the vehicle. Patent Document 2 discloses a technique for generating a three-dimensional image of the vehicle as if looking down on it from a virtual viewpoint outside the vehicle, and superimposing an image of a vehicle model on this three-dimensional image.

[0003] Patent Document 1 discloses a technique for superimposing an alternative image corresponding to an obstacle detected around the vehicle on the overhead image by changing the direction and tilt of the alternative image to match a virtual viewpoint when displaying the overhead image. Patent Document 3 discloses a technique for displaying an obstacle display unit on the overhead image of the vehicle when an obstacle is detected around the vehicle, the obstacle display unit indicating the detected position of the obstacle around the vehicle and the distance between the vehicle and the obstacle.

[0004] Patent Document 4 discloses a technology for a peripheral image display control device that displays an image of the surrounding area captured by a camera, detects the position and height of an obstacle present in the vehicle's travel path, and displays in color the image area of ​​the obstacle that the vehicle may not be able to overcome.Patent Document 5 discloses a technology that, when an obstacle of a predetermined height is present in a parking area detected based on a camera image, displays a position corresponding to the position of the obstacle as the parking completion position. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-251939 [Patent Document 2] Japanese Patent Application Publication No. 2019-503007 [Patent Document 3] Japanese Patent Application Publication No. 2017-162015 [Patent Document 4] Japanese Patent Application Publication No. 2019-016043 [Patent Document 5] Japanese Patent Application Publication No. 2019-043455 Summary of the Invention [Problem to be solved by the invention]

[0006] The inventions described in Patent Documents 1, 2, and 5 do not inform the driver whether there is a possibility of the vehicle coming into contact with an obstacle. The invention described in Patent Document 3 makes it difficult for the driver to know whether the entire obstacle display displayed on the vehicle will come into contact with the obstacle, or whether only a portion of it is likely to come into contact. The invention described in Patent Document 4 does not inform the driver which area of ​​the vehicle is likely to come into contact with an obstacle displayed in color.

[0007] The present disclosure aims to give drivers a more accurate understanding of areas of their vehicle that may come into contact with obstacles. [Means for solving the problem]

[0008] In order to achieve the above object, an image generation device of the present disclosure is an image generation device that generates a vehicle image as if the vehicle were viewed from a viewpoint located outside the vehicle, and includes: an acquisition unit that acquires the position of an obstacle relative to the vehicle and the three-dimensional shape of the obstacle; an identification unit that identifies an area of ​​the vehicle that may come into contact with the obstacle based on a pre-stored shape of the vehicle and the position and shape of the obstacle; and a drawing unit that generates the vehicle image as if the vehicle were viewed from a viewpoint located outside the vehicle and draws the area identified by the identification unit in an emphasized manner in the vehicle image. The identification unit identifies an area that may come into contact with the obstacle from among a plurality of areas obtained by dividing the vehicle at different heights in the height direction, based on the shape of the vehicle and the position and shape of the obstacle. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to allow a driver to more accurately understand the areas of the vehicle where contact with an obstacle is likely. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing the functional configuration of an image generation system including an image generation device according to a first embodiment. [Figure 2] FIG. 2 is a diagram schematically illustrating a vehicle and an obstacle detected by a sensor unit. [Figure 3] FIG. 10 is a diagram for explaining a procedure for creating a space map. [Figure 4] 10 is a diagram for explaining the distance between a point located on the side of an obstacle and a point located on the side of a vehicle, and a specific area. FIG. [Figure 5] 10A and 10B show three-dimensional images drawn by the drawing unit, where (a) is a three-dimensional image when the obstacle is a cone, (b) is a three-dimensional image when the obstacle is a bicycle wheel, and (c) is a three-dimensional image when the obstacle is a post. [Figure 6] 10 is a diagram for explaining the distance between a point located on the side of an obstacle and a point located on the side of a vehicle, and a specific area in the image generation system of the second embodiment. FIG. [Figure 7] 10A and 10B are diagrams showing three-dimensional images drawn by the drawing unit of the second embodiment, where (a) is a three-dimensional image when the obstacle is a cone, (b) is a three-dimensional image when the obstacle is a bicycle wheel, and (c) is a three-dimensional image when the obstacle is a post. [Figure 8A] 10 is a flowchart illustrating an example of the flow of operations of the image generating device. [Figure 8B] 10 is a flowchart illustrating an example of the flow of operations of the image generating device. [Figure 8C] 10 is a flowchart illustrating an example of the flow of operations of the image generating device. DETAILED DESCRIPTION OF THE INVENTION

[0011] (First embodiment) An image generation system 100 including an image generation device 30 according to a first embodiment of this disclosure will be described below with reference to the drawings. As shown in FIG. 1, the image generation system 100 mainly includes a sensor unit 10, an imaging unit 20, the image generation device 30, a display unit 40, and an operation unit 50.

[0012] As shown in FIG. 2, the sensor unit 10 is installed outside the vehicle 1 and is a device that detects obstacles 2 around the vehicle 1. For example, the sensor unit 10 is composed of an ultrasonic sonar. An ultrasonic sonar is a device that detects obstacles by transmitting ultrasonic waves into the surrounding area and receiving the waves reflected by objects such as obstacles around the vehicle. The sensor unit 10 is not limited to an ultrasonic sonar, and may be any device that can detect obstacles 2, such as a three-dimensional lidar (LiDAR), a camera, or a stereo camera.

[0013] One or more sensor units 10 are provided in front and / or behind the vehicle 1, and output measurement information obtained by measuring an obstacle 2 to the image generation device 30. If the sensor unit 10 is an ultrasonic sonar or a three-dimensional lidar, the sensor unit 10 calculates the distance from the vehicle 1 to the obstacle 2 based on the elapsed time between emitting an ultrasonic wave and detecting a reflected wave, and outputs measurement information including the calculated distance from the vehicle 1 to the obstacle 2, the intensity of the reflected wave, the relative position, etc. to the image generation device 30. If the sensor unit 10 is a camera or a stereo camera, the sensor unit 10 calculates the distance from the vehicle 1 to the obstacle 2 based on the output obtained by inputting an image captured by the camera into a deep learning model, the parallax between the cameras, etc., and outputs measurement information including the calculated distance from the vehicle 1 to the obstacle 2, the relative position, etc. to the image generation device 30.

[0014] As shown in Fig. 2, the imaging unit 20 is installed outside the vehicle 1 and captures images of the surroundings of the vehicle 1. The imaging unit 20 converts the captured images into image signals according to a predetermined protocol and outputs them to the image generating device 30. The imaging unit 20 includes a front camera installed in front of the vehicle 1. The cameras included in the imaging unit 20 are not limited to a front camera, but may also include a rear camera installed in the rear of the vehicle 1, side cameras installed on the left and right front and left and rear of the vehicle 1, etc.

[0015] The display unit 40 is composed of a liquid crystal display or the like installed inside the vehicle, and displays images captured by the imaging unit 20, three-dimensional images generated by the image generation device 30, etc. The display unit 40 doubles as a monitor for the navigation device, but can also be a monitor installed inside the vehicle separately from this monitor. The display unit 40 has a touch panel type display surface, and this display surface functions as the operation unit 50. The operation unit 50 can also have various operation buttons provided around the display unit 40.

[0016] By operating the touch panel display surface of the display unit 40, which is the operation unit 50, with a finger, the driver or the like can change the viewpoint of the three-dimensional image displayed on the display unit 40 and freely change the display direction of the three-dimensional image.

[0017] The image generating device 30 is installed inside the vehicle and is an information processing device that generates a three-dimensional image 60 including a vehicle image 61 (see FIG. 5) that looks as if the vehicle 1 is viewed from a viewpoint (hereinafter also referred to as a "virtual viewpoint") located outside the vehicle 1, based on the image signal output from the imaging unit 20. The image generating device 30 can generate a three-dimensional stereoscopic vehicle image 61 and a three-dimensional image 60 that looks as if the vehicle 1 is viewed from any viewpoint selected by the driver, which is a virtual viewpoint located outside the vehicle 1.

[0018] The image generating device 30 identifies an area where the vehicle 1 may come into contact with the obstacle, based on the measurement information output from the sensor unit 10, and in accordance with the distance between the detected obstacle 2 and the vehicle 1, and displays the area in an emphasized manner on the vehicle image 61. The area where the vehicle 1 may come into contact with the obstacle is not limited to an area identified in accordance with the distance.

[0019] The image generation device 30 is composed of an ECU having processing devices such as a CPU and a GPU, and storage devices such as a RAM and a ROM. The image generation device 30 can be configured with one ECU, or can be configured with multiple ECUs to distribute the functions of the control unit 31 and to distribute the data to be stored. The image generation device 30 can also be configured with an FPGA, an ASIC, etc.

[0020] 1, the image generating device 30 functions as a control unit 31 and a storage unit 32. The control unit 31 is mainly composed of a processing device such as a CPU included in the ECU, and controls the overall operation of the image generating device 30 by loading a predetermined program stored in a ROM into a RAM and executing it. The storage unit 32 is mainly composed of a storage device such as a RAM included in the ECU, but may also include an external server or database.

[0021] The memory unit 32 stores a control program for operating the image generation device 30, an application program including an image generation program, and the like. The memory unit 32 temporarily or non-temporarily stores various data and parameters, such as thresholds, vehicle shape data, height information, and width information, used by the control unit 31. The threshold is a parameter used to determine whether or not there is a possibility of contact with an obstacle. The vehicle shape data is data representing the shape of the vehicle 1, and vehicle shape data for various shapes of the vehicle 1 is associated with the vehicle 1 and stored in the memory unit 32. The height information is a parameter that can identify the height direction of an area where there is a possibility of contact with an obstacle 2 by specifying the radius of a circle centered on point Pi, the number of pixels, the number of polygons, and the like, as a predetermined range described below, in the height direction of the vehicle 1 or vehicle model V in the three-dimensional coordinate system of the spatial map M. The width information is a parameter that can identify the width direction of an area where there is a possibility of contact with an obstacle 2 by specifying the number of pixels or the number of polygons in the width direction of the vehicle 1 or vehicle model V in the three-dimensional coordinate system of the spatial map M described below, regardless of the width of the obstacle model O.

[0022] As shown in FIG. 1, the control unit 31 functions as an acquisition unit 33, a model generation unit 34, a space map creation unit 35, an identification unit 36, a drawing unit 37, and a display control unit 38.

[0023] The acquisition unit 33 acquires the position of the obstacle 2 relative to the vehicle 1 and the three-dimensional shape of the obstacle 2. Based on the measurement information output from the sensor unit 10, the acquisition unit 33 acquires position data indicating the position of the obstacle 2 present in the traveling direction of the vehicle 1. The position data is data indicating the distance and direction between the sensor unit 10 and each point of an innumerable point cloud set on the side of the obstacle 2, i.e., the surface of the obstacle 2 facing the vehicle 1. Based on the acquired position data of the obstacle 2, the acquisition unit 33 acquires obstacle shape data indicating the three-dimensional shape of the obstacle 2. The obstacle shape data is three-dimensional coordinate data of each point of the point cloud set on the side of the obstacle 2, which is the point cloud that forms the shape of the obstacle 2.

[0024] The model generation unit 34 generates a vehicle model V (see FIG. 3) that shows the vehicle 1 in three dimensions based on the shape of the vehicle 1. The model generation unit 34 acquires vehicle shape data corresponding to the vehicle 1 from vehicle shape data that shows the shapes of various vehicles 1 stored in advance in the storage unit 32. The model generation unit 34 generates polygons as the vehicle model V of the vehicle 1 based on the acquired vehicle shape data, but the vehicle model V is not limited to polygons as long as it has a three-dimensional shape.

[0025] As shown in FIG. 3 , the spatial map creation unit 35 places the vehicle model V and the obstacle model O on a spatial map M of a three-dimensional coordinate system. The three-dimensional coordinate system may be formed, for example, with the center of the vehicle 1 as the origin and with an axis corresponding to the width direction of the vehicle 1, an axis corresponding to the length direction of the vehicle 1, and an axis corresponding to the height direction of the vehicle 1. The axis corresponding to the width direction of the vehicle 1 is the x-axis, the axis corresponding to the length direction of the vehicle 1 is the y-axis, and the axis corresponding to the height direction of the vehicle 1 is the z-axis. First, the spatial map creation unit 35 places the vehicle model V at a predetermined position on the spatial map M, for example, at the origin of the three-dimensional coordinate system. In this case, the center of the vehicle model V is the origin of the three-dimensional coordinate system, and the x-axis direction, which is the width direction of the vehicle model V, approximately coincides with the axis corresponding to the width direction of the vehicle 1 in the three-dimensional coordinate system. Similarly, the y-axis direction, which is the length direction of the vehicle model V, or the z-axis direction, which is the height direction, approximately coincides with the axis corresponding to the length direction of the vehicle 1 or the axis corresponding to the height direction of the vehicle. Next, the spatial map creation unit 35 creates an obstacle model O based on the obstacle shape data indicating the three-dimensional shape of the obstacle 2 acquired by the acquisition unit 33, and places the obstacle model O on the spatial map M in correspondence with the position and direction of the obstacle 2 from the vehicle 1 based on the position data.

[0026] 4, the space map creation unit 35 divides the side surface of the vehicle model V, which is a model of the vehicle 1, facing the obstacle model O in the height direction in the space map M into a plurality of regions at different heights based on the vehicle shape data. In other words, the space map creation unit 35 sets a plurality of points p1 to p5 at different heights H1 to H5 for each predetermined height h on the side surface of the obstacle model O at a density according to the resolution of the sensor unit 10 or image rendering constraints, and divides the side surface of the vehicle model V into a plurality of regions with division lines L1 to L5 extending horizontally from each point pi as a starting point.

[0027] Based on the position data and obstacle shape data of the obstacle 2, the space map creation unit 35 calculates, using a known calculation method, distances di, which are the lengths of dividing lines Li, which are line segments connecting multiple points Pi located on the side of the obstacle corresponding to the obstacle model O in the space map M, from multiple points pi located on the side of the vehicle corresponding to the vehicle model V. For example, in FIG. 4, the space map creation unit 35 calculates, using a known calculation method, distances d1 to d5 from points p1 to p5, which intersect with the dividing lines L1 to L5, to points P1 to P5, where the dividing lines L1 to L5 extend horizontally from points p1 to p5 and intersect with the vehicle model V. The space map creation unit 35 stores the calculated distances di in the memory unit 32. In FIG. 4, five dividing lines L1 to L5 define five points p1 to p5 and five points P1 to P5 on the obstacle model O and the vehicle model V, respectively, but this is not limiting. i is an integer greater than or equal to 1, and i can be increased or decreased as appropriate depending on the resolution of the sensor unit 10 and graphic limitations.

[0028] In this embodiment, the spatial map creation unit 35 divides the side of the vehicle model V into multiple regions at different heights and calculates the distance di between a point pi located on the side of the obstacle and a point Pi located on the side of the vehicle, but these processes can also be configured to be performed by the identification unit 36.

[0029] Based on the shape of the vehicle 1 and the position and shape of the obstacle 2, the identification unit 36 ​​identifies an area on the side of the vehicle 1 that may come into contact with the obstacle from among a plurality of areas obtained by dividing the vehicle at different heights in the height direction. Based on the distance di calculated by the space map creation unit 35, it is determined whether the distance di from point pi located on the side of the obstacle 2 corresponding to obstacle model O in the space map M to point Pi located on the side of the vehicle 1 corresponding to vehicle model V is less than a threshold value. Using point Pi where the distance di is less than the threshold value as a reference, the identification unit 36 ​​identifies an area within a predetermined range in the height direction and width direction centered on this point Pi as an area that may come into contact with the obstacle (hereinafter referred to as a "identified area").

[0030] The identification unit 36 ​​of the first embodiment identifies the height direction of the identified region based on height information stored in the memory unit 32. The identification unit 36 ​​identifies the width direction of the identified region using width information stored in the memory unit 32. The height information and width information can be the number of polygons in the height direction and width direction. The number of polygons can be a fixed number, or can be a number of polygons selected corresponding to the vehicle 1 from a plurality of numbers of polygons set according to the shape and size of the vehicle 1. The height information and width information are not limited to the number of polygons, and can also be the length in the height direction and width direction (units: cm, m, etc.), the number of pixels, etc.

[0031] The identification unit 36 ​​identifies, based on the height information, one or more polygons included in a predetermined range centered on point Pi from among multiple regions obtained by dividing the vehicle model V at different heights in the height direction of the vehicle 1 in the spatial map M. For example, assume that the distance d1 between p1 and P1, the distance d2 between p2 and P2, and the distance d3 between p3 and P3 in FIG. 4 are less than a threshold. In this case, the following are identified as the specific regions: a region a1 divided by multiple polygons included in a predetermined range centered on point P1; a region a2 divided by multiple polygons included in a predetermined range centered on point P2; and a region a3 divided by multiple polygons included in a predetermined range centered on point P3.

[0032] The identification unit 36 ​​can identify the width of the specific area in the width direction of the obstacle model O. For example, in FIG. 4 , the space map creation unit 35 or the identification unit 36 ​​can limit the width of the specific area in the width direction of the vehicle model V in the space map M based on the width information stored in the storage unit 32.

[0033] The identification unit 36 ​​can identify the identified area using one threshold value. In the first embodiment, multiple threshold values ​​are used to identify areas that are likely to come into contact with an obstacle and areas that are unlikely to come into contact with an obstacle, among areas identified as having the possibility of coming into contact with an obstacle. In other words, the identification unit 36 ​​determines the likelihood of coming into contact with an obstacle based on the length of the distance di between points, and identifies the identified area in more detail depending on the likelihood.

[0034] When the distance from point pi on the side of the obstacle to point Pi on the side of the vehicle is less than a first threshold, the identification unit 36 ​​identifies the area corresponding to the height of point pi on the side of the obstacle as the area most likely to come into contact with the obstacle (hereinafter referred to as the "first identified area"). In the example of Figure 4, if the distance d2 between p2 and P2 and the distance d2 between p3 and P3 are less than the first threshold, the identified areas a2 and a3 set corresponding to the heights of these points P2 and P3 are identified as the first identified area.

[0035] When the distance from point pi on the side of the obstacle to point Pi on the side of the vehicle is equal to or greater than a first threshold and less than a second threshold, the identification unit 36 ​​identifies an area corresponding to the height of point pi on the side of the obstacle as an area where the possibility of contact with the obstacle is low (hereinafter referred to as a "second identified area"). In the example of FIG. 4, if the distance d1 between p1 and P1 is equal to or greater than the first threshold and less than the second threshold, the identified area a2 set corresponding to the height of point P2 is identified as the second identified area. The second threshold is a value greater than the first threshold.

[0036] In this embodiment, when the distance di from point pi on the side of the obstacle to point Pi on the side of the vehicle is equal to or greater than a second threshold and less than a third threshold, the identification unit 36 ​​identifies an area corresponding to the height of point Pi on the side of the obstacle as an area where the possibility of contact with the obstacle is lower (hereinafter referred to as a "third identified area"). The third threshold is greater than the second threshold. In a case where the identification unit 36 ​​identifies an identified area using a single threshold, when the distance di is equal to or greater than the third threshold, the identification unit 36 ​​can identify a polygon included within a predetermined range from point P at a height corresponding to distance di as an identified area, and when the distance di is less than the third threshold, the identification unit 36 ​​can not identify a polygon included within a predetermined range from point Pi at a height corresponding to distance di as an identified area.

[0037] The rendering unit 37 generates a vehicle image 61 based on the vehicle model V and the position of the virtual viewpoint, as if the vehicle were viewed from a viewpoint located outside the vehicle, and also generates a three-dimensional image 60 including the vehicle image 61 and an obstacle image 62 (see FIGS. 5 and 6). To generate this three-dimensional image 60, the rendering unit 37 places a virtual camera C at a virtual viewpoint P that is set by default or by the driver operating the operation unit 50 in the space map M shown in FIG. 3, and creates a three-dimensional image 60 that looks as if the vehicle model V and obstacle model O were photographed by this virtual camera C. The rendering unit 37 can also include in the three-dimensional image 60 a three-dimensional landscape image based on an image of the surroundings of the vehicle 1 captured by the imaging unit 20.

[0038] The drawing unit 37 draws the vehicle image 61 by emphasizing the portions of the vehicle image 61 that correspond to the specific areas a1 to a3 identified by the identification unit 36 ​​more than the portions outside the specific areas. The drawing unit 37 draws the vehicle image 61 based on the vehicle model V generated by the model generation unit 34, but is not limited to this, and may also use a CG image or a photograph of the vehicle 1 that is stored in advance.

[0039] The drawing unit 37 draws and highlights portions of the vehicle image 61 that correspond to specific regions in a different manner than portions that are not specific regions. When a plurality of specific regions are identified, such as first to third specific regions, the drawing unit 37 draws and highlights each specific region in a different manner depending on the likelihood of the specific region contacting an obstacle. As shown in FIG. 5 , the drawing unit 37 draws a portion corresponding to the first specific region, which has the highest likelihood of contacting an obstacle, in red as a first highlighted region 63, a portion corresponding to the second specific region, which has a lower likelihood of contacting an obstacle than the first specific region, in orange as a second highlighted region 64, and a portion corresponding to the third region, which has a lower likelihood of contacting an obstacle than the second specific region, in yellow as a third highlighted region 65. In contrast, the drawing unit 37 draws the vehicle image 61 without highlighting portions of the side of the vehicle that correspond to regions that have not been identified as specific regions.

[0040] The method of highlighting a region is not limited to the coloring described above, and it is sufficient to distinguish between a region in a specific area and a region that is not a specific area, and if multiple specific areas are identified, it is sufficient to distinguish between each specific area. The drawing unit 37 can highlight the region corresponding to a specific area with shading, a pattern, a motif, etc. The method of distinguishing and highlighting multiple specific areas with different probabilities of contacting an obstacle is not limited to color coding, and it is sufficient to highlight them in different ways, so the drawing unit 37 can highlight the region corresponding to each specific area with different shading, patterns, motifs, etc.

[0041] FIG. 5 shows a three-dimensional image 60 drawn by the drawing unit 37. FIG. 5(a) is a three-dimensional image 60 in the case where the obstacle is a traffic cone. FIG. 5(b) is a three-dimensional image 60 in the case where the obstacle is a bicycle wheel. FIG. 5(c) is a three-dimensional image 60 in the case where the obstacle is a post. In each drawing, reference numeral 63 denotes a first emphasized region drawn in red, which corresponds to the first specific region. Reference numeral 64 denotes a second emphasized region drawn in orange, which corresponds to the second specific region. Reference numeral 65 denotes a third emphasized region drawn in yellow, which corresponds to the third specific region.

[0042] A cone has a triangular pyramid shape that widens downward, a bicycle wheel is circular, and a post has a square prism body with thin, narrow legs. Therefore, even if the vehicle is the same, the possibility and position of contact with each obstacle will differ depending on the shape of the obstacle, so as shown in Figures 5(a) to 5(c), the first to third highlighted regions 63 to 65 are drawn with different height positions and ranges.

[0043] Since the identification unit 36 ​​identifies the first to third identified regions based on predetermined width information, the width dimensions of the first to third highlighted regions 63 to 65 are constant, and each is drawn in an approximately rectangular shape, as shown in each diagram in Figure 5.

[0044] The display control unit 38 controls the display unit 40 to display a three-dimensional image 60 including a vehicle image 61 and an obstacle image 62 generated by the drawing unit 37 on the display surface.

[0045] An example of the operational flow of the image generation system 100 according to the first embodiment configured as described above will be described below with reference to the flowcharts of Figures 8A to 8C. The flowcharts of Figures 8A to 8C show an example of the operational flow of the image generation device 30, but the operation of the image generation device 30 is not limited to the operations shown in these flowcharts, and the operational flow of the image generation system 100 is not limited to the operations described below.

[0046] The operations shown in the flowcharts of Figures 8A to 8C begin when the engine of vehicle 1 starts, the sensor unit 10 starts detecting obstacles 2, and the imaging unit 20 starts capturing images of the surroundings of vehicle 1, and are repeated while the engine is running.

[0047] In step S1, the acquisition unit 33 acquires measurement information from the sensor unit 10, and acquires from the acquired measurement information the position (position data) and shape (obstacle shape data) of the obstacle 2. In the next step S2, the model generation unit 34 acquires from the memory unit 32 the shape (vehicle shape data) corresponding to the vehicle 1.

[0048] In the next step S3, the model generation unit 34 generates a vehicle model V of the vehicle 1 based on the acquired vehicle shape data, and generates an obstacle model O of the obstacle 2 based on the acquired measurement information. In the next step S4, the space map creation unit 35 places the generated vehicle model V and obstacle model O on a space map M in a three-dimensional coordinate system as shown in FIG.

[0049] In the next step S5, the space map creation unit 35 sets multiple points pi and Pi on the side of the obstacle model O and the side of the vehicle model V along the height direction in the space map M. In the example shown in FIG. 4, the space map creation unit 35 sets multiple points p1 to p5 on the side of the obstacle model O along the height direction of the three-dimensional coordinate system of the space map M. The heights of the points p1 to p5 are H1 to H5. The points pi are set at a density according to the resolution of the sensor unit 10 or image rendering constraints. The space map creation unit 35 sets points P1 to P5 where division lines L1 to L5, which are extended horizontally from the points p1 to p5 as starting points, intersect with the vehicle model V, and divides the side of the vehicle model V into multiple regions at different heights. In the next step S6, distances d1 to d5 from the points p1 to p5 to the points P1 to P5 located on the side of the vehicle model V are calculated. In the next step S7, the model generation unit 34 generates a three-dimensional image of the vehicle model V and the obstacle model O arranged on the space map M viewed from the virtual viewpoint P.

[0050] The program proceeds to step S8, but steps S9 to S22 are executed until distance determination is completed for all points p i located on the side of the obstacle model O. In step S8, the control unit 31 sets 1 to a loop counter i. The total number of points p i located on the side of the obstacle model O in the height direction is set to n, where n is an integer equal to or greater than 1. In the next step S9, the identification unit 36 ​​selects the i-th point p i from the multiple points p1 to pn located on the side of the obstacle model O, and determines whether the distance di from this point pi to point Pi located on the side of the vehicle model V is less than a first threshold. If the determination result is Yes (less than the first threshold), the program proceeds to step S10, and if the determination result is No (equal to or greater than the first threshold), the program proceeds to step S13.

[0051] In step S10, the identification unit 36 ​​identifies, as a first identified area, an area corresponding to the height of point pi located on the side of the obstacle model O and included within a predetermined range centered on point Pi. In the next step S11, the identification unit 36 ​​identifies the width of the first identified area as the width corresponding to the height of point Pi from the width information acquired from the storage unit 32. In the next step S12, the drawing unit 37 draws the portion of the vehicle image 61 corresponding to the first identified area by highlighting it in red as a first highlighted area 63. The program then proceeds to step S22.

[0052] In step S13, the identification unit 36 ​​determines whether the distance di from point pi located on the side of the obstacle model O to point Pi located on the side of the vehicle model V is less than a second threshold (first threshold<second threshold). If the determination result is Yes (less than the second threshold), the program proceeds to step S14, and if the determination result is No (equal to or greater than the second threshold), the program proceeds to step S17.

[0053] In step S14, the identification unit 36 ​​identifies, as a second identified area, an area corresponding to the height of point pi located on the side of the obstacle model O and included within a predetermined range centered on point Pi. In the next step S15, the identification unit 36 ​​identifies the width of the second identified area 64 in the same manner as in step S10. In the next step S16, the drawing unit 37 draws the portion of the vehicle image 61 corresponding to the second identified area, highlighted in orange, as the second highlighted area 64. The program then proceeds to step S22.

[0054] In step S17, the identification unit 36 ​​determines whether the distance di from point pi located on the side of the obstacle model O to point Pi located on the side of the vehicle model V is less than a third threshold (second threshold<third threshold). If the determination result is Yes (less than the third threshold), the program proceeds to step S18, and if the determination result is No (equal to or greater than the third threshold), the program proceeds to step S21.

[0055] In step S18, the specifying unit 36 specifies, as the third specified region, a region corresponding to the height of the point pi located on the side surface of the obstacle model O and included within a predetermined range centered on the point Pi. In the next step S19, the specifying unit 36 specifies the width of the third specified region 65 in the same manner as in step S10. In the next step S20, the drawing unit 37 draws, in yellow and with emphasis, as the third emphasized region 65, the part of the vehicle image 61 corresponding to the third specified region. Thereafter, the program proceeds to step S22.

[0056] Step S21 is a step that proceeds when a region corresponding to the height of the point pi located on the side surface of the obstacle model O has not been specified as a specified region. In this step S21, the drawing unit 37 draws the vehicle image 61 without emphasizing the part corresponding to the region (that is, the region corresponding to the height of the point pi located on the side surface of the obstacle model).

[0057] In step S22, the control unit 31 determines whether distance determination has been performed for all the points p1 to pn located on the side surface of the obstacle model O, more specifically, determines whether i≥n is satisfied. If the determination result is Yes (i < n: distance determination has not been completed for all the points p1 to pn), the program proceeds to step S23, increments the loop counter i (i = i + 1), proceeds to step S9, and repeats the steps of S9 to S22 for the next point pi. On the other hand, if the determination result is No (i≥n: distance determination for all the points p1 to pn has been completed), the program proceeds to step S24.

[0058] In the last step S24, the display control unit 38 controls the display unit 40 to display, on the display surface, the three-dimensional image 60 as shown in each of the figures of FIG. 5 drawn by the drawing unit 37. Thus, the operations shown in the flowcharts of FIGS. 8A to 8C are completed.

[0059] (Second Embodiment) An image generation system 100 including an image generation device 30 according to a second embodiment of the present disclosure has the same basic configuration as the image generation system 100 according to the first embodiment shown in Fig. 1, except that the method for identifying a specific region by the identification unit 36 ​​is different. Therefore, the following mainly describes the functions of the second embodiment that differ from those of the first embodiment.

[0060] In the second embodiment, the space map creation unit 35 also divides the side of the vehicle model V (the surface facing the obstacle model O) in the height direction within the space map M into multiple parts at different heights based on the vehicle shape data, as indicated by the dashed dotted lines in Figure 6. The space map creation unit 35 calculates distances di from multiple points pi located on the side of the obstacle model O to multiple points Pi located on the side of the vehicle model V.

[0061] The identification unit 36 ​​of the second embodiment identifies the height and width directions of the identified region as follows: Based on the distance di calculated by the spatial map creation unit 35, the identification unit 36 ​​determines whether the distance di from a point pi located on the side of the obstacle model O in the spatial map M to a point Pi located on the side of the vehicle model V is less than a threshold value. The identification unit 36 ​​identifies the region obtained by connecting the point Pi, whose distance di in the height direction is less than the threshold value, and the adjacent points Pi+1 and / or Pi-1 along the contour of the vehicle model V as a identified region that may come into contact with an obstacle.

[0062] For example, in FIG. 6, assume that the distance d1 between p1 and P1, the distance d2 between p2 and P2, and the distance d3 between p3 and P3 are all less than the threshold. In this case, the identification unit 36 ​​identifies the area a1 connecting points P1 and P2, the area a2 connecting points P2 and P3, and the area a3 connecting point P3 and its adjacent point P4 as identified areas. The identification unit 36 ​​determines the width of the identified area based on the width of the obstacle 2 calculated from the obstacle shape data, with the width corresponding to the width direction of the obstacle. In other words, the identification unit 36 ​​limits the width of the identified area according to the length of the obstacle O in the width direction. The width direction length of the obstacle O is calculated for each different height, as if scanning the width in the height direction. An example of the width of the obstacle 2 calculated from the obstacle shape data is the value calculated by calculating the distance between points forming the obstacle model O for each different height in the width direction of the vehicle model V in the spatial map M. The distance between the points forming the obstacle model O may be calculated for each height at which point pi is located, or may be the average value obtained by averaging the width of the area obtained by connecting point pi and point pi+1 or point pi-1 along the contour of the obstacle model O, or may be the maximum value of the width of the area obtained by connecting point pi and point pi+1 or point pi-1 along the contour of the obstacle model O.

[0063] The drawing unit 37 draws, in the vehicle image 61, portions corresponding to the specific regions a1 to a3 identified by the identification unit 36, with emphasis. FIG. 6 shows a three-dimensional image 60 drawn by the drawing unit 37 in the second embodiment. FIG. 6(a) is a three-dimensional image when the obstacle is a traffic cone. The identification unit 36 ​​identifies the widths of the first to third specific regions according to the width of the traffic cone, and the drawing unit 37 draws the first to third highlighted regions 63 to 65 so that their overall shape is a triangle. FIG. 6(b) is a three-dimensional image when the obstacle is a bicycle wheel. In this case, the drawing unit 37 draws the first to third highlighted regions 63 to 65 as narrow rectangular shapes overall. FIG. 6(c) is a three-dimensional image when the obstacle is a post. In this case, the drawing unit 37 draws the first to third highlighted regions 63 to 65 so that their overall shape is approximately T-shaped.

[0064] The operation of the image generating device 30 of the second embodiment is basically the same as the operation shown in the flowcharts of Figures 8A to 8C. When identifying the height directions of the first to third identified regions in steps S10, S14, and S18, the identification unit 36 ​​of the image generating device 30 of the second embodiment identifies, as the identified region, a region connecting point Pi, for which the distance di is determined to be less than a predetermined threshold, with adjacent points Pi+1 and / or Pi-1. When identifying the width directions of the first to third identified regions in steps S11, S15, and S19, the identification unit 36 ​​identifies the identified region by the width of obstacle 2, which corresponds to the height of point Pi, among the widths of the obstacles calculated based on the obstacle shape data.

[0065] As described above, the image generation device 30 according to each of the above embodiments of the present disclosure is an image generation device that generates a vehicle image 61 as if the vehicle 1 were viewed from a viewpoint located outside the vehicle 1. The image generation device 30 includes an acquisition unit 33 that acquires the position of an obstacle 2 relative to the vehicle 1 and the three-dimensional shape of the obstacle 2, an identification unit 36 ​​that identifies an area of ​​the vehicle 1 that may come into contact with the obstacle 2 based on the pre-stored shape of the vehicle 1 and the position and shape of the obstacle 2, and a drawing unit 37 that generates a vehicle image 61 as if the vehicle 1 were viewed from a viewpoint located outside the vehicle 1 and draws the area identified by the identification unit 36 ​​in the vehicle image 61 in an emphasized manner. The identification unit 36 ​​identifies an area that may come into contact with the obstacle 2 from among a plurality of areas obtained by dividing the vehicle 1 at different heights in the height direction, based on the shape of the vehicle 1 and the position and shape of the obstacle 2.

[0066] The image generation system 100 according to each of the above-described embodiments of the present disclosure is a system including the image generation device 30. An image generation method according to each of the above-described embodiments of the present disclosure is a method executed by the control unit 31 of the image generation device 30 according to each of the above-described embodiments. This image generation method includes an acquisition step of acquiring the position of an obstacle 2 relative to the vehicle 1 and the three-dimensional shape of the obstacle 2; an identification step of identifying an area of ​​the vehicle 1 that may come into contact with the obstacle based on the pre-stored shape of the vehicle 1 and the position and shape of the obstacle 2; and a drawing step of generating a vehicle image 61 as if the vehicle 1 were viewed from a viewpoint outside the vehicle 1, and drawing the area identified by the identification step in the vehicle image 61 with emphasis. The identification step identifies an area that may come into contact with the obstacle from among multiple areas obtained by dividing the vehicle 1 into different heights in the height direction, based on the shape of the vehicle 1 and the position and shape of the obstacle 2. An image generation program according to each embodiment of the present disclosure is a program for causing a computer to execute each step of the image generation method, and causes the computer to execute an acquisition step of acquiring the position of an obstacle 2 relative to the vehicle 1 and the three-dimensional shape of the obstacle 2, an identification step of identifying an area of ​​the vehicle 1 that may come into contact with the obstacle based on the pre-stored shape of the vehicle 1 and the position and shape of the obstacle 2, and a drawing step of generating a vehicle image 61 as if the vehicle 1 were viewed from a viewpoint outside the vehicle 1 and depicting the area identified by the identification step in the vehicle image 61 with emphasis. The identification step identifies an area that may come into contact with the obstacle based on the shape of the vehicle 1 and the position and shape of the obstacle 2, from among multiple areas obtained by dividing the vehicle 1 into different heights in the height direction.

[0067] Therefore, the image generation device 30, image generation system 100, image generation method and program of each embodiment highlight areas that may come into contact with an obstacle, and present a vehicle image 61 that is drawn as if the vehicle 1 were being viewed from a viewpoint outside the vehicle 1, thereby enabling the driver to more accurately understand areas of the vehicle 1 that may come into contact with an obstacle.

[0068] Depending on the type of vehicle 1, the vehicle height may be freely changed, and the image generation device 30 etc. of each embodiment can appropriately identify the area of ​​the vehicle 1 that may come into contact with the obstacle 2 according to the vehicle height. Therefore, when the vehicle 1 can overcome the obstacle 2 by adjusting the vehicle height or the like and there is no possibility of contact, the vehicle image 61 is not drawn with emphasis. Therefore, when there is a possibility of contact with the obstacle 2, the image generation device 30 etc. of each embodiment warns the driver by drawing the area with emphasis, but when there is no possibility of contact, the vehicle image 61 is not drawn with emphasis, so unnecessary warnings are not given.

[0069] The identification unit 36 ​​of the second embodiment identifies an area that may come into contact with the obstacle, with a width corresponding to the width direction of the obstacle 2, from among a plurality of areas obtained by dividing the vehicle 1 at different heights, based on the width of the obstacle 2 calculated from the shape of the obstacle 2. With this configuration, the drawing unit 37 draws the vehicle image 61 by emphasizing the area that may come into contact with the obstacle with a width corresponding to the width direction of the obstacle 2. Therefore, the image generation device 30 etc. according to another different embodiment can allow the driver to more accurately understand both the width and height of the area where the vehicle 1 is likely to come into contact with the obstacle 2.

[0070] When the distance di from point pi located on the side of the obstacle 2 to point Pi located on the side of the vehicle 1 based on the shape of the obstacle 2 is less than a threshold value, the identification unit 36 ​​in each of the above embodiments identifies, among multiple regions obtained by dividing the vehicle 1 at different heights, a region corresponding to the height of point pi located on the side of the obstacle 2 as a region where there is a possibility of contact with the obstacle. With this configuration, the image generation device 30 etc. according to each embodiment can more appropriately identify a region where there is a possibility of contact with an obstacle, and enable the driver to more accurately understand the region of the vehicle 1 where there is a possibility of contact with the obstacle 2.

[0071] The identification unit 36 ​​in each of the above embodiments identifies areas of the vehicle 1 that are likely to come into contact with the obstacle and areas that are unlikely to come into contact with the obstacle, based on the shape of the vehicle 1 and the position and shape of the obstacle 2. The rendering unit 37 highlights the areas of the vehicle 1 that are likely to come into contact with the obstacle and the areas of the vehicle 1 that are unlikely to come into contact with the obstacle in different ways. With this configuration, the image generation device 30, etc., according to each of the above embodiments can specifically show areas of the obstacle 2 that are close to the vehicle 1 and areas that are far from the vehicle 1, depending on the shape of the obstacle 2, even if the width of the obstacle 2 varies depending on its height. As a result, the image generation device 30, etc., can allow the driver to more accurately understand the areas of the vehicle 1 that are likely to come into contact with the obstacle 2.

[0072] The identification unit 36 ​​in each of the above embodiments identifies an area corresponding to the height of point pi on the side of the obstacle 2 as an area where the vehicle 1 is likely to come into contact with the obstacle when the distance di from point pi on the side of the obstacle 2, which is included in the shape of the obstacle 2, to point Pi on the side of the vehicle 1 is less than a first threshold value. Furthermore, when the distance di from point pi on the side of the obstacle 2, which is included in the shape of the obstacle 2, to point Pi on the side of the vehicle 1 is equal to or greater than the first threshold value and less than a second threshold value, the identification unit 36 ​​identifies an area corresponding to the height of point pi on the side of the obstacle 2 as an area where the vehicle 1 is unlikely to come into contact with the obstacle. With this configuration, the image generation device 30 and the like according to each embodiment can present areas where the vehicle 1 is likely to come into contact with the obstacle 2 by emphasizing them in different ways depending on the distance between the obstacle 2 and the vehicle 1. As a result, the image generation device 30 and the like can allow the driver to more accurately understand areas of the vehicle 1 where the vehicle 1 is likely to come into contact with the obstacle 2.

[0073] The image generation device 30 of each of the above embodiments includes a model generation unit 34 that generates a vehicle model V that shows the vehicle in three dimensions based on the shape of the vehicle 1. The drawing unit 37 draws a vehicle image 61 based on the vehicle model V, and draws and emphasizes portions of the vehicle image 61 that correspond to the areas identified by the identification unit 36 ​​(first to third emphasized areas 63 to 65). The model generation unit 34 generates an obstacle model O that shows the obstacle 2 in three dimensions based on the shape of the obstacle 2. The drawing unit 37 draws an obstacle image 62 based on the obstacle model O. By drawing both the vehicle image 61 and the obstacle image 62 in this manner, the image generation device 30 and the like according to each embodiment can allow the driver to more accurately understand areas of the vehicle 1 that may come into contact with the obstacle 2, and can also make it easier for the driver to understand information about their surroundings.

[0074] Although the embodiments of the present disclosure have been described in detail above with reference to the drawings, the specific configurations are not limited to these embodiments, and design changes that do not deviate from the gist of the present disclosure are included in the present disclosure.

[0075] In each of the above embodiments, the space map creation unit 35 calculates the distance di from a point pi on the side of the obstacle 2 to a point Pi on the side of the vehicle 1, and identifies an area where there is a possibility of contact with the obstacle based on the calculated distance di, but this is not limiting. In another different embodiment, the space map creation unit 35 moves the polygons of the vehicle model V (see FIG. 3) within the space map M toward the obstacle model O. The space map creation unit 35 moves each polygon surface forming the vehicle model V along the direction of the normal vector. The space map creation unit 35 identifies a portion where the polygon surface comes into contact with the obstacle model O as an area where there is a possibility of contact with the obstacle. The drawing unit 37 draws the portion in an emphasized manner.

[0076] The space map creation unit 35 can also determine the likelihood of contact with an obstacle according to the order in which the polygon surface contacts the obstacle model O, and identify first to third specific areas according to the likelihood of contact. The space map creation unit 35 can also determine the amount of movement of the polygon when it contacts the obstacle model O as the distance di from point pi on the side of the obstacle 2 to point P on the side of the vehicle 1, and identify the first to third contact areas according to this distance di. [Explanation of symbols]

[0077] 1: vehicle, 2: obstacle, 30: image generation device, 33: acquisition unit, 34: model generation unit, 36: identification unit, 37: drawing unit, 61: vehicle image, V: vehicle model

Claims

1. An image generating device that generates a vehicle image as if the vehicle were viewed from a viewpoint located outside the vehicle, an acquisition unit that acquires a position of an obstacle relative to the vehicle and a three-dimensional shape of the obstacle; an identification unit that identifies an area of ​​the vehicle that may come into contact with the obstacle based on a pre-stored shape of the vehicle and a position and shape of the obstacle; a rendering unit that generates an image of the vehicle as if the vehicle were viewed from a viewpoint outside the vehicle, and renders the vehicle image in an emphasized manner the area identified by the identifying unit; The identification unit identifies an area that may come into contact with the obstacle among multiple areas obtained by dividing the vehicle at different heights in the vertical direction based on the shape of the vehicle and the position and shape of the obstacle.

2. 2. The image generating device according to claim 1, wherein the identification unit identifies an area that may come into contact with the obstacle, among a plurality of areas obtained by dividing the vehicle at different heights, based on the width of the obstacle calculated from the shape of the obstacle, with a width corresponding to a width direction of the obstacle.

3. 2. The image generation device according to claim 1, wherein when a distance from a point located on a side of the obstacle to a point located on a side of the vehicle based on the shape of the obstacle is less than a threshold, the identification unit identifies an area of ​​the vehicle that corresponds to the height of the point located on the side of the obstacle as an area that may come into contact with the obstacle.

4. the identification unit identifies, among the identified areas where the vehicle may come into contact with the obstacle, areas that are highly likely to come into contact with the obstacle and areas that are low likely to come into contact with the obstacle, based on the shape of the vehicle and the position and shape of the obstacle; 2. The image generating device according to claim 1, wherein the drawing unit emphasizes in different ways an area of ​​the vehicle that is likely to come into contact with the obstacle and an area of ​​the vehicle that is unlikely to come into contact with the obstacle.

5. the identification unit, when a distance from a point on a side surface of the obstacle that is included in the shape of the obstacle to a point on a side surface of the vehicle is less than a first threshold, identifies an area corresponding to a height of the point on the side surface of the obstacle as an area that is likely to come into contact with the obstacle; 5. The image generation device according to claim 4, wherein when a distance from a point on a side of the obstacle included in the shape of the obstacle to a point on a side of the vehicle is equal to or greater than the first threshold and less than the second threshold, an area corresponding to the height of the point on the side of the obstacle is identified as an area where there is a low possibility of contact with the obstacle.

6. a model generation unit that generates a vehicle model that shows the vehicle in three dimensions based on the shape of the vehicle; 2. The image generating device according to claim 1, wherein the drawing unit draws the vehicle image based on the vehicle model, and draws the vehicle image by emphasizing a portion of the vehicle image that corresponds to the area identified by the identification unit.

7. A program executed by a computer, an acquisition step of acquiring the position of an obstacle relative to the vehicle and the three-dimensional shape of the obstacle in the computer; a specifying step of specifying an area of ​​the vehicle that may come into contact with the obstacle based on the pre-stored shape of the vehicle and the position and shape of the obstacle; a drawing step of generating an image of the vehicle as if the vehicle were viewed from a viewpoint outside the vehicle, and drawing the area of ​​the vehicle image identified by the identifying step in an emphasized manner, The identification process is a program characterized by identifying an area that may come into contact with the obstacle among multiple areas obtained by dividing the vehicle at different heights in the vertical direction based on the shape of the vehicle and the position and shape of the obstacle.

Images