Display control device and display control program

Abstract

To make a distance from a vehicle to an obstacle easily visible to a driver.SOLUTION: A display control device 100 includes: an image generation unit 132 that, based on a picked-up image PC obtained by picking up an image of the surroundings of a vehicle 1, generates a three-dimensional image PP obtained by viewing the vehicle 1 from a virtual camera VC located outside the vehicle 1; a vehicle drawing unit 133 that draws, on the three-dimensional image PP, a vehicle image PV showing the vehicle 1 using a three-dimensional model VM showing the vehicle 1; an obstacle detection unit 134 that detects a shape of an obstacle T and a distance LD in the surroundings of the vehicle 1; and an obstacle drawing unit 135 that, based on the shape of the obstacle T and the distance LD, generates a three-dimensional model TM showing the obstacle T and draws an obstacle image on the three-dimensional image PP. The obstacle drawing unit 135 draws the obstacle image by replacing it with a bounding box BB surrounding the obstacle T. A surface SP of the bounding box BB faces the three-dimensional model VM showing the vehicle 1.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display control device and a display control program. [Background technology]

[0002] Conventionally, there are known techniques for displaying images of the surroundings of a vehicle. For example, Patent Document 1 describes a technique in which real images of an environmental area are captured by multiple real cameras on a vehicle, and an image is generated from these real images. The technique also describes that the real images at least partially display the environmental area, and that the images are displayed from the viewpoints of virtual cameras arranged within the environmental area. The technique also describes that the images are generated in a bowl shape, and that the distance to the vehicle is represented within the virtual bowl shape by landmarks displayed within the images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2021-513152 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology described in Patent Document 1, it was sometimes difficult to visually recognize the distance from the vehicle to an obstacle. An object of the present invention is to provide a display control device and a display control program that enable a driver to easily visually recognize the distance from the vehicle to an obstacle. [Means for solving the problem]

[0005] In order to achieve the above object, for example, the display control device of this embodiment is a display control device that controls an image to be displayed on a display, and includes: an image generation unit that converts the viewpoint of an image taken of the surroundings of the vehicle to generate a 3D image of the vehicle viewed from a viewpoint located outside the vehicle; a vehicle drawing unit that draws an image of the vehicle in the 3D image based on a 3D model representing the vehicle; an obstacle detection unit that detects the shape and distance of an obstacle around the vehicle; and an obstacle drawing unit that generates a 3D model representing the obstacle based on the shape and distance of the obstacle and draws the obstacle in the 3D image, and the obstacle drawing unit replaces the obstacle image with a bounding box surrounding the obstacle based on the shape and distance of the obstacle and draws it, and one side of the bounding box faces the 3D model representing the vehicle. [Effects of the Invention]

[0006] According to the display control device and the display control program of the present invention, the driver can easily visually recognize the distance from the vehicle to an obstacle. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an in-vehicle device including a display control device. [Figure 2] FIG. 2 is a diagram illustrating an example of a method for generating a three-dimensional image by an image generating unit. [Figure 3] FIG. 10 is a diagram illustrating an example of the relationship between a three-dimensional model representing an obstacle and a bounding box. [Figure 4] FIG. 10 is a screen diagram showing an example of a surroundings display screen that displays a three-dimensional model showing an obstacle. [Figure 5] FIG. 10 is a screen diagram showing an example of a surroundings display screen that displays a three-dimensional model showing an obstacle. [Figure 6] FIG. 10 is a screen diagram showing another example of a surroundings display screen that displays a bounding box in a line format. [Figure 7] FIG. 10 is a screen diagram showing another example of a surroundings display screen that displays a bounding box in a plane manner. [Figure 8] 10 is a flowchart illustrating an example of processing by a display control device. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the configuration of an in-vehicle device 3 including a display control device 100. The in-vehicle device 3 is mounted on a vehicle 1. The in-vehicle device 3 includes a detection unit 20, an operation unit 50, a display unit 60, and the display control device 100.

[0009] The detection unit 20 captures an image of the surroundings of the vehicle 1 and detects the shape of an obstacle T and the distance LD from the vehicle 1 to the obstacle T. The detection unit 20 includes an imaging unit 30 and a distance sensor 40. The obstacle T is an object that exists around the vehicle 1. Examples of the obstacle T include a person, a bicycle, a safety cone, a tree, etc.

[0010] The photographing unit 30 photographs images of the surroundings of the vehicle 1. The photographing unit 30 includes a front camera 31 that photographs the area in front of the vehicle 1, a rear camera 33 that photographs the area behind the vehicle 1, a left side camera 35 that photographs the area to the left of the vehicle 1, and a right side camera 37 that photographs the area to the right of the vehicle 1. Each of these cameras includes an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor), and a data processing circuit that generates a photographed image PC from the image sensor.

[0011] The photographing unit 30 adjusts the angle of view of each of the cameras facing in four directions (i.e., forward, backward, left side, and right side) so that the cameras can photograph a 360° range centered on the vehicle 1. Each of the front camera 31, rear camera 33, left side camera 35, and right side camera 37 photographs the photographing range at a predetermined frame rate to generate a photographed image PC. The photographing unit 30 outputs the generated photographed image PC to the display control device 100. The display control device 100 stores the input photographed image PC in memory 140. Note that each of the front camera 31, rear camera 33, left side camera 35, and right side camera 37 may be composed of a single camera or multiple cameras. The photographed image PC corresponds to an example of an "image photographed of the surroundings of the vehicle."

[0012] The distance sensor 40 detects the distance LD from the vehicle 1 to the obstacle T. The distance sensor 40 is provided with a LiDAR (Light Detection and Ranging) at a position on the body of the vehicle 1 where it can sense the front, rear, left and right sides, etc., and acquires point cloud data DG around the vehicle 1. Each piece of point data constituting the point cloud data DG indicates the distance to an object (e.g., obstacle T) that exists within a range of a predetermined distance from the vehicle 1. The point cloud data DG indicates the shape of an object (e.g., obstacle T) that exists within a range of a predetermined distance from the vehicle 1.

[0013] In this embodiment, a case will be described in which the distance sensor 40 is a LiDAR, but this is not limiting. The distance sensor 40 may be, for example, a 3D (Three Dimensions) camera. The number of LiDARs is not limited. The number of LiDARs may be one or more.

[0014] The operation unit 50 accepts operations from a user in the vehicle 1. The user is, for example, a driver. The operation unit 50 outputs an operation signal corresponding to the accepted operation to the display control device 100. The operations accepted by the operation unit 50 include, for example, an operation to instruct the start of image display processing and an operation to end image display processing. The operation unit 50 is equipped with, for example, an on switch (not shown) and an off switch (not shown), and when the on switch is pressed, the display control device 100 accepts an operation to instruct the start of image display processing. When the off switch is pressed while image display processing is being executed, the display control device 100 accepts an operation to end image display processing.

[0015] The "image display process" is a process in which the display control device 100 displays a surroundings display screen, which is an image including, for example, a vehicle image PV, a line image SD, a plane image PS, and a three-dimensional image PP, on the display 61. In other words, the display control device 100 controls the image to be displayed on the display 61. The "image display process" will be described with reference to FIG. 1, and will be further described with reference to FIGS. 2 to 8.

[0016] The display unit 60 includes a display 61 and a touch sensor 63. A liquid crystal display, an organic EL display, or the like is used for the display 61. The display unit 60 displays a peripheral display screen (described later) on the display 61 based on display data input from the display control device 100. A resistive film type, a capacitive type, or the like sensor is used for the touch sensor 63. The touch sensor 63 is disposed on the display surface of the display 61. The display unit 60 detects a touch operation of a user's finger on the display 61 using the touch sensor 63, and generates a position signal indicating the operation position of the detected touch operation. In other words, the display 61 and the touch sensor 63 constitute a so-called "touch panel."

[0017] The display control device 100 is a computer including a processor 130 such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit), and a memory 140 such as a ROM (Read Only Memory) or RAM (Random Access Memory). In addition to these devices, the display control device 100 also includes a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), an interface circuit for connecting sensors, peripheral devices, etc., and an in-vehicle network communication circuit for communicating with other in-vehicle devices via an in-vehicle network. The display control device 100 realizes various functional configurations by having the processor 130 execute a control program PGM stored in the memory 140. The control program PGM corresponds to an example of a "display control program."

[0018] The display control device 100 includes, for example, as functional units, an image acquisition unit 131, an image generation unit 132, a vehicle drawing unit 133, an obstacle detection unit 134, an obstacle drawing unit 135, and an image storage unit 141. Specifically, the processor 130 executes the control program PGM stored in the memory 140, whereby the processor 130 functions as the image acquisition unit 131, the image generation unit 132, the vehicle drawing unit 133, the obstacle detection unit 134, and the obstacle drawing unit 135. The processor 130 executes the control program PGM stored in the memory 140, whereby the memory 140 functions as the image storage unit 141.

[0019] The image storage unit 141 stores the captured image PC. The captured image PC is generated by the photographing unit 30. The image storage unit 141 stores a three-dimensional model VM that represents the vehicle 1 using a single polygon or a combination of multiple polygons made up of lines connecting the coordinates of three or more points. The three-dimensional model VM of the vehicle 1 is read by the vehicle drawing unit 133 when generating the vehicle image PV. The vehicle drawing unit 133 generates the vehicle image PV, which is an image of the three-dimensional model VM of the vehicle 1 viewed from the viewpoint of a virtual camera VC, which will be described later. The vehicle image PV will be further described with reference to FIG. 3. Note that the three-dimensional model is not limited to three-dimensional polygons, and may be an image in which the vehicle is drawn in a planar manner.

[0020] The image storage unit 141 stores point cloud data DG. The point cloud data DG is generated by the distance sensor 40. The point cloud data DG is read by the obstacle drawing unit 135 when generating an image of an obstacle T, and an obstacle image PT representing the obstacle T is generated from the point cloud data DG. The obstacle image PT will be further described with reference to FIG. 3.

[0021] The image acquisition unit 131 acquires the photographed image PC from the photographing unit 30. The image generation unit 132 converts the viewpoint of the captured image PC to generate a three-dimensional image PP in which the vehicle 1 is viewed from a viewpoint located outside the vehicle 1. The three-dimensional image PP will be further described with reference to FIG.

[0022] The vehicle drawing unit 133 draws a vehicle image PV showing the vehicle 1 on the three-dimensional image PP generated by the image generation unit 132, using the three-dimensional model VM showing the vehicle 1. The vehicle image PV will be further described with reference to FIG. 3.

[0023] The obstacle detection unit 134 detects the shape and distance of an obstacle T around the vehicle 1. The obstacle detection unit 134 acquires point cloud data DG from, for example, the distance sensor 40. The shape and distance of the obstacle T refer to the shape of the obstacle T and the distance from the vehicle 1 to the obstacle T.

[0024] The obstacle drawing unit 135 generates a 3D model TM representing the obstacle T based on the shape and distance of the obstacle T. The obstacle drawing unit 135 draws an obstacle image, which is an image of the 3D model TM representing the obstacle T viewed from a viewpoint located outside the vehicle, in the 3D image PP generated by the image generation unit 132 based on the 3D model TM and the position of the virtual camera VC. An example of the 3D model TM is a polygon, which is formed by multiple edges connecting points included in the point cloud data GD. The obstacle drawing unit 135 draws the obstacle image by replacing it with a bounding box BB surrounding the obstacle T based on the shape and distance LD of the obstacle T. The bounding box BB refers to a rectangular parallelepiped that represents the area of the target object (here, the obstacle T) when the area of the target object is surrounded by the smallest rectangular parallelepiped and separated by a boundary from the external area. Note that one side SP of the bounding box BB faces the 3D model VM representing the vehicle 1. In other words, the direction of the normal vector of the surface SP is parallel to the traveling direction DT of the vehicle 1. The bounding box BB and the surface SP will be further described with reference to Figures 3 and 4. The obstacle rendering unit 135 displays a three-dimensional image PP including the bounding box BB on the display 61.

[0025] When the area of an area AR, which will be described later, is equal to or greater than a first threshold value TH1, the obstacle drawing unit 135 draws the bounding box BB as a line image SD indicating the multiple sides that make up the bounding box BB. The line image SD, which is the multiple sides that make up the bounding box BB, will be further described with reference to FIGS. 4 and 7.

[0026] The obstacle drawing unit 135 draws the bounding box BB in a line manner AL, which draws a line image SD indicating the multiple sides that make up the bounding box BB, or in a surface manner AS, which draws the surfaces of the bounding box BB as a surface image PS with a predetermined transparency. As the transparency increases, the surface image PS approaches transparency, and as the transparency decreases, the surface image PS becomes opaque. The predetermined transparency is, for example, 20% to 90%. The surface image PS will be further described with reference to FIG. 6.

[0027] When the distance LD between the obstacle T and the vehicle 1 is equal to or greater than the second threshold value TH2, the obstacle drawing unit 135 draws a bounding box BB with a line pattern AL, and when the distance LD between the obstacle T and the vehicle 1 is less than the second threshold value TH2, the obstacle drawing unit 135 draws a bounding box BB with a surface pattern AS.

[0028] The obstacle drawing unit 135, in the line pattern AL, draws, as a line image SD, the edges that constitute the bounding box BB and that surround the surface SP that faces the three-dimensional model VM representing the vehicle 1, in an emphasized manner. Methods for emphasizing the edges include, for example, the color of the line image SD representing the edge, the transparency of the line image SD representing the edge, the blinking display of the line image SD representing the edge, and the blinking cycle of the blinking display of the line image SD representing the edge.

[0029] In the surface aspect AS, the obstacle drawing unit 135 draws one surface SP that faces the vehicle 1, among the surfaces that make up the bounding box BB, as a surface image PS with a predetermined non-transparent transparency. The predetermined transparency is, for example, 20% to 90%.

[0030] An example of a method for generating a three-dimensional image PP will be described with reference to FIG. 2. FIG. 2 is a diagram illustrating an example of a method for generating a three-dimensional image PP by the image generation unit 132. As shown in FIG. 2, the image generation unit 132 places a hemispherical projection surface 7 in a virtual three-dimensional space. The virtual three-dimensional space is a virtual three-dimensional space in which the projection surface 7, a three-dimensional model VM representing the vehicle 1, and a virtual camera VC are placed in order to generate the three-dimensional image PP. An X-axis, a Y-axis, and a Z-axis, which are mutually orthogonal, are set on the projection surface 7. The three-dimensional model VM representing the vehicle 1 is placed on the bottom surface of the projection surface 7. The Z-axis is parallel to the vertical direction in the virtual three-dimensional space. The X-axis and the Y-axis are parallel to the horizontal direction in the virtual three-dimensional space. The X-axis is parallel to the long axis direction (longitudinal direction) of the vehicle body of the three-dimensional model VM. The Y-axis is parallel to the short axis direction (short-side direction) of the vehicle body of the three-dimensional model VM. The positive direction of the Z-axis is the upward direction in the virtual three-dimensional space. The positive direction of the X axis is the forward direction of the body of the 3D model VM and the rightward direction in the virtual 3D space. The positive direction of the Y axis is the rightward direction of the body of the 3D model VM and the downward direction in the virtual 3D space. Note that the X axis, Y axis, and Z axis are also shown in Figures 3 to 7.

[0031] The virtual camera VC is, for example, located outside the vehicle 1 in the virtual three-dimensional space and around the vehicle 1. The virtual camera VC corresponds to an example of a "viewpoint located outside the vehicle." The virtual camera VC is a camera installed in the virtual three-dimensional space at a position that does not correspond to the position of the camera in the physical space. The projection surface 7 is composed of multiple divided projection surfaces 72. The divided projection surfaces 72 include a first divided projection surface 721, a second divided projection surface 722, a third divided projection surface 723, a fourth divided projection surface 724, a fifth divided projection surface 725, and a sixth divided projection surface 726. The first divided projection surface 721 to the sixth divided projection surface 726 are located to the left of the three-dimensional model VM of the vehicle 1 with respect to the center plane LC of the projection surface 7. The center plane LC includes the major axis of the three-dimensional model VM and is a plane parallel to the XZ plane.

[0032] The first to sixth divided projection surfaces 721 to 726 are each disposed at 30-degree intervals with respect to the center plane LC. For example, the first divided projection surface 721 corresponds to a range of 0 to 30 degrees with respect to the center plane LC. The second divided projection surface 722 corresponds to a range of 30 to 60 degrees with respect to the center plane LC. The third divided projection surface 723 corresponds to a range of 60 to 90 degrees with respect to the center plane LC. The fourth divided projection surface 724 corresponds to a range of 90 to 120 degrees with respect to the center plane LC. The fifth divided projection surface 725 corresponds to a range of 120 to 150 degrees with respect to the center plane LC. The sixth divided projection surface 726 corresponds to a range of 150 to 180 degrees with respect to the center plane LC.

[0033] On the first divided projection surface 721 and the second divided projection surface 722, for example, the left half of the image PC captured by the front camera 31 is arranged. On the third divided projection surface 723 and the fourth divided projection surface 724, for example, the image PC captured by the left side camera 35 is arranged. On the fifth divided projection surface 725 and the sixth divided projection surface 726, for example, the left half of the image PC captured by the rear camera 33 is arranged. In this manner, the captured image PC is arranged on the first divided projection surface 721 to the sixth divided projection surface 726. When the captured image PC is arranged on the divided projection surfaces 72, the captured image PC is deformed so as to fit the shape of the divided projection surfaces 72. Similarly, on the six divided projection surfaces 72 arranged to the right of the three-dimensional model VM with respect to the center plane LC of the projection surface 7, the right half of the image PC captured by the front camera 31, the image PC captured by the right side camera 37, and the right half of the image PC captured by the rear camera 33 are arranged.

[0034] In this way, the image generator 132 generates a three-dimensional image PP by generating an image as if the projection surface 7 were viewed from the viewpoint of the virtual camera VC with the images PC captured by the front camera 31, rear camera 33, left side camera 35, and right side camera 37 arranged on the projection surface 7. Note that the method of generating the three-dimensional image PP is not limited to this.

[0035] The processing of the obstacle rendering unit 135 will be described with reference to FIGS. 3 to 7. FIG. 3 is a diagram showing an example of the relationship between a 3D model TM representing an obstacle T and a bounding box BB. FIG. 3 illustrates a case where the obstacle T is a tree. The obstacle T is located behind the vehicle 1. An arrow DT indicates the traveling direction of the vehicle 1. The vehicle 1 moves backward as indicated by the arrow DT. The backward direction is the negative direction of the X-axis. The obstacle T is composed of trees T1, T2, and T3. The trees T1, T2, and T3 are arranged along the Y-axis direction. In FIG. 3, the obstacle T is rendered as an obstacle image, which is an image of the 3D model TM representing the obstacle T viewed from a viewpoint located outside the vehicle 1. The obstacle rendering unit 135 generates the 3D model TM representing the obstacle T based on point cloud data DG from the distance sensor 40. The point cloud data DG indicates the shape of the obstacle T and the distance LD from the vehicle 1 to the obstacle T. The obstacle drawing unit 135 draws an obstacle image PT based on a three-dimensional model TM representing the obstacle T.

[0036] The bounding box BB is a rectangular parallelepiped that surrounds the obstacle T. One side SP of the bounding box BB faces the three-dimensional model VM that represents the vehicle 1. In other words, the bounding box BB is a rectangular parallelepiped that includes the side SP that faces the three-dimensional model VM that represents the vehicle 1 and surrounds the obstacle T. The side SP is parallel to the Y-axis and Z-axis. In other words, the side SP is parallel to the YZ plane. In other words, the side SP is perpendicular to the X-axis.

[0037] 4 to 7, a screen on which an obstacle T drawn as a bounding box by the obstacle drawing unit 135 is displayed on the display 61 will be described. FIG. 4 is a screen diagram showing an example of a surroundings display screen 800 in which the 3D model TM representing the obstacle T is displayed on the 3D image as an obstacle image seen from a viewpoint located outside the vehicle 1, because the 3D model TM representing the obstacle T does not overlap the 3D model VM representing the vehicle 1. The arrow DT shown in FIGS. 3 to 7 indicates the traveling direction of the vehicle 1. In the examples of FIGS. 3 to 7, the vehicle 1 is moving backward, which is the direction indicated by the arrow DT.

[0038] 5 depicts an area AR where the three-dimensional model TM representing the obstacle T overlaps with the three-dimensional model VM representing the vehicle 1. In FIG. 5, the area of the area AR is less than a first threshold TH1. The first threshold TH1 is a threshold related to area, and may be, for example, 300 cm 2 Here, assuming that the area of the area AR is area SA, the area SA is the area when the vehicle image PV is displayed at the actual size of the vehicle 1. When the area of the area AR is equal to or greater than the first threshold TH1, the obstacle drawing unit 135 displays the bounding box BB in a line pattern AL or a plane pattern AS on the surroundings display screen, thereby replacing the obstacle image with the bounding box BB and drawing it. This simplifies image processing in the display control device 100, allowing the user to understand the distance between the vehicle 1 and the obstacle T from a simpler screen. Furthermore, when the area SA of the area AR is less than the first threshold TH1, an obstacle image based on a three-dimensional model TM representing the obstacle is drawn on the surroundings display screen. In the examples of FIGS. 4 and 5, the area SA of the area AR is less than the first threshold TH1, so the obstacle drawing unit 135 draws the obstacle T as an obstacle image based on the three-dimensional model TM representing the obstacle.

[0039] FIG. 6 is a screen diagram showing an example of the surroundings display screen 820 displaying a bounding box BB in a line pattern AL. In FIG. 6, the area SA of the region AR is equal to or greater than the first threshold TH1, so the obstacle drawing unit 135 replaces the obstacle image with a bounding box BB on the surroundings display screen 820 and draws it. Here, the distance LD from the vehicle 1 to the obstacle T is the first distance LD1. The first distance LD1 is equal to or greater than the second threshold TH2. The second threshold TH2 is a distance-related threshold, e.g., 5 m. When the distance LD is equal to or greater than the second threshold TH2, the obstacle drawing unit 135 draws the bounding box BB in a line pattern AL. For example, the obstacle drawing unit 135 draws a line image SD, which is a bounding box BB in which, of the edges constituting the bounding box BB, the edges surrounding the surface SP facing the three-dimensional model VM representing the vehicle 1 are emphasized. This display of the bounding box BB as a line image SD is the line pattern AL. In the example of FIG. 4 , line images SD1, SD2, SD3, and SD4 of the line image SD are emphasized. In the line image SD, line images SD1, SD2, SD3, and SD4, which are line images constituting the surface SP facing the three-dimensional model VM, can be emphasized by being displayed thicker than other edges that do not constitute the surface SP facing the three-dimensional model VM. For example, the line images constituting the surface SP facing the three-dimensional model VM can be emphasized by being displayed in a different color from other edges that do not constitute the surface SP facing the three-dimensional model VM. The color of the line image SD may change as the first distance LD1 decreases. For example, the color of the line images constituting the surface SP facing the three-dimensional model VM may change from green to yellow or from yellow to red depending on the first distance LD1.

[0040] FIG. 7 is a screen diagram showing an example of a surroundings display screen 830 that displays a bounding box BB in a surface manner AS. In FIG. 7, because the area SA of the region AR is equal to or greater than the first threshold value TH1, the obstacle drawing unit 135 replaces the obstacle image with a bounding box BB and draws it on the surroundings display screen 830. Here, the distance LD from the vehicle 1 to the obstacle T is a second distance LD2. The second distance LD2 is less than the second threshold value TH2. The second threshold value TH2 is, for example, 5 m. When the distance LD is less than the second threshold value TH2, the obstacle drawing unit 135 displays the bounding box BB in a surface manner AS. For example, the obstacle drawing unit 135 draws the bounding box BB as a surface image PS by highlighting, with a predetermined non-transparent transparency, one surface SP that faces the three-dimensional model VM representing the vehicle 1 among the surfaces that make up the bounding box BB. This display of the bounding box BB as a surface image PS is called the surface manner AS. To make it easier for the user to understand that the bounding box has changed from the line aspect AL to the surface aspect AS, it is preferable that the color of the surface image PS be the same as the color when the distance LD1 is closest to the second threshold TH2. The transparency of the surface image PS may change as the distance LD decreases. For example, the transparency of the surface image PS may be 90% when the distance LD is the second distance LD2, and 50% when the distance LD is a third distance LD3 that is smaller than the second distance LD2.

[0041] In other words, when the distance LD is less than the second threshold TH2, the obstacle rendering unit 135 preferably reduces the transparency of the surface image PS as the distance LD becomes shorter. By reducing the transparency of the surface image PS, the surface image PS is emphasized. Therefore, the user can visually recognize that the distance LD from the vehicle 1 to the obstacle T is becoming shorter.

[0042] The processing of the display control device 100 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the processing of the display control device 100. First, in step S101, the image acquisition unit 131 acquires a captured image PC from the imaging unit 30. In step S103, the image generation unit 132 converts the viewpoint of the captured image PC to generate a three-dimensional image PP in which the vehicle 1 is viewed from a viewpoint located outside the vehicle 1. In step S105, the vehicle drawing unit 133 uses a three-dimensional model VM representing the vehicle 1 to draw a vehicle image PV representing the vehicle 1 in the three-dimensional image PP.

[0043] Next, in step S109, the obstacle detection unit 134 detects the shape and distance of an obstacle T around the vehicle 1, and the obstacle drawing unit 135 generates a three-dimensional model TM representing the obstacle T based on the shape and distance of the obstacle T. In step S111, the obstacle drawing unit 135 generates a bounding box BB surrounding the obstacle T based on the shape and distance of the obstacle T;

[0044] In step S113, the obstacle drawing unit 135 determines whether the area SA of the region AR is equal to or greater than a first threshold value TH1. The region AR is a region where the three-dimensional model TM representing the obstacle T overlaps with the three-dimensional model VM representing the vehicle 1. If the obstacle drawing unit 135 determines that the area SA of the region AR is not equal to or greater than the first threshold value TH1 (step S113; NO), the process proceeds to step S121. If the obstacle drawing unit 135 determines that the area SA of the region AR is equal to or greater than the first threshold value TH1 (step S113; YES), the process proceeds to step S115. In step S115, the obstacle drawing unit 135 determines whether the distance LD between the vehicle 1 and the obstacle T is equal to or greater than a second threshold value TH2. If the obstacle drawing unit 135 determines that the distance LD is equal to or greater than the second threshold value TH2 (step S113; YES), the process proceeds to step S119. If the obstacle rendering unit 135 determines that the distance LD is less than the second threshold value TH2 (step S113; NO), the process proceeds to step S117.

[0045] In step S117, the obstacle drawing unit 135 displays the bounding box BB in the surface aspect AS on the display 61. For example, the obstacle drawing unit 135 draws one surface SP of the edges constituting the bounding box BB that faces the three-dimensional model VM representing the vehicle 1 as a surface image PS with a predetermined non-transparent transparency and displays it on the display 61. Thereafter, the process proceeds to step S123.

[0046] In step S119, the obstacle drawing unit 135 displays the bounding box BB in line pattern AL on the display 61. Of the edges that make up the bounding box BB, the obstacle drawing unit 135 emphasizes and draws the edges around the surface SP that faces the three-dimensional model VM representing the vehicle 1 as a line image SD in line pattern AL, and displays the image on the display 61. Thereafter, the process proceeds to step S123.

[0047] In step S121, the obstacle drawing unit 135 displays an obstacle image of a three-dimensional model TM representing an obstacle T viewed from a viewpoint located outside the vehicle 1 on the display 61. Thereafter, the process proceeds to step S123. In step S123, the display control device 100 determines whether or not to end the image display process. If the display control device 100 determines that the image display process should be ended (step S123; YES), the process is ended. If the display control device 100 determines that the image display process should not be ended (step S123; NO), the process returns to step S101, and the processes from step S101 onwards are repeatedly executed.

[0048] As described above with reference to Figures 1 to 8, the display control device 100 of this embodiment is a display control device 100 that controls the image to be displayed on the display 61, and is equipped with an image generation unit 132 that converts the viewpoint of a captured image PC that captures the surroundings of the vehicle 1 and generates a three-dimensional image PP of the vehicle 1 as seen from a virtual camera VC located outside the vehicle 1, a vehicle drawing unit 133 that draws a vehicle image PV that shows the vehicle 1 in the three-dimensional image PP based on a three-dimensional model VM that shows the vehicle 1, an obstacle detection unit 134 that detects the shape and distance LD of an obstacle T around the vehicle 1, and an obstacle drawing unit 135 that generates a three-dimensional model TM that shows the obstacle T based on the shape and distance LD of the obstacle T and draws the obstacle T in the three-dimensional image PP, and the obstacle drawing unit 135 replaces the obstacle image with a bounding box BB that surrounds the obstacle T based on the shape and distance LD of the obstacle T and draws it, and one side SP of the bounding box BB faces the three-dimensional model VM that shows the vehicle 1.

[0049] Therefore, by drawing the obstacle image as a bounding box BB that surrounds the obstacle T and faces the three-dimensional model VM whose side SP represents the vehicle 1, the user can easily visually recognize the distance between the obstacle T and the vehicle 1 without the obstacle image blocking the space between the obstacle T and the vehicle 1. This improves user convenience.

[0050] In the display control device 100, when a bounding box BB covers an area AR in a three-dimensional model VM representing a vehicle 1 that has an area equal to or greater than a first threshold value TH1, the obstacle drawing unit 135 draws the bounding box BB as a line image SD showing the multiple edges that make up the bounding box BB. If the area AR in the three-dimensional model VM representing the vehicle 1 is covered by the bounding box BB, displaying the bounding box BB as a surface image PS will hide the area AR of the three-dimensional model VM of the vehicle 1 by the surface image PS. If the area of the area AR is equal to or greater than the first threshold TH1, the bounding box BB is rendered as a line image SD, which prevents the three-dimensional model VM of the vehicle 1 from being hidden by the surface image PS. If the area of the area AR is equal to or greater than the first threshold TH1, the user can easily see the distance between the obstacle T and the vehicle 1.

[0051] In the display control device 100, the obstacle drawing unit 135 draws the bounding box BB in a line aspect AL, which draws the bounding box BB as a line image SD indicating the multiple edges that make up the bounding box BB, or in a surface aspect AS, which draws the surfaces of the bounding box BB as a surface image PS with a predetermined transparency. Therefore, because the obstacle drawing unit 135 draws the bounding box BB in the line aspect AL or the surface aspect AS, it can appropriately switch between the line aspect AL and the surface aspect AS for display. Therefore, by appropriately switching between the line aspect AL and the surface aspect AS, the user can easily visually recognize the distance between the obstacle T and the vehicle 1.

[0052] In the display control device 100, when the distance LD between the obstacle T and the vehicle 1 is equal to or greater than the second threshold TH2, the obstacle drawing unit 135 draws the bounding box BB in the line pattern AL, and when the distance LD between the obstacle T and the vehicle 1 is less than the second threshold TH2, the obstacle drawing unit 135 draws the bounding box BB in the surface pattern AS. Therefore, when the distance LD between the obstacle T and the vehicle 1 changes from a state where it is equal to or greater than the second threshold TH2 to a state where it is less than the second threshold TH2, the obstacle drawing unit 135 switches from the line pattern AL to the surface pattern AS to draw the bounding box BB. Therefore, because the bounding box BB is easier to visually recognize in the surface pattern AS than in the line pattern AL, the user can easily visually recognize the distance between the obstacle T and the vehicle 1 when the distance LD is less than the second threshold TH2.

[0053] In the display control device 100, the obstacle drawing unit 135 draws, in the line pattern AL, the edges that constitute the bounding box BB, emphasizing the edges around the surface SP that faces the three-dimensional model VM that represents the vehicle 1, as a line image SD. Therefore, in the line pattern AL, the edges around the surface SP that faces the three-dimensional model VM that represents the vehicle 1 are drawn as an emphasized line image SD, so that the user can easily visually recognize the distance between the obstacle T and the vehicle 1.

[0054] In the display control device 100, the obstacle drawing unit 135 draws, in the surface aspect AS, one surface SP that faces the vehicle 1, among the surfaces that make up the bounding box BB, as a surface image PS with a predetermined non-transparent transparency. Therefore, in the surface aspect AS, because the one surface SP that faces the vehicle 1 is drawn as a surface image PS with a predetermined non-transparent transparency, the user can easily visually recognize the distance between the obstacle T and the vehicle 1.

[0055] The control program PGM of this embodiment is a display control device 100 that has a processor 130 and controls images to be displayed on a display 61, and causes the processor 130 to function as an image generation unit 132 that converts the viewpoint of a captured image PC of the surroundings of the vehicle 1 to generate a three-dimensional image PP of the vehicle 1 as seen from a virtual camera VC located outside the vehicle 1, a vehicle drawing unit 133 that draws a vehicle image PV of the vehicle 1 in the three-dimensional image PP based on a three-dimensional model VM of the vehicle 1, an obstacle detection unit 134 that detects the shape and distance LD of an obstacle T around the vehicle 1, and an obstacle drawing unit 135 that generates a three-dimensional model TM of the obstacle T based on the shape and distance LD of the obstacle T and draws the obstacle T in the three-dimensional image PP, and the obstacle drawing unit 135 replaces the obstacle image with a bounding box BB that surrounds the obstacle T based on the shape and distance LD of the obstacle T and draws it, and one side SP of the bounding box BB faces the three-dimensional model VM of the vehicle 1. Therefore, the control program PGM according to this embodiment has the same effects as the display control device 100 according to this embodiment.

[0056] The above-described embodiment is merely an example of one embodiment of the present invention, and any modifications and applications are possible without departing from the spirit of the present invention.

[0057] For example, FIG. 1 shows components classified according to their main processing content to facilitate understanding of the present invention, and the components can be further classified into more components according to the processing content. A single component can also be classified to perform more processes. The processing of each component may be executed by one piece of hardware or by multiple pieces of hardware. The processing of each component may be realized by one program or by multiple programs. In FIG. 1, the display control device 100 may be integrally provided with the display unit 50.

[0058] In the present embodiment, the display control device 100 includes, but is not limited to, an image generation unit 132, a vehicle drawing unit 133, and an obstacle drawing unit 135. A server device communicatively connected to the display control device 100 via a network such as the Internet may include at least one of the image generation unit 132, the vehicle drawing unit 133, and the obstacle drawing unit 135. The server device may include, for example, the image generation unit 132. In this case, the load on the display control device 100 can be reduced. In the present embodiment, the obstacle drawing unit 135 determines whether to display a 3D model TM representing the obstacle T as an obstacle image or to draw a bounding box BB depending on whether the area SA of the region AR is equal to or greater than a first threshold value TH1, but is not limited to this. The obstacle drawing unit 135 may determine whether to display a 3D model TM representing the obstacle T as an obstacle image or to draw a bounding box BB based on, for example, the distance LD from the vehicle 1 to the obstacle T.

[0059] When the display control device 100 of the present invention is implemented using a computer, the control program PGM executed by the computer can be configured as a recording medium or a transmission medium for transmitting the control program PGM. The recording medium can be a magnetic or optical recording medium or a semiconductor memory device. Specific examples include portable or fixed recording media such as a flexible disk, HDD, CD-ROM (Compact Disk Read Only Memory), DVD, Blu-ray (registered trademark) Disc, magneto-optical disk, flash memory, and card-type recording medium. The recording medium can also be a non-volatile storage device such as RAM, ROM, or HDD provided in the display control device 100. The control program PGM can also be downloaded by the display control device 100 from a server device communicably connected to the display control device 100 via a network.

[0060] For example, the processing units in the flowchart shown in Fig. 8 are divided according to the main processing content to facilitate understanding of the processing of the display control device 100, and the method of dividing the processing units or the names thereof do not limit the present invention. The processing of the display control device 100 may be divided into more processing units according to the processing content. The processing of the display control device 100 may also be divided so that one processing unit includes more processes. [Explanation of symbols]

[0061] 100...display control device, 1...vehicle, 30...capturing unit, 40...distance sensor, 61...display, 130...processor, 132...image generation unit, 133...vehicle drawing unit, 134...obstacle detection unit, 135...obstacle drawing unit, 140...memory, 141...image storage unit, AL...line pattern, AR...area, AS...surface pattern, BB...bounding box, DG...point cloud data, LD...distance, PC...captured image, PGM...control program (display control program), PP...3D image, PS...surface image, PV...vehicle image, SD...line image, SP...one surface, T...obstacle, TM...3D model of obstacle, TH1...first threshold, TH2...second threshold, VM...3D model of vehicle.

Claims

1. A display control device that controls an image to be displayed on a display, an image generating unit that converts a viewpoint of an image captured of the surroundings of the vehicle to generate a three-dimensional image of the vehicle viewed from a viewpoint located outside the vehicle; a vehicle drawing unit that draws an image of the vehicle on the three-dimensional image based on a three-dimensional model that represents the vehicle; an obstacle detection unit that detects the shape and distance of an obstacle around the vehicle; an obstacle drawing unit that generates a three-dimensional model representing the obstacle based on the shape and distance of the obstacle, and draws an obstacle image based on the three-dimensional model representing the obstacle in the three-dimensional image; Equipped with the obstacle drawing unit draws the obstacle image by replacing it with a bounding box that surrounds the obstacle based on the shape and distance of the obstacle; One side of the bounding box faces the three-dimensional model representing the vehicle. Display control device.

2. the obstacle drawing unit, when an area of a region where the three-dimensional model representing the obstacle overlaps with the three-dimensional model representing the vehicle is equal to or larger than a first threshold, draws the obstacle by replacing it with a bounding box surrounding the obstacle. The display control device according to claim 1 .

3. the obstacle drawing unit draws the obstacle image based on the three-dimensional model representing the obstacle when an area of a region where the three-dimensional model representing the obstacle overlaps with the three-dimensional model representing the vehicle is less than a first threshold value; The display control device according to claim 1 .

4. the obstacle drawing unit draws the bounding box in a line manner by drawing a line image showing a plurality of sides that constitute the bounding box, or in a surface manner by drawing a surface of the bounding box as a surface image with a predetermined transparency; The display control device according to claim 1 .

5. When the distance between the obstacle and the vehicle is equal to or greater than a second threshold, the obstacle drawing unit draws the bounding box in the line manner, and when the distance between the obstacle and the vehicle is less than the second threshold, the obstacle drawing unit draws the bounding box in the surface manner. The display control device according to claim 4 .

6. the obstacle drawing unit draws, in the line style, the edges of the bounding box that are located around the one surface facing the three-dimensional model representing the vehicle, in an emphasized manner. The display control device according to claim 4 or 5.

7. the obstacle drawing unit draws, in the surface aspect, the one surface that faces the vehicle among the surfaces that constitute the bounding box with a predetermined transparency that is not transparent; The display control device according to claim 4 or 5.

8. A display control device including a processor for controlling an image to be displayed on a display, The processor, an image generating unit that converts the viewpoint of an image captured of the surroundings of the vehicle to generate a three-dimensional image of the vehicle viewed from a viewpoint located outside the vehicle; a vehicle drawing unit that draws an image of the vehicle on the three-dimensional image based on a three-dimensional model representing the vehicle; an obstacle detection unit that detects the shape and distance of an obstacle around the vehicle; and an obstacle drawing unit that generates a three-dimensional model representing the obstacle based on the shape and distance of the obstacle, and draws an obstacle image based on the three-dimensional model representing the obstacle in the three-dimensional image; It functions as the obstacle drawing unit draws the obstacle image by replacing it with a bounding box that surrounds the obstacle based on the shape and distance of the obstacle; One side of the bounding box faces the three-dimensional model representing the vehicle. Display control program.

Images